Why is my script getting random 401's and 503's calling the search REST API

[Jason Knight]

I run a python script locally to asynchronously retrieve large result sets (1,000 or more records) based on a jql query. I use local, asynchronously GET requests to speed up the retrieval of large sets. I specify a page size of 50 when retrieving the paged results.

I just ran a "created > -365d" query which produced over 12K results. After about 2100 results were returned, I started receiving intermittent 401 errors until it finished. I'll also see 503 errors occasionally.

Here's an example of my output log for this most recent run:

Found 12537 total results.
Done: 08:39:23, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=350 (200)
Done: 08:39:23, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=600 (200)
Done: 08:39:23, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1100 (200)
Done: 08:39:23, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=900 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=850 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=200 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=550 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=0 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1000 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=50 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1050 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=100 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=150 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=800 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1150 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=750 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=400 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=500 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=700 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1200 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=300 (200)
Done: 08:39:24, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=450 (200)
Done: 08:39:25, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=950 (200)
Done: 08:39:25, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=250 (200)
Done: 08:39:25, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=650 (200)
Done: 08:39:27, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1250 (200)
Done: 08:39:27, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1300 (200)
Done: 08:39:27, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1600 (200)
Done: 08:39:27, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1800 (200)
Done: 08:39:27, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1750 (200)
Done: 08:39:27, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1550 (200)
Done: 08:39:27, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=2050 (200)
Done: 08:39:27, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1500 (200)
Done: 08:39:27, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1450 (200)
Done: 08:39:27, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1400 (200)
Done: 08:39:27, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=2100 (200)
Done: 08:39:28, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=1350 (200)
############FAILURE############
Done: 08:39:28, https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=2700 (401)
<ClientResponse(https://myjira.atlassian.net/rest/api/3/search?expand=changelog&jql=created+%3E+-365d&startAt=2700) [401 None]>
<CIMultiDictProxy('Server': 'Atlassian Proxy/1.13.6.2', 'Www-Authenticate': 'OAuth realm="https%3A%2F%2Fmyjira.atlassian.net"', 'Content-Type': 'text/html;charset=UTF-8', 'Strict-Transport-Security': 'max-age=315360000; includeSubDomains; preload', 'Date': 'Wed, 30 Jan 2019 14:39:28 GMT', 'ATL-TraceId': '631552c17bb7af09', 'X-AREQUESTID': '02a15f51-8b78-4871-a4b7-6d8ceeb9a0db', 'X-XSS-Protection': '1; mode=block', 'Transfer-Encoding': 'chunked', 'Timing-Allow-Origin': '*', 'X-Content-Type-Options': 'nosniff', 'Connection': 'keep-alive', 'X-Seraph-LoginReason': 'AUTHENTICATED_FAILED')>

 What's really puzzling is after the failures start, I'll still see 200 a-ok requests with every other 3rd or 4th request erroring with a 401.

Other details of note. I'm spinning up a client session which stays alive the whole time and limits itself to 30 tcp connections at a time.