JIRA API code to resolve pagination for issue search

[James Anderson]

Trying to get all issues from JIRA cloud using the below code. But due to pagination I am unable to get more than 50 rows but I have a total of 500+ rows in data. I tried a lot to do a for loop but unable to get the output as JSON file with more than 50 rows. Please help.

I have to find the total number of records in the call and use the parameters startAt and maxResults to get the data. Kindly help me with pagination.

import requests
from requests.auth 
import HTTPBasicAuth
import json

url = "https://dataanalystteam.atlassian.net/rest/api/3/search"

auth = HTTPBasicAuth("Email", "API Token")

headers = {
  "Accept": "application/json"
}

query = {
  'jql': 'project = ITSAMPLE',"startAt": 0, "maxResults": 500
  
}

response = requests.request(
   "GET",
   url,
   headers=headers,
   params=query,
   
   auth=auth
)

print(json.dumps(json.loads(response.text), sort_keys=True, indent=4, separators=(",", ": ")))

[Marini Marini · Accepted solution]

Hi @James Anderson 

The maxResult = 50 is hardcoded by the API. It means that you can only get a max of 50 records whenever you execute an API.

If you have more than 50 records, you will need to send a request multiple times. In the first response, you will get the value of total that represent the total records from the JQL.

You can use for or while loop to send multiple requests. For example:

.
.
.
.

results = response.json()

while query["startAt"] < results["total"]:
    query["startAt"] += 50
    response = requests.request(
        "GET",
        url,
        headers=headers,
        params=query,
        auth=auth
    )
    results = response.json()
    for issue in results["issues"]:
        print(issue["key"] + " | " + issue["fields"]["summary"])

I hope it helps.

Best regards,

Marini

[James Anderson]

@Marini Marini Thank you so much.. Code worked like a charm.. This is really gonna help a lot others like me. 

[James Anderson]

@Marini Marini Once again thank you so much for helping. I ran the code in Python and It started printing the results and after 2000 rows it is exactly failing at certain point with below errors.

I have no clue on how to fix this..

Traceback (most recent call last):

File "C:\Users\John\Anaconda3\lib\site-packages\requests\models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)

File "C:\Users\John\Anaconda3\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)

File "C:\Users\John\Anaconda3\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "C:\Users\John\Anaconda3\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None

[Marini Marini]

Hi @James Anderson 

Whenever we try to execute an API, there is always a possibility of a bad response. It could be due to a network issue or the target system being unable to process your requests. Since you are trying to retrieve large records from Jira, I would recommend always checking the response status code before processing it.

.
.
.
.
if response.status_code == 200:
    try:
        print(response.status_code)
        print("query.startAt: " + str(query["startAt"]))
        print("result.total: " + str(results["total"]))
        results = response.json()
        for issue in results["issues"]:
            print(issue["key"] + " | " + issue["fields"]["summary"])
     except KeyError:
        print(KeyError)
        print("response.status_code: " + response.status_code) 

From there, you will be able to identify which request returns a bad response and start the investigation from there.

Best regards,

Marini

[Phuong Vu]

What should I do if the response do not return the 'total' field?
I don't know  the total number of results to get the startAt number limit