Python failing to retrieve all issues after 2000 rows

[James Anderson]

Hi.. I am using the below code as suggested in my previous query in the community.

This code is helping me to iterate through all the pages for extracting the complete list of issues from API. But the code is failing exactly after 2000+ rows approx and throwing an error message as highlighted below. I have no clue on how to make this code work.

Traceback (most recent call last):

File "C:\Users\John\Anaconda3\lib\site-packages\requests\models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)

File "C:\ Users\John\Anaconda3\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)

File "C:\Users\John\Anaconda3\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "C:\Users\John\Anaconda3\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None

Solved: JIRA API code to resolve pagination for issue sear... (atlassian.com)

import requests
from requests.auth
import HTTPBasicAuth
import json

url = "https://dataanalystteam.atlassian.net/rest/api/3/search"

auth = HTTPBasicAuth("Email", "API Token")

headers = {
"Accept": "application/json"
}

query = {
'jql': 'project = ITSAMPLE',"startAt": 0, "maxResults": 500

}

response = requests.request(
"GET",
url,
headers=headers,
params=query,

auth=auth
)

results = response.json()

while query["startAt"] < results["total"]:
query["startAt"] += 50
response = requests.request(
"GET",
url,
headers=headers,
params=query,
auth=auth
)
results = response.json()
for issue in results["issues"]:
print(issue["key"] + " | " + issue["fields"]["summary"])

@Marini Marini @Marini Marini 

[Prince Nyeche · Accepted solution]

The error states that JSON couldn't parse a None value. This means that the response received wasn't JSON serializable content, which most probably you received an empty string or content. Probably, you should reduce maxResults to 100 in the query variable, so it can match up your pagination rows as you traverse through the records.

[James Anderson]

@Prince Nyeche I tried reducing the max results to 100 but still getting the below message.

TypeError: can only concatenate str (not "NoneType") to str

Yes the fields like reporter , assignee are blank for some tickets. Plese advice how to resolves this...

[Prince Nyeche]

Looking at your code, the only place you did concatenation is here

print(issue["key"] + " | " + issue["fields"]["summary"])

So probably, the summary field is coming up with the none value. Also, change query["startAt"] += 100 since you're using maxResults 100. 

However, if your request is to fetch all the issues from your project, it would be easier if you just exported the data. Probably this article would be useful to you since you're using python.

[James Anderson]

@Prince Nyeche I tried the jiraone and it is really a wonderful post. But got the below error and i believe it is due to utf encode error or the attachments (as mentioned by one of the user who tried your code). Kindly help me with a final version of complete code to use this solution..

file_writer(folder, file_name,
            content=issues.content.decode('utf-8', errors="replace"),
            mark="file", mode="w+")

But I dont dont know how or where to use this in the below code to resolve my issue...

from jiraone import LOGIN, issue_export
import json

file = "config.json"
config = json.load(open(file))
LOGIN(**config)

jql = "project in (BB) order by created DESC"
issue_export(jql=jql)

ERROR MESSAGE

 File "C:/Users/KumarV2/Desktop/PBI Files/Python Scripts/jiraone_test.py", line 9, in <module>
issue_export(jql=jql)
File "C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\jiraone\reporting.py", line 1585, in export_issues
file_writer(folder, file_name,
File "C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\jiraone\reporting.py", line 2531, in file_writer
f.write(content)
File "C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u03a3' in position 418: character maps to <undefined>

[Prince Nyeche]

Hey @James Anderson 

That should be used on line 1586 of the actual reporting.py module code. I'm supposed to provide an updated version of this for a while now but I have got a few other updates to the issue_export() and other methods which I would like to bring out at once rather than in batches. The concept here is that you can modify the line above with a downloaded version of jiraone and call the methods directly, this would involve downloading the code from GitHub, modifying the line above and calling your script from within the downloaded version, this way it doesn't use your python3.11 version of jiraone. If you're unable to do that, then probably sometime this week, I can push an updated version with this same fix to the actual code that you can try as the current version doesn't have this fix yet.

[Prince Nyeche]

@James Anderson you know what, I just did an update to v0.7.4 which comes with a fix to this. Try it out and let me know if it works for you.

[James Anderson]

@Prince Nyeche I still getting the same error message. Not sure what is wrong. Please help me to fix it. I even updated JiraOne Lib. 

Below is the code I used

from jiraone import LOGIN, issue_export
import json

file = "config.json"
config = json.load(open(file))
LOGIN(**config)

jql = "project in (TEST) order by created DESC"
issue_export(jql=jql)

ERROR MESSAGE

 File "C:/Users/KumarV2/Desktop/PBI Files/Python Scripts/jiraone_test.py", line 9, in <module>
issue_export(jql=jql)
File "C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\jiraone\reporting.py", line 1585, in export_issues
file_writer(folder, file_name,
File "C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\jiraone\reporting.py", line 2531, in file_writer
f.write(content)
File "C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u03a3' in position 418: character maps to <undefined>

[Prince Nyeche]

@James Anderson you're still using an older version and not v0.7.4 based on a stack. To update your current version, you can uninstall and reinstall jiraone or you can do python3 -m pip install --upgrade jiraone

[James Anderson]

@Prince Nyeche I upgraded and tried again.  Now getting the below..

Downloading issue export in CSV format.
<Response [200]> OK ::downloading issues at page: 0 of 9
Traceback (most recent call last):

File "C:\Users\user123\Desktop\PBI Files\Python Scripts\untitled10.py", line 12, in <module>
issue_export(jql=jql)

File "C:\Users\user123\Anaconda3\lib\site-packages\jiraone\reporting.py", line 2189, in export_issues
file_writer(

File "C:\Users\user123\Anaconda3\lib\site-packages\jiraone\reporting.py", line 3223, in file_writer
f.write(content)

File "C:\Users\user123\Anaconda3\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]

UnicodeEncodeError: 'charmap' codec can't encode character '\u03a3' in position 409: character maps to <undefined>

[James Anderson]

@Prince Nyeche Kindly help

[Prince Nyeche]

It is an encoding problem, I'll try and see how this can work probably with your help I can finally stop this encoding issue in the window device. I'll push an update later today.

[Prince Nyeche]

Hey, @James Anderson please can you download the v0.7.5 I hope this solves it for you on the windows machine.

[James Anderson]

@Prince Nyeche -- Its a grand success. Amazing work and thanks a ton for answering my queries with much patience. Now all the tickets got saved as CSV without any issues.

Is there a way to add a condition as retrieve the tickets where the Updated Date is Last 3 days?

[Prince Nyeche]

@James Anderson Glad to hear that. You can simply just write any valid JQL of such a request and run the script to get the results you need.

jql = "project in (BB) AND updatedDate >=-3d"

Have a great day ahead.

[Mohamed Benziane]

Hi,

You have this response because the variable results is empty so when you call the json method it throw this error message.

Can you make sure that you have more that 2K issue with your JQL.

[James Anderson]

I don’t understand which variable is empty? How to make sure the JQL is more than 2k records? I have totally 8k records in the data. Appreciate your help