Updating thousands of tickets using web request API

[Erica Robinson]

Background:

We recently copied over tickets from one Cloud Jira to another. In the process, we updated the migrated projects' tickets to have a summary of {{issue.key}} | {{issue.summary}}. For some reason, one project didn't update properly and now we have thousands of tickets with a summary of " | " - fairly useless.

Why {{issue.key}} | {{issue.summary}}? We're going to move all of the migrated tickets into a single project and archive/delete the original projects. This is to allow teams to be able to access the tickets as needed still and keeping housekeeping (once done) to a minimum.

Goal:

I'm attempting to update all of the tickets with the summary of " | " to the needed value of {{issue.key}} | {{issue.summary}}. As of right now, I've managed to get this working for a single ticket in our sandbox. But once I apply it to multiple tickets, it fails.

Here's the sandbox rule that works for a single ticket:

Here's the automation for the production for multiple:

I have the final action emailing me as I want the final history log to be unchanged. Until I can get this working, I'll continue to have the results emailed to me.
As this is currently sitting, it's be reiterating over all the results (currently limited to ~65 tickets), but it's not providing the summary change from the history. Is there something I'm doing wrong for multiple tickets that I didn't account for?

[Evgenii]

Hi, @Erica Robinson 

I read through this thread, and I think the approach suggested is not the best way to go. Instead of using the REST API, I recommend using Automation smart values — they’re much better suited for this task.

With smart values, you can retrieve the previous value of the summary field quite easily.
I’ve even set up an example rule to show how it works (please ignore the trigger, as it's only set up for debugging purposes):

Just make rule, which is started by schedule, add there serach of required issues by JQL, and for each of them edit summary with this smart values combinations:

{{issue.key}} | {{#changelog.summary}}{{fromString}}{{/}}

 

[Bill Sheboy]

Hi @Evgenii 

Please note Erica was already trying to use the REST API endpoint for the changelog because the work items had already been updated when copied. 

The approach you suggest would help if each work item is being individually changed...but that has already happened.  The copy would need to be "undone" and repeated somehow to use the rule you suggest.

Kind regards,
Bill

[Evgenii]

Hi, @Bill Sheboy 

Not sure that I understood problem and your solution correctly.
If I got problem right, @Erica Robinson bulk moved issues and tried to change summaries, to "issue.key | issue.summary". But it changed to " | " for one project.
There must be old summary in issues changelogs, because Erica managed to change it for one test issue.
So, why she can't use changelog smart value? She will select in automation rule issues by JQL query. And rule will execute change for each issue, one by one.

[Bill Sheboy]

Thanks for clarifying your approach, Evgenii.

When Erica used a test rule to update one work item, that used the changelog records from the endpoint call.  I do not believe she manually changed the work item. 

And the changelog smart value is only available for a rule with a trigger for a field change: it is not available at any point in the future for other rules with different trigger types.  This is why the REST API was needed.

[Bill Sheboy]

Hi @Erica Robinson 

Although your images appear to show the steps, it is quite difficult to see your complete rule.  Perhaps post images of the complete rule in one single image and of the audit log details showing the rule execution to provide context.

Until we see those...

Your rule calls the REST API endpoint to get the changelog entries.  That endpoint can only return up to 100 records at a time.  And so if your Summary field change was not in the records returned, the rule will not find them.

And...because of what the rule is trying to reverse (i.e., a specific change to the Summary), you may want to check both the from and to values, ensuring you have identified the correct changelog record.

And...should the trigger's JQL only select work items to update when the Summary has one of the updated values?

Kind regards,
Bill

[Erica Robinson]

Hey Bill,

I see what you mean, apologies. Not my cleanest screenshots.

Here's the two rules in full side-by-side:

Here's the audit logs & the resulting output from the sandbox envo:

Here's the audit logs and the resulting output for one of the tickets identified by the JQL:

[Erica Robinson]

And...should the trigger's JQL only select work items to update when the Summary has one of the updated values?

I don't believe so, no. The trigger's JQL is searching for work items that were updated within a specific time frame (the time the original automation kicked in and mis-updated things).

What I ultimate need is the very last "fromString". What I noted when testing the web hook in postman is that the changelog is sorted from oldest to newest, not newest to oldest. So the very last entry in the changelog should be the previous summary value that I need.

[Bill Sheboy]

Thanks for that information, and some notes about it...

The JQL your trigger shown does not check the timeframe, and so that would run for the same issues repeatedly.  Rule triggers such as Scheduled with JQL can only process up to 100 issues at a time.  And so I was suggesting to add a check on the Summary value to the JQL to prevent re-processing the same ones.

Regarding the changelog, that will contain all changes, and so it is more difficult to find the last one for just the Summary field (as the endpoint does not support filtering by field changed). 

One way to do that is in steps:

• filter the web response data by the field equaling Summary, and store that result in a variable
• then split that variable back into a list (or into JSON) and then grab just the first record as that will likely be the most recent one

One final thing to check: it appears Atlassian plans to sunset the default, plain text renderer: https://community.atlassian.com/forums/Jira-articles/We-re-removing-the-default-text-renderer-in-Jira/ba-p/3025886

That means is there could be markup in the Summary field, including smart links when a work item key is included.  There is no timeline noted, and hopefully your conversion / cleanup will be done before that fully rolls out...Otherwise your rule may need to alter its logic.

[Erica Robinson]

The JQL your trigger shown does not check the timeframe, and so that would run for the same issues repeatedly. Rule triggers such as Scheduled with JQL can only process up to 100 issues at a time. And so I was suggesting to add a check on the Summary value to the JQL to prevent re-processing the same ones.

I know the sandbox doesn't have a timeframe specified, but I thought I had that for the production JQL?

project = "PROJECT" AND updated >= "-2d 5h" AND updated <= "-2d 3h" and (key NOT IN (PROJECT-5000,PROJECT-5001,PROJECT-5002)) and (key >= PROJECT-1500 AND key <= PROJECT-1700) ORDER BY key ASC, updated DESC, summary ASC

I had originally attempted to search for the summaries containing " | " since those were the ones affected, but JQL couldn't find them:

filter the web response data by the field equaling Summary, and store that result in a variable

I believe I've already done this step, or similar, in the second variable in both of my rules. However my variable pulls in the last "fromString" instead of the last "field" since they're all on the same level when looking at the output (i.e. "fromString" isn't nested under "field").

then split that variable back into a list (or into JSON) and then grab the first record as that will likely be the most recent one

I'm not sure I understand what you mean on this step. Could you elaborate?

That means there could be markup in the Summary field, including smart links when a work item key is included. There is no timeline noted, and hopefully your conversion / cleanup will be done before that fully rolls out...Otherwise your rule may need to alter its logic.

I'm hoping to have this migration finished by next week. This automation is my only blocker at the moment.

[Bill Sheboy]

One challenge using this web response is the nested lists: histories and items.  Thus we need nested iteration and list filtering to find the Summary field changelog records.

• action: create variable
  • name: varSummaryHistory
  • value:

{{#webhookResponse.body.changelog.histories}}
  {{#items}}
    {{#if(equals(field, "summary"))}}fromString: {{fromString}}, toString: {{toString}}~~{{/}}
  {{/}}
{{/}}

Please note the use of labels for the fields we want and the use of a double-tilde ~~ for a record delimiter.  You may adjust this based on the fields you need.

When we want the latest record, that would be the first one found when we split() the variable back into a list, and then a match() function can extract the fields:

The fromString would be: 

{{varSummaryHistory.split("~~").first.match("fromString: (.*), toString: .*")}}


And the toString would be:

{{varSummaryHistory.split("~~").first.match(".*toString: (.*)")}}

[Erica Robinson]

Hey Bill,

Thank you for the breakdown, I appreciate it. :) What you've outlined is exactly what I'm looking for.

I did add this to my rule this morning and attempted it; however the variable is returning empty. :(
Full rule adjustment:

Variable added:

Email generated:

Output received:

Audit log:

[Erica Robinson]

I wanted to update here with what I ended up doing as I wasn't able to get this working as we needed to do.

We ended up just deleting the migrated project and re-migrating it. We found the offending automation that caused the errors in the first place (fields were set to use "{{issueKey}} | {{issueSummary}}" instead of {{issue.key}} | {{issue.summary}}"). This has been updated and we're re-updating them now.

@Bill Sheboy I really appreciate all of your time and attention to detail in this matter. Apologies for any wasted time.

[Bill Sheboy]

Hi, @Erica Robinson 

No worries and I am glad you got it working with the "reset" to redo the project migration.

Regarding why that last test rule did not work, I suspect if you had written the entire web response's body to the log, it may have showed a different structure (or content) than what I was parsing.  This is a good technique to use when diagnosing problems parsing the response from an endpoint.

I hope you have a great week!

[Bill Sheboy]

@Erica Robinson -- Just to close the loop on this question, please consider adding your own answer describing what you did and then "accept" that as the answer.  This will help others in the future with similar needs.  Thanks!