How to handle concurrent incidents with automatic incident rules defined

[Coltin Rose]

I have stumbled into an issue where my team (incident management) receives an alert for an incident from team A and while managing this incident team B also needs to start an incident so they send an alert to our team as well. Team B alert get's automatically associated to team A's incident.

We rely heavily on slack so this is where most alerts originate from. We tried to make the alerts unique by adding the message as a tag but that doesn't seem to work.

We are open to suggestions.

Thanks in advance!

[Justin Sitarz · Accepted solution]

Hi Coltin, 

Justin here, from Opsgenie support - happy to help!

If I'm understanding correctly, it sounds like you have multiple groups channeling alerts to a single team within Opsgenie ('Incident Management'), and that Incident Management team is what contains the rules to spin up new incidents - is that correct?

If so - that may be problematic, as Opsgenie is set up to separate these incident rules out by team. Each team defines its own incident rules, and those rules are only evaluated for alerts assigned to that same team. 

So, one potential solution would be to set up team A and team B within Opsgenie, have them each define their own incident rules, and then route each team's alerts to their team in Opsgenie only, resulting in individual incidents for each team.

The other option would be to leave things as they are, and set up some additional filtering on the incident rule/s defined in the 'Incident Management' team. There's not currently a way to dynamically create new incidents, if, for example, there's a unique tag on the triggering alert, but you could add some data regarding the source of the alert (perhaps the team name), and then filter based on that value in the incident rule. 

There are potentially other options, but these are the two that jump out as the easiest to implement. However, if you have other questions about this, please feel free to reply here, or start up a new support ticket/chat with us - we'd be happy to help!

Best, 

Justin S.

Atlassian Cloud Support

[Sterling Houghton]

@Coltin Rose How did you guys end up resolving this issue? I am running into a similar issue as well.

[Coltin Rose]

@Sterling Houghton We opted to move from all the incident rules being defined in our team to having them defined for each team in OpsGenie. We then updated our internal docs to include adding our team as a responder if in fact the incident that was created is a true incident. This helped with the transition while the teams were adjusting their alerting so false incidents weren’t reported and prevented alert fatigue with my team getting paged for non incidents.

hope this helps!