Assistance creating Jira ticket as Prometheus alert fires

[Ashma Shriyan]

Hello 

I have a requirement where in a Jira ticket automatically needs to be created as Prometheus alert fires. As of now, we also have a setup where the alert gets received on Slack as well. 

I'm trying to use webhook url in the alertmanager configuration to use jira link. Below is how my alertmanager config looks like:

--------------------------- 

receivers:

- name: 'slack-general'
webhook_configs:
- url: "https://example.atlassian.net/rest/api/2/issue/createmeta?projectKeys=TRE&issuetypeNames=Support"
slack_configs:
- api_url: "https://<>"
username: '{{ template "slack.default.username" . }}'
color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
title: '{{ template "slack.default.title" . }}'
title_link: '{{ template "slack.default.titlelink" . }}'
pretext: '{{ .CommonAnnotations.summary }}'
text: |-
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
*Description:* {{ .Annotations.description }}
*Details:*
{{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
{{ end }}
• *Start Time:* {{ .StartsAt }}
{{ end }}
fallback: '{{ template "slack.default.fallback" . }}'
icon_emoji: '{{ template "slack.default.iconemoji" . }}'
icon_url: '{{ template "slack.default.iconurl" . }}'
#title: '{{ template "custom_title" . }}'
#text: '{{ template "custom_slack_message" . }}'
#text: '{{ template "email.instance.html" . }}'
- name: 'all-support'
email_configs:
- to: 'abc@abc.com'
html: '{{ template "email.instance.html" . }}'
headers:
subject: "ALERT!!: {{ .CommonLabels.instance}} | {{ .CommonLabels.job }} | {{ .CommonLabels.alertname }}"

templates:
- '/etc/prometheus/templates/instance.tpml'
- '/etc/alertmanager/template/*.tmpl'

But with this setup I'm gettting 405 error as seen below:

alertmanager[4870]: level=error ts=2019-06-26T05:59:02.698618182Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"webhook\" due to unrecoverable error: unexpected status code 405 from https://example.atlassian.net/rest/api/2/issue/createmeta?projectKeys=TRE&issuetypeNames=Support"

I have referred the document  https://developer.atlassian.com/server/jira/platform/jira-rest-api-examples/#getting-metadata-for-creating-issues-examples and from what I see it requires POST body with 'input' fields to be entered. Wondering how do i input these in the alertmanager config or anywhere else?. Can someone please guide me how to set this up.

Thanks

[Dario B · Accepted solution]

Hi @Ashma Shriyan ,

I don't know how Prometheus alertmanager configuration works and what is the right syntax. However, if you are trying to have tickets  created in Jira once an alert is fired, then the easiest way to achieve this would be the following:

1. Configure Prometheus to send an email to Jira when an alert fires
2. In Jira create a user having as email address the one Prometheus uses to send emails (or configure a default reporter as explained in the documentation page in the next step)
3. Create an email handler that will create tickets from these emails in the desired project in Jira as explained in the following documentation page: Creating issues and comments from email

In case this is not helping, or you are trying to do something different then please provide more details and/or an example.

Finally, I am tagging @Alexey Matveev in this thread since he is the one that wrote the below article and he may be able to help in here (also, review the below page, you may find it interesting):

• Monitoring of Atlassian applications with Prometheus

I hope this helps.

Cheers,
Dario

[Ashma Shriyan]

Hi Dario,

Thanks for your response. 

I liked the solution that you proposed. I will go with it if I'm unable to achieve what  I want with my current setup.

I understand that you are not familiar with Prometheus setup but to define briefly, I have been following https://github.com/free/jiralert/blob/master/README.md to create automated ticket in Jira as Prometheus alert triggers. The current status is that, the test command mentioned below (as per the link) returns no error and a ticket gets created in Jira but when an alert is triggered in Prometheus there is no ticket created and an error "level=error ts=2019-06-28T06:04:44.252355609Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="context deadline exceeded" is logged, which is where I'm stuck.

 curl -H "Content-type: application/json" -X POST \
-d '{"receiver": "slack-general", "status": "firing", "alerts": [{"status": "firing", "labels": {"alertname": "TestAlert", "key": "value"} }], "groupLabels": {"alertname": "TestAlert"}}' \
http://localhost:9097/alert  

Thanks!

[Dario B]

Hi @Ashma Shriyan ,

If I understand correctly:

1. If you simulate that an alert was fired by calling the  below curl command everything works fine:
   

   curl -H "Content-type: application/json" -X POST -d '{"receiver": "slack-general", "status": "firing", "alerts": [{"status": "firing", "labels": {"alertname": "TestAlert", "key": "value"} }], "groupLabels": {"alertname": "TestAlert"}}' http://localhost:9097/alert  

2. An error is returned when a real alert is fired in Prometheus.

If this is correct, then the issue seems to be that JiraAlert either did not receive a webhook from Prometheus at all, or it received a wrong one. 

Now, having a quick look at the below documentation page, it looks like you are setting the wrong url for the WebHook receiver:  

• https://prometheus.io/docs/alerting/configuration/#webhook_config

Indeed, when simulating the WebHook call by using curl you are sending the POST request to: 

• http://localhost:9097/alert

While in your configuration file I can see  that the url for the webhook receiver is set to:  

• https://example.atlassian.net/rest/api/2/issue/createmeta?projectKeys=TRE&issuetypeNames=Support (this looks like an example url, I assume in the config you were having your real Jira url).

To summarize: in Prometheus WebHook URL, you have to configure the URL to reach the machine on which JiraAlert is running (localhost is only fine if they are both running on the same machine) and not the one to reach Jira. 
Something like:

• http://MACHINE_RUNNING_JIRALERT:9097/alert

Please make sure that the url is reachable from the machine where Prometheus is running.

Finally, please notice that neither jiralert nor Prometheus are Atlassian products and in case you need further help it might be a good idea to get in touch with Prometheus community as explained in: https://prometheus.io/community/

Have a nice weekend

Dario

[Brad Collins]

Hey,

We're working with ZigiWave for some time. They're having a large number of integrations, so you can try reaching out to them for help. Their specialists and support team are extremely good. Their website is https://zigiwave.com/jira-software/ and might be able to help.

Regards,
Brad

[evasheeva]

Hey guys, apologies on the delay getting back to this. I was successfully able to configure jiralert and the ticket is also getting created as alert fires. 

Thanks all for your help and suggestion.

[Dario B]

@evasheeva  Just to know, was it the WebHooks URL in the configuration or anything else?  :) 

Have a nice weekend!

[evasheeva]

yes, it was just the incorrect webhook url :) 

[Kumar]

@Ashma Shriyan I am also trying to implement Automatic jira ticket creation for critical prometheus alert.
could you please help me in that what is the exact procedure to follow?

[Ashma Shriyan]

@Kumar You can refer to my blog here https://medium.com/@ashmasuv/how-to-automatically-create-jira-service-desk-ticket-as-prometheus-alert-triggers-1232c4d6a0d9

[Kumar]

@Ashma Shriyan I found some bugs in jiralert tool and I feel ,it is not secured one.
Do you have any other idea of creating jira ticket automatically other than jiralert and email handler method.

[Andrew L]

Hi,

When i run this:

curl -H "Content-type: application/json" -X POST -d '{"receiver": "jira-ab", "status": "firing", "alerts": [{"status": "firing", "labels": {"alertname": "TestAlert", "key": "value"} }], "groupLabels": {"alertname": "TestAlert"}}' http://localhost:9097/alert

I get an error about "unsupported value type" :

level=error ts=2020-01-09T01:34:49.698466622Z caller=main.go:143 msg="error handling request" statusCode=500 statusText="Internal Server Error" err="JIRA request https://myjira.mycompany.com/rest/api/2/search?jql=project%3D%22MYPROJECT%22+and+labels%3D%22ALERT%7Balertname%3D%5C%22TestAlert%5C%22%7D%22+order+by+resolutiondate+desc&maxResults=2&fields=summary,status,resolution,resolutiondate returned status 400 , body \"\"" receiver=jira-ab groupLabels="unsupported value type"

What am i doing wrong?

Thanks

--Andrew

[Dario B]

Hi @Andrew L ,

Error 400 (bad request) means that something is wrong with the body of the request. 

Specifically, what is returning error 400 is the below REST API GET request to the /rest/api/2/search endpoint:

• https://myjira.mycompany.com/rest/api/2/search?jql=project="MYPROJECT"+and+labels="ALERT{alertname=\"TestAlert\"}"+order+by+resolutiondate+desc&maxResults=2&fields=summary,status,resolution,resolutiondate

Looking at the above url, I strongly believe that what is causing the error is: ALERT{alertname=\"TestAlert\"} 

In order to whether confirm or deny, you can just copy paste the full URL (above) into your web browser (assuming that you can access myjira.mycompany.com from there) and see if the request is correctly executed or if it returns the same error, with and without the offending bit.

Cheers,
Dario

[Andrew L]

Thanks, Dario.

Yes... The same exact API url produces a HTTP 500 in my web browser.

If i run the query via the Jira UI (https://myjira.mycompany.com/issues/?jql=.....),
I don't get any errors (but the search yields no records)

Does this mean my version of Jira (i'm running v7.4.4) doesn't support that query (ALERT{alertname=\"TestAlert\"} ?

Then what Jira version is supported?

Thanks

--Andrew

[Dario B]

Hi @Andrew L ,

Can you please clarify below points:

1. Do you have some issues in your Jira having label: ALERT{alertname="TestAlert"} ?
2. If my understanding is correct, the same search query works fine if called from the UI (https://myjira.mycompany.com/issues/?jql=.....) and fails returning error 500 if called using the REST API endpoint: (https://myjira.mycompany.com/rest/api/2/search?jql=....).
   This is odd since the JQL should work the same way, regardless the way you are calling is (UI or REST API). So, if my understanding is the correct, please paste the URL that works and the one that does not work in your reply. Provide more details otherwise.

Cheers,
Dario

[Richard Cross]

@AndrewL / @Dario B 

Bear in mind that Jiralert has not seen a release in nearly a year.  Additionally, the documentation seems to imply that it's designed for Jira Cloud, rather than Jira Server/Data Center.

I'm not getting the exact same issue as Andrew, but Jiralert is failing to re-open existing closed tickets... even though it can correctly avoid duplicating tickets in Open and In Progress, as well as closed tickets with the "Won't Fix" resolution. 

To be honest, you're better off writing your own webhook for this.

[Dario B]

Looking at Jiralert GitHub issues I can see that the last thread, from November 2019, is proposing to move to a different project (prometheus-community):

• https://github.com/free/jiralert/issues/49
• https://github.com/prometheus-community/community/issues/6

Not sure if there is any progress since then.