Everyday we are receiving jira High CPU Usage Alert

[Shubham Purohit]

Hi there 
Everyday we are receiving High CPU Usage alert around 1:30 Am and auto resolved at 2:00 Am.
We have checked the processes which are running over the tlpajira-1/2 server and found the culprit process Java running with user "appjira
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
56245 appjira   20   0   21.5g  16.7g   1.4g S 376.3 53.9   9254:20 java
57816 p228973   20   0  172908   2428   1632 R   4.7  0.0   0:03.70 top
4077 splunk    20   0  596072 247892  12740 S   1.6  0.8 603:39.82 splunkd

Checked the Parent process with the PID. and found some issue related to the memory allocation and garbage collection, but after checking the vim atlassian-jira-gc-2024-06-03_22-55-37.log.3.current  file found the GC process is running and completing successfully within relatively short pauses, which proves GC is not primary cause. also checked the jira application logs and found error related to plugins module.

vim access_log.2024-06-13
172.21.255.237 15x9575031x1 - [13/Jun/2024:00:15:51 -0500] "GET /s/d292f865eed19dadcf6169462458d65b-CDN/jjsogp/940008/1dlckms/9.4.8/_/download/batch/com.atlassian.jira.jira-project-config-plugin:error-module/com.atlassian.jira.jira-project-config-plugin:error-module.js?locale=en-US HTTP/1.1" 200 1469 15 "https://jira.edwardjones.com/projects/CSSV?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0" "p67vq7"

Under the catalina.2024-06-13.log an issue was identified where a thread in the application was stuck and had been active for an extended period of time (over 124 seconds) and A new log entry indicated that this previous stuck thread eventually completed its task. From Grafana we verified  after completion of the task the CPU get down at its normal state. there are multiple stuck threads under log file.

vim catalina.2024-06-13.log:
18-Jun-2024 09:07:16.085 WARNING [Catalina-utility-1] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [https-jsse-nio-8443-exec-320] (id=[249734]) has been active for [129,490] milliseconds (since [6/18/24 9:05 AM]) to serve the same request for [https://jira.edwardjones.com/rest/api/latest/search?jql=project+%3D+%22QE+Shared+Services%22+AND+status+not+in+%28Canceled%2C+Cancel%2C+Cancelled%29+AND+type+%3D+Bug+AND+createdDate+%3E%3D+%222024%2F01%2F22%22+AND+cf%5B10502%5D+in+%28+%22CLONE+-+Pre-Prod%22%2C+%22clone+-+UAT%22%29+and+%22Epic+Link%22+%3D+ENFWKS-23513+ORDER+BY++created+DESC&startAt=0&maxResults=500&expand=changelog] and may be stuck (configured threshold for this StuckThreadDetectionValve is [120] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.


18-Jun-2024 09:07:46.120 WARNING [Catalina-utility-2] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [https-jsse-nio-8443-exec-320] (id=[249734]) was previously reported to be stuck but has completed. It was active for approximately [152,477] milliseconds

Could you please guide us how to resolve this issue. as it is recurring every day 

[Jim Knepley - ReleaseTEAM]

Maybe traffic between your servers and the IP address(es) behind jira.edwardjones.com is being blocked/dropped?

I've seen stuck threads running up the CPU if Jira isn't able to connect to an Internet-facing website. If Jira's attempts to connect to the Internet are silently dropped for whatever reason (egress firewall in my case), that could be the root of the issue.

[Rob Horan]

Every day off hours sounds like a recurring backup or indexing operation.