Atlassian Jira failed to start after upgrading jira from version 8.9 to 8.20.10

[Rashmi]

Hi Team!

I have upgraded jira from version 8.9 to jira 8.20 software. After starting jira i tried to connect server to database (oracle) but then I got error in connection that database is not empty. due to which I manually craeted dbconfig.xml file in Jira_home and restarted my jira.. now I see jira running in browser but status of jira is failed-

 systemctl status jira
● jira.service - Atlassian Jira
Loaded: loaded (/usr/lib/systemd/system/jira.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2022-07-08 09:47:50 CEST; 4h 8min ago
Process: 276516 ExecStart=/opt/jira/jira_inst/bin/start-jira.sh (code=exited, status=1/FAILURE)
Main PID: 225299 (code=killed, signal=KILL)

Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: Using JIRA_HOME: /opt/jira/jira_home
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: Server startup logs are located in /opt/jira/jira_inst/logs/cata>
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: Existing PID file found during start.
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: Tomcat appears to still be running with PID 235412. Start aborte>
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: If the following process is not a Tomcat process, remove the PID>
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: UID PID PPID C STIME TTY TIME CMD
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: jira 235412 1 3 Jul07 ? 00:25:53 /usr/lib/jvm>
Jul 08 09:47:50 x15855tzz.test.tre.se systemd[1]: jira.service: Control process exited, code=exited status=1
Jul 08 09:47:50 x15855tzz.test.tre.se systemd[1]: jira.service: Failed with result 'exit-code'.
Jul 08 09:47:50 x15855tzz.test.tre.se systemd[1]: Failed to start Atlassian Jira.
lines 1-16/16 (END)

In catalina.out- 

I see two warnings-

In which one is for Re-indexing which I delibrately set to false I did not want the index of db change as I am using the same db to connect with upgraded host.

created com.atlassian.jira.plugins.webhooks.ao.WebhookDaoImpl@7163132f 1902318383 with events jira:issue_updated
version_unreleased
version_created
jira:issue_created
version_updated
jira:issue_deleted
jira:worklog_updated
version_released
version_moved
AO_4AEACD_WEBHOOK_DAO {ID = 22} 55306
2022-07-08 11:52:26,601+0200 Caesium-1-4 WARN [c.a.r.internal.settings.RateLimitModificationSettingsService] No RL settings found for user: [JIRAUSER32553] - can't delete!!
2022-07-08 11:52:35,756+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.UpgradeScheduler] Running scheduled upgrades
2022-07-08 11:52:35,761+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.LoggingUpgradeService] run upgrades has started
2022-07-08 11:52:35,864+0200 Caesium-1-2 INFO ServiceRunner [c.a.j.upgrade.util.BuildNumberDao] Setting current build number to 820010
2022-07-08 11:52:35,866+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.UpgradeIndexManager] There are no reindex requests of type [IMMEDIATE, DELAYED] so none will be run
2022-07-08 11:52:35,866+0200 Caesium-1-2 INFO ServiceRunner [c.a.j.upgrade.util.BuildNumberDao] Setting current version to 8.20.10
2022-07-08 11:52:35,866+0200 Caesium-1-2 INFO ServiceRunner [c.a.j.upgrade.util.BuildNumberDao] Setting downgrade version to 7.1.2
2022-07-08 11:52:35,883+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.LoggingUpgradeService] run upgrades has finished successfully, and took 122 milliseconds to process.
2022-07-08 11:52:35,883+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.UpgradeScheduler] JIRA upgrades completed successfully
2022-07-08 11:52:35,895+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.UpgradeScheduler] Plugins upgrades completed successfully
2022-07-08 11:52:35,895+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.UpgradeIndexManager] Reindexing is not allowed after this upgrade and there is no immediate reindex requests
2022-07-08 11:52:46,645+0200 Caesium-1-1 INFO [c.a.jira.startup.CacheWarmerLauncher] Warmed cache(s) in 23885 ms.

On browser I see the upgraded version-

Hi Team!

I have upgraded jira from version 8.9 to jira 8.20 software. After starting jira i tried to connect server to database (oracle) but then I got error in connection that database is not empty. due to which I manually craeted dbconfig.xml file in Jira_home and restarted my jira.. now I see jira running in browser but status of jira is failed-

 systemctl status jira
● jira.service - Atlassian Jira
Loaded: loaded (/usr/lib/systemd/system/jira.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2022-07-08 09:47:50 CEST; 4h 8min ago
Process: 276516 ExecStart=/opt/jira/jira_inst/bin/start-jira.sh (code=exited, status=1/FAILURE)
Main PID: 225299 (code=killed, signal=KILL)

Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: Using JIRA_HOME: /opt/jira/jira_home
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: Server startup logs are located in /opt/jira/jira_inst/logs/cata>
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: Existing PID file found during start.
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: Tomcat appears to still be running with PID 235412. Start aborte>
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: If the following process is not a Tomcat process, remove the PID>
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: UID PID PPID C STIME TTY TIME CMD
Jul 08 09:47:50 x15855tzz.test.tre.se start-jira.sh[276516]: jira 235412 1 3 Jul07 ? 00:25:53 /usr/lib/jvm>
Jul 08 09:47:50 x15855tzz.test.tre.se systemd[1]: jira.service: Control process exited, code=exited status=1
Jul 08 09:47:50 x15855tzz.test.tre.se systemd[1]: jira.service: Failed with result 'exit-code'.
Jul 08 09:47:50 x15855tzz.test.tre.se systemd[1]: Failed to start Atlassian Jira.
lines 1-16/16 (END)

In catalina.out- 

I see two warnings-

In which one is for Re-indexing which I delibrately set to false I did not want the index of db change as I am using the same db to connect with upgraded host.

created com.atlassian.jira.plugins.webhooks.ao.WebhookDaoImpl@7163132f 1902318383 with events jira:issue_updated
version_unreleased
version_created
jira:issue_created
version_updated
jira:issue_deleted
jira:worklog_updated
version_released
version_moved
AO_4AEACD_WEBHOOK_DAO {ID = 22} 55306
2022-07-08 11:52:26,601+0200 Caesium-1-4 WARN [c.a.r.internal.settings.RateLimitModificationSettingsService] No RL settings found for user: [JIRAUSER32553] - can't delete!!
2022-07-08 11:52:35,756+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.UpgradeScheduler] Running scheduled upgrades
2022-07-08 11:52:35,761+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.LoggingUpgradeService] run upgrades has started
2022-07-08 11:52:35,864+0200 Caesium-1-2 INFO ServiceRunner [c.a.j.upgrade.util.BuildNumberDao] Setting current build number to 820010
2022-07-08 11:52:35,866+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.UpgradeIndexManager] There are no reindex requests of type [IMMEDIATE, DELAYED] so none will be run
2022-07-08 11:52:35,866+0200 Caesium-1-2 INFO ServiceRunner [c.a.j.upgrade.util.BuildNumberDao] Setting current version to 8.20.10
2022-07-08 11:52:35,866+0200 Caesium-1-2 INFO ServiceRunner [c.a.j.upgrade.util.BuildNumberDao] Setting downgrade version to 7.1.2
2022-07-08 11:52:35,883+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.LoggingUpgradeService] run upgrades has finished successfully, and took 122 milliseconds to process.
2022-07-08 11:52:35,883+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.UpgradeScheduler] JIRA upgrades completed successfully
2022-07-08 11:52:35,895+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.UpgradeScheduler] Plugins upgrades completed successfully
2022-07-08 11:52:35,895+0200 Caesium-1-2 INFO ServiceRunner [c.a.jira.upgrade.UpgradeIndexManager] Reindexing is not allowed after this upgrade and there is no immediate reindex requests
2022-07-08 11:52:46,645+0200 Caesium-1-1 INFO [c.a.jira.startup.CacheWarmerLauncher] Warmed cache(s) in 23885 ms.
[root@x15855tzz logs]#

Versions & licenses

Upload an application

Jira Software 8.20.10

1000 users (522 used) 

License expires04/Feb/23

Support entitlement number 

License typeDeveloper (Data Center)

Can you please suggest me what I can do in this case?

Note-I have disabled some of the apps during upgrade as it was not showing easy updrage to newer version. Some of them is still disabled. I have not enabled them.

[Bastian Stehmann]

Hi @Rashmi ,

If Jira is running well and can be accessed in the browser, it should be ok.

Did you start Jira with systemctl or directly by the start script. Maybe stopping the service and restarting will fix the status of the service.

[Rashmi]

I did that and issue seems to be resolved now thanks

[Rashmi]

I have one more concern that I want to create some cleanup activity in jira , which means it should delete data which is very old like 2 or 3 years back.. how can we do that?

[Rashmi]

Or some suggestions on how can we manage the space issue in Jira.. instead of keep upgrading the server hardware ? 

[Rashmi]

And if everything is fine then why I am seeing this kind of alerts after upgrade

[Bastian Stehmann]

You have to check, what projects can be removed. An indicator might be the last updated date in the projects overview.

But you'll need to check with the users.

Space issues  can be solved only by deleting projects or issues, so you need that cleanup you are planning.

Otherwise you have only the option to increase disk space.

The warning regarding the user is known and is not really warning more information. So they can be ignored.

The re-index thing is only an information that no index happens. 

[Rashmi]

okay.. but when I search for any application in jira.. its shows me it is searching and then it looks like a forever thing and never returns the result of search.. why is that happening? How can I solve the same,its not a network problem because i have another jira server where it finds any apps it gives the result quickly.

Regards,

Rashmi

[Bastian Stehmann]

Maybe you have to configure an outbound proxy.

https://confluence.atlassian.com/jirakb/configure-an-outbound-proxy-for-use-in-jira-server-247857187.html

[Rashmi]

@Bastian Stehmann My Jira server is already connected with load balancer still i need to configure Outbound proxy? I don not have it configured for production jira still that works fine

[Bastian Stehmann]

The load balancer is for incoming traffic only, the outbound proxy is used to access the internet  check with your network team if you need an proxy to access the internet and if the server can reach internet at all..

[Rashmi]

In my organisation we don't have outbound internet proxy. It is just handled through firewall opening , I have checked and 

Firewall is already opened:
[Myjiraserver~]$ telnet marketplace.atlassian.com 80
Trying 185.166.143.34...
Connected to marketplace.atlassian.com.
Escape character is '^]'.
^C^C^CConnection closed by foreign host.
[hiadmin@x15855tzz ~]$
[hiadmin@x15855tzz ~]$
[hiadmin@x15855tzz ~]$ curl -vk https://marketplace.atlassian.com/
* Trying 185.166.143.33...
* TCP_NODELAY set
* Connected to marketplace.atlassian.com (185.166.143.33) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=AU; ST=New South Wales; L=Sydney; O=Atlassian Pty Ltd; CN=*.atlassian.com
* start date: May 10 00:00:00 2022 GMT
* expire date: Jun 10 23:59:59 2023 GMT
* issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* Using Stream ID: 1 (easy handle 0x557f4c32f5e0)
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET / HTTP/2
> Host: marketplace.atlassian.com
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* Connection state changed (MAX_CONCURRENT_STREAMS == 64)!
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/2 200
< date: Fri, 15 Jul 2022 10:44:40 GMT
< content-type: text/html
< content-length: 7615
< last-modified: Fri, 15 Jul 2022 05:34:59 GMT
< x-amz-version-id: DrFkOvlSxIOrcmset8K0PEoIOWcAyXSG
< server: globaledge-envoy
< etag: W/"41f6332eb107f7485fac4b2545c103e2"
< via: 1.1 2b0c54ffe9876882253b010d44184bdc.cloudfront.net (CloudFront),1.1 varnish (Varnish/6.5)
< x-amz-cf-pop: IAD89-P2
< x-amz-cf-id: b5eajE0cDfCd0tu5ckllsMeEY2qaB827xyzjCsQcuKcNBiOUXbaRgg==
< access-control-allow-origin: *
< x-varnish: 24639517 26513520
< age: 279
< vary: Accept-Encoding,Origin
< x-cache: HIT
< accept-ranges: bytes
< x-envoy-upstream-service-time: 93
< expect-ct: report-uri="https://web-security-reports.services.atlassian.com/expect-ct-report/marketplace-athreos", max-age=86400
< strict-transport-security: max-age=63072000; preload
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< atl-traceid: fff065675d0c979f
< report-to: {"group": "endpoint-1", "max_age": 600, "endpoints": [{"url": "https://dj9s4kmieytgz.cloudfront.net"}], "include_subdomains": true}
< nel: {"report_to": "endpoint-1", "max_age": 600, "include_subdomains": true, "failure_fraction": 0.001}
<
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="google-site-verification" content="TCA8d7oPXhN4x93FWL3soth6MYZ39E8yAsdeNOUYTMs">
<meta name="google-site-verification" content="b1S05iZIuIoRG6cbYVJ2kPofZLMKfbTBuvxH25Pcxac">
<meta name="viewport" content="width=device-width,initial-scale=1,minimum-scale=1">
<meta name="botify-site-verification" content="aDhacERJ5Y2tZYqlQzIdMx8RgvQOJFu8">
<link rel="shortcut icon" href="/s/images/favicon.ico" type="image/x-icon">
<link rel="search" type="application/opensearchdescription+xml" href="s/public/opensearch.xml" title="Atlassian Apps Search">
<title>Atlassian Marketplace</title>

<style>
html, body {
margin: 0;
padding: 0;
}
.app-skeleton {
min-height:100vh;
background:linear-gradient(#0052CC 70px,white 1px);
}
</style>
<style id="critical-css"></style>

<!-- 546411f58d38e98441f4c052aef853e75bd7dfd2 Thu Jul 14 2022 10:44:40 GMT+0000 (Coordinated Universal Time) -->
<!-- Resource hits start-->

<link rel="preload" href="https://marketplace.atlassian.com/rest/internal/frontend/bootstrap" as="fetch" type="application/json" crossorigin="anonymous">

<link rel="preconnect" href="https://api.atlassian.com" crossorigin="anonymous">

<link rel="preconnect" href="https://cdn.contentful.com" crossorigin>
<link rel="preconnect" href="https://7LVR1JKTDF-dsn.algolia.net" crossorigin>

<link rel="dns-prefetch" href="//logx.optimizely.com" crossorigin>
<!-- Resource hits end-->

<link href="https://marketplace-cdn.atlassian.com/amkt-frontend-static/amkt.3f15d1be4a89ef9bac3b.css" rel="stylesheet"></head>
<body class="amkt app">
<!--[if lte IE 9 ]>
<div>For a better experience, update to Internet Explorer 11 or later, or use a different browser.</strong>. Your browser is incompatible, so things might not work properly.</div>
<![endif]-->
<div id="browser-support-banner"></div> <section id="amkt-frontend-content" role="main">
<div class="app-skeleton"></div>
</section>
<footer role="contentinfo">
<section class="logo-container">
<a class="bottom-logo" href="https://www.atlassian.com"></a>
<nav>
<ul>
<li>
<a href="/archive">
Archived apps
</a>
</li>
<li>
<a href="https://www.atlassian.com/legal/privacy-policy">
Privacy
</a>
</li>
<li>
<a href="https://www.atlassian.com/licensing/marketplace/termsofuse">
Terms of use
</a>
</li>
<li>
©
2022
Atlassian
</li>
</ul>
</nav>
</section>
</footer> <script src="https://atl-global.atlassian.com/js/atl-global.min.js"></script>

<script>
function isBot() {
// Pollinator (Selenium & Cypress) uses Chrome webdriver
return !!(navigator.webdriver ||
window.document.documentElement.getAttribute('webdriver') ||
window.callPhantom ||
window._phantom ||
(navigator.userAgent.indexOf('jsdom') > -1));
}
(function(i, s, o, g, r, a, m) {
i['GoogleAnalyticsObject'] = r;
i[r] = i[r] || function() {
(i[r].q = i[r].q || []).push(arguments);
}, i[r].l = 1 * new Date();
a = s.createElement(o),
m = s.getElementsByTagName(o)[0];
a.async = 1;
a.src=g;

if (isBot()) {
_ga = function(){};
return;
};

m.parentNode.insertBefore(a, m);
})(window, document, 'script', '//www.google-analytics.com/analytics.js', '_ga');

_ga('create', "UA-6032469-13", {
cookieDomain: '.atlassian.com',
legacyCookieDomain: '.atlassian.com'
});
_ga('set', 'anonymizeIp', true);
</script>
<script id="application-config">
window.applicationConfig = {
"environment": "production",
"adminHubBaseUrl": "https://admin.atlassian.com",
"algolia": {
"applicationId": "7LVR1JKTDF",
"apiKey": "3ef48f0fc67ee7e9c5a9dbbf6b45908c",
"environment": "prod"
},
"featureFlags": {
"apiKey": "2bc62244-9575-4f26-a903-fdea84c36471",
"environment": "prod"
},
"analytics": {
"aceAnalytics": {
"writeKey": "xjtlog1hqa",
"script": "https://atl-global.atlassian.com/js/atl-global.min.js"
},
"productAnalytics": {
"env": "prod"
},
"googleAnalyticsId": "UA-6032469-13"
},
"sentry": {
"dsn": "https://89802266c6e141b5aeb8eb73acd0ec77@o55978.ingest.sentry.io/5367237",
"environment": "production"
},
"metal": {
"id": "88fc286d-5db5-49f4-b48c-7f16b91d1675",
"env": "prod"
},
"enableIdentityHeartbeats": true,
"optimizelyKey": "8201120742",
"mpacBaseUrl": "https://marketplace.atlassian.com",
"clicktaleId": "ef81b94c-8498-4f12-b358-eb76a000a247",
"stargateBaseUrl": "https://api.atlassian.com",
"mpacCdnBaseUrl": "https://marketplace-cdn.atlassian.com",
"preferencesBaseUrl": "https://preferences.atlassian.com",
"graphQLGatewayUrl": "/gateway/api/graphql",
"macBaseUrl": "https://my.atlassian.com",
"wacBaseUrl": "https://www.atlassian.com",
"atlassianCommunityBaseUrl": "https://community.atlassian.com",
"jiraServiceDeskEmbedBaseUrl": "https://jsd-widget.atlassian.com"
}
</script>
<script id="initial-state"></script>

<script>
if (window.__INITIAL_STATE__ === undefined) {
window.applicationConfig.mpacBaseUrl = window.location.origin;
}
</script>
<script src="https://marketplace-cdn.atlassian.com/amkt-frontend-static/manifest.70d916a77c56a1c8b745.js"></script><script src="https://marketplace-cdn.atlassian.com/amkt-frontend-static/common-libs~amkt.6bf8e3f4bad5b17a5891.js"></script><script src="https://marketplace-cdn.atlassian.com/amkt-frontend-static/atlaskit-libs~amkt.917b8cf956bc71d7b3fc.js"></script><script src="https://marketplace-cdn.atlassian.com/amkt-frontend-static/vendor~amkt.fd848dc2199f7ede0148.js"></script><script src="https://marketplace-cdn.atlassian.com/amkt-frontend-static/amkt.688dcb67af925e3ad627.js"></script></body>
<script src="https://cdn.optimizely.com/js/8201120742.js" async></script>
<script src="https://cdn.cookielaw.org/consent/141bbadf-8293-4d74-9552-81ec52e16c66.js" charset="UTF-8" async></script>
<script>function OptanonWrapper() { }</script>
</html>
* Connection #0 to host marketplace.atlassian.com left intact
[hiadmin@x15855tzz ~]$

[Rashmi]

@Bastian Stehmann I have seen most of the cases on net where up gradation is done with greater than 8.16 or greater they have faced the same problem. But unfortunately I have not yet come accross any resolution of it.

[Bastian Stehmann]

Please raise an issue with the Atlassian support here https://support.atlassian.com/contact

If that is a known issue, they might have a solution.