After Migration to New Server, Jira Crashes on Startup

[David Rhoderick]

I just moved Jira Software from a Bluehost CentOS server to a Digital Ocean CentOS droplet and every time I try to start it up, it "loads" (shows the loading graphic and steps and completes the process) and then immediately crashes (I get a blank page and when I refresh, there is nothing to connect to).  I first tried it with 1 CPU and 2 GB RAM (50 GB disk space so that isn't the problem) and then moved it to 2 CPUs with 2 GB RAM and I'm still getting GC Allocation Failures.  Here is my latest log:

Java HotSpot(TM) 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 7 2018 00:56:38 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 1882184k(1559320k free), swap 0k(0k free)
CommandLine flags: -XX:GCLogFileSize=20971520 -XX:InitialHeapSize=402653184 -XX:MaxHeapSize=1610612736 -XX:NumberOfGCLogFiles=5 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCCause -XX:+PrintGCDateStamps$
2019-04-11T22:14:59.088+0000: 4.360: [GC (Allocation Failure) [PSYoungGen: 98304K->11455K(114688K)] 98304K->11463K(376832K), 0.0415028 secs] [Times: user=0.07 sys=0.01, real=0.04 secs]
2019-04-11T22:15:00.786+0000: 6.059: [GC (Allocation Failure) [PSYoungGen: 109759K->14422K(114688K)] 109767K->14430K(376832K), 0.0678371 secs] [Times: user=0.06 sys=0.01, real=0.07 secs]
2019-04-11T22:15:02.933+0000: 8.205: [GC (Allocation Failure) [PSYoungGen: 112726K->14846K(114688K)] 112734K->14862K(376832K), 0.0493970 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
2019-04-11T22:15:04.978+0000: 10.250: [GC (Allocation Failure) [PSYoungGen: 113150K->15749K(212992K)] 113166K->15765K(475136K), 0.0368889 secs] [Times: user=0.04 sys=0.01, real=0.04 secs]
2019-04-11T22:15:08.675+0000: 13.948: [GC (Allocation Failure) [PSYoungGen: 212357K->16369K(212992K)] 212373K->18341K(475136K), 0.0622102 secs] [Times: user=0.05 sys=0.02, real=0.06 secs]
2019-04-11T22:15:13.290+0000: 18.562: [GC (Allocation Failure) [PSYoungGen: 212977K->19669K(411136K)] 214949K->21649K(673280K), 0.0709272 secs] [Times: user=0.10 sys=0.02, real=0.08 secs]
2019-04-11T22:15:17.812+0000: 23.084: [GC (Allocation Failure) [PSYoungGen: 410837K->19635K(413184K)] 412817K->33574K(675328K), 0.0703384 secs] [Times: user=0.07 sys=0.02, real=0.07 secs]
2019-04-11T22:15:18.546+0000: 23.819: [GC (Metadata GC Threshold) [PSYoungGen: 108925K->8926K(500224K)] 122864K->23168K(762368K), 0.0546500 secs] [Times: user=0.08 sys=0.01, real=0.06 secs]
2019-04-11T22:15:18.601+0000: 23.873: [Full GC (Metadata GC Threshold) [PSYoungGen: 8926K->0K(500224K)] [ParOldGen: 14242K->20700K(262144K)] 23168K->20700K(762368K), [Metaspace: 20858K->20858K(1069056K)], 0.13273$
2019-04-11T22:15:23.140+0000: 28.412: [GC (System.gc()) [PSYoungGen: 354596K->15426K(493056K)] 375297K->36135K(755200K), 0.0460703 secs] [Times: user=0.05 sys=0.02, real=0.04 secs]
2019-04-11T22:15:23.186+0000: 28.459: [Full GC (System.gc()) [PSYoungGen: 15426K->0K(493056K)] [ParOldGen: 20708K->32511K(262144K)] 36135K->32511K(755200K), [Metaspace: 32196K->32196K(1079296K)], 0.1460707 secs] $
2019-04-11T22:15:25.855+0000: 31.127: [GC (Allocation Failure) [PSYoungGen: 477184K->11325K(500224K)] 509695K->43845K(762368K), 0.0225828 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2019-04-11T22:15:27.088+0000: 32.360: [GC (Metadata GC Threshold) [PSYoungGen: 230204K->6011K(500224K)] 262723K->43359K(762368K), 0.1326186 secs] [Times: user=0.12 sys=0.01, real=0.38 secs]
2019-04-11T22:15:27.221+0000: 32.493: [Full GC (Metadata GC Threshold) [PSYoungGen: 6011K->0K(500224K)] [ParOldGen: 37347K->36864K(262144K)] 43359K->36864K(762368K), [Metaspace: 53996K->53996K(1097728K)], 0.24559$
2019-04-11T22:15:30.787+0000: 36.059: [GC (Allocation Failure) [PSYoungGen: 477696K->15550K(501248K)] 514560K->52414K(763392K), 0.0651511 secs] [Times: user=0.06 sys=0.01, real=0.07 secs]
2019-04-11T22:15:34.616+0000: 39.888: [GC (Allocation Failure) [PSYoungGen: 493246K->10467K(500736K)] 530110K->54352K(762880K), 0.0643368 secs] [Times: user=0.12 sys=0.00, real=0.07 secs]
2019-04-11T22:15:38.762+0000: 44.034: [GC (Allocation Failure) [PSYoungGen: 488163K->22588K(497152K)] 532048K->72723K(759296K), 0.1025614 secs] [Times: user=0.18 sys=0.01, real=0.10 secs]
2019-04-11T22:15:42.045+0000: 47.318: [GC (Allocation Failure) [PSYoungGen: 496700K->25058K(499200K)] 546835K->86417K(761344K), 0.0623828 secs] [Times: user=0.10 sys=0.01, real=0.06 secs]
2019-04-11T22:15:42.950+0000: 48.222: [GC (Allocation Failure) [PSYoungGen: 499170K->9868K(485888K)] 560529K->81683K(748032K), 0.0413956 secs] [Times: user=0.06 sys=0.01, real=0.04 secs]
2019-04-11T22:15:46.134+0000: 51.406: [GC (Allocation Failure) [PSYoungGen: 470668K->22800K(483840K)] 542483K->99322K(745984K), 0.0382481 secs] [Times: user=0.06 sys=0.00, real=0.04 secs]
2019-04-11T22:15:49.960+0000: 55.232: [GC (Allocation Failure) [PSYoungGen: 483600K->32742K(475136K)] 560122K->119158K(737280K), 0.0704310 secs] [Times: user=0.11 sys=0.01, real=0.08 secs]
2019-04-11T22:15:52.007+0000: 57.279: [GC (Metadata GC Threshold) [PSYoungGen: 280138K->16592K(483328K)] 366554K->132628K(745472K), 0.1228881 secs] [Times: user=0.20 sys=0.02, real=0.12 secs]
2019-04-11T22:15:52.130+0000: 57.402: [Full GC (Metadata GC Threshold) [PSYoungGen: 16592K->0K(483328K)] [ParOldGen: 116036K->116920K(345088K)] 132628K->116920K(828416K), [Metaspace: 87219K->87211K(1128448K)], 0.$
2019-04-11T22:15:55.898+0000: 61.170: [GC (GCLocker Initiated GC) [PSYoungGen: 442368K->25394K(482304K)] 560401K->143435K(827392K), 0.0342574 secs] [Times: user=0.06 sys=0.00, real=0.04 secs]
2019-04-11T22:15:57.580+0000: 62.852: [GC (Allocation Failure) [PSYoungGen: 469810K->28227K(483840K)] 587886K->157471K(828928K), 0.0612612 secs] [Times: user=0.11 sys=0.01, real=0.07 secs]
2019-04-11T22:15:58.996+0000: 64.268: [GC (Allocation Failure) [PSYoungGen: 472643K->21906K(482816K)] 601887K->159935K(827904K), 0.0792672 secs] [Times: user=0.15 sys=0.00, real=0.08 secs]
2019-04-11T22:16:00.384+0000: 65.656: [GC (Allocation Failure) [PSYoungGen: 466322K->20147K(483328K)] 604351K->169261K(828416K), 0.0812310 secs] [Times: user=0.15 sys=0.00, real=0.08 secs]
2019-04-11T22:16:01.564+0000: 66.836: [GC (Allocation Failure) [PSYoungGen: 464563K->12592K(484352K)] 613685K->168802K(829440K), 0.1106257 secs] [Times: user=0.20 sys=0.00, real=0.11 secs]
2019-04-11T22:16:01.738+0000: 67.011: [GC (GCLocker Initiated GC) [PSYoungGen: 20078K->192K(484352K)] 176288K->166512K(829440K), 0.1000656 secs] [Times: user=0.12 sys=0.03, real=0.10 secs]
2019-04-11T22:16:04.896+0000: 70.168: [GC (Allocation Failure) [PSYoungGen: 447168K->23370K(485376K)] 613488K->189823K(830464K), 0.0698068 secs] [Times: user=0.07 sys=0.00, real=0.07 secs]
2019-04-11T22:16:07.237+0000: 72.509: [GC (Allocation Failure) [PSYoungGen: 469834K->38394K(484864K)] 636287K->212770K(829952K), 0.0861054 secs] [Times: user=0.12 sys=0.01, real=0.09 secs]
2019-04-11T22:16:09.791+0000: 75.063: [GC (Allocation Failure) [PSYoungGen: 484858K->23747K(477696K)] 659234K->211448K(822784K), 0.1100200 secs] [Times: user=0.18 sys=0.01, real=0.11 secs]
2019-04-11T22:16:12.244+0000: 77.517: [GC (Allocation Failure) [PSYoungGen: 462019K->22476K(481280K)] 649720K->222346K(826368K), 0.0629436 secs] [Times: user=0.09 sys=0.02, real=0.06 secs]
2019-04-11T22:16:13.489+0000: 78.762: [GC (Metadata GC Threshold) [PSYoungGen: 179268K->6324K(480768K)] 379138K->214736K(825856K), 0.0460246 secs] [Times: user=0.07 sys=0.00, real=0.05 secs]
2019-04-11T22:16:13.536+0000: 78.808: [Full GC (Metadata GC Threshold) [PSYoungGen: 6324K->0K(480768K)] [ParOldGen: 208411K->184224K(447488K)] 214736K->184224K(928256K), [Metaspace: 139105K->139101K(1179648K)], 0$
2019-04-11T22:16:16.864+0000: 82.136: [GC (Allocation Failure) [PSYoungGen: 439808K->22837K(481792K)] 624032K->207070K(929280K), 0.0515943 secs] [Times: user=0.10 sys=0.00, real=0.05 secs]
2019-04-11T22:16:19.113+0000: 84.385: [GC (Allocation Failure) [PSYoungGen: 462645K->14664K(480256K)] 646878K->213104K(927744K), 0.1006326 secs] [Times: user=0.20 sys=0.00, real=0.10 secs]
2019-04-11T22:16:21.411+0000: 86.684: [GC (Allocation Failure) [PSYoungGen: 456008K->19560K(481280K)] 654448K->229095K(928768K), 0.0859468 secs] [Times: user=0.16 sys=0.00, real=0.08 secs]
2019-04-11T22:16:25.496+0000: 90.768: [GC (Allocation Failure) [PSYoungGen: 460904K->25688K(482816K)] 670439K->249447K(930304K), 0.1002318 secs] [Times: user=0.17 sys=0.00, real=0.10 secs]
2019-04-11T22:16:28.284+0000: 93.556: [GC (Allocation Failure) [PSYoungGen: 470616K->16069K(483328K)] 694375K->260361K(930816K), 0.1073145 secs] [Times: user=0.17 sys=0.02, real=0.11 secs]
2019-04-11T22:16:32.410+0000: 97.682: [GC (Allocation Failure) [PSYoungGen: 460997K->11386K(485376K)] 705289K->268080K(932864K), 0.0605184 secs] [Times: user=0.09 sys=0.02, real=0.07 secs]
2019-04-11T22:16:35.330+0000: 100.602: [GC (Allocation Failure) [PSYoungGen: 459898K->12208K(485376K)] 716592K->277754K(932864K), 0.0668590 secs] [Times: user=0.10 sys=0.01, real=0.07 secs]
2019-04-11T22:16:38.577+0000: 103.850: [GC (Allocation Failure) [PSYoungGen: 460720K->6133K(486912K)] 726266K->280852K(934400K), 0.0834794 secs] [Times: user=0.15 sys=0.01, real=0.09 secs]
2019-04-11T22:16:41.399+0000: 106.671: [GC (Allocation Failure) [PSYoungGen: 456693K->5487K(486912K)] 731412K->284726K(934400K), 0.0322785 secs] [Times: user=0.05 sys=0.00, real=0.04 secs]
2019-04-11T22:16:46.969+0000: 112.241: [GC (Allocation Failure) [PSYoungGen: 456047K->25771K(484864K)] 735286K->305011K(932352K), 0.0821281 secs] [Times: user=0.17 sys=0.00, real=0.09 secs]
2019-04-11T22:16:49.079+0000: 114.351: [GC (Allocation Failure) [PSYoungGen: 474283K->18748K(486400K)] 753523K->297996K(933888K), 0.0736444 secs] [Times: user=0.15 sys=0.01, real=0.08 secs]
2019-04-11T22:16:50.857+0000: 116.129: [GC (Allocation Failure) [PSYoungGen: 467260K->24669K(486400K)] 746508K->303917K(933888K), 0.0530683 secs] [Times: user=0.11 sys=0.00, real=0.05 secs]
2019-04-11T22:16:56.212+0000: 121.484: [GC (Allocation Failure) [PSYoungGen: 474205K->37355K(486912K)] 753453K->338172K(934400K), 0.1215715 secs] [Times: user=0.18 sys=0.02, real=0.12 secs]
2019-04-11T22:16:59.439+0000: 124.711: [GC (Allocation Failure) [PSYoungGen: 486891K->37351K(429568K)] 787708K->371669K(877056K), 0.1785601 secs] [Times: user=0.26 sys=0.04, real=0.18 secs]
2019-04-11T22:17:01.735+0000: 127.007: [GC (Allocation Failure) [PSYoungGen: 429543K->58945K(451584K)] 763861K->399464K(899072K), 0.1321743 secs] [Times: user=0.22 sys=0.01, real=0.13 secs]
2019-04-11T22:17:03.741+0000: 129.013: [GC (Allocation Failure) [PSYoungGen: 451137K->29134K(454144K)] 791656K->399760K(901632K), 0.2035832 secs] [Times: user=0.33 sys=0.02, real=0.20 secs]
2019-04-11T22:17:05.703+0000: 130.975: [GC (Allocation Failure) [PSYoungGen: 410574K->28999K(410624K)] 781200K->420699K(858112K), 0.1174105 secs] [Times: user=0.19 sys=0.02, real=0.12 secs]
2019-04-11T22:17:07.665+0000: 132.937: [GC (Allocation Failure) [PSYoungGen: 410439K->25517K(450560K)] 802139K->431503K(898048K), 0.2072922 secs] [Times: user=0.34 sys=0.02, real=0.21 secs]
2019-04-11T22:17:09.485+0000: 134.758: [GC (Allocation Failure) [PSYoungGen: 410029K->9930K(447488K)] 816015K->428636K(894976K), 0.0998661 secs] [Times: user=0.13 sys=0.02, real=0.10 secs]
2019-04-11T22:17:09.585+0000: 134.858: [Full GC (Ergonomics)

I started the migration following the steps at https://confluence.atlassian.com/adminjiraserver/migrating-jira-applications-to-another-server-938846962.html only when I got to the point where I upload the XML backup, it crashed.  Then the next time, after restarting Jira, I was asked to provide my license (no XML backup upload option) and then it crashed.  The time after that, after restarting Jira, I was asked again for my license and then to create an admin account and then it crashed, and now it just "loads" and crashes.

My old server did have a different hostname (of course, since I didn't get to the point where I can change it or feel secure enough to point my old domain to the new one) and it was on port 80, but that shouldn't matter because I was able to access my new server by IP address and on port 8080 (I should be able to change the hostname later).

One thought I had is because Digital Ocean droplets are so bare-bones (I had to install nano), perhaps I'm missing some crucial utilities.  I did install and configure MySQL as best I can according to the documentation but maybe I'm missing something else?

Let me know if there is anything else I can provide.  I also could switch to a different distribution of Linux if it's easier (perhaps specifically for Digital Ocean).  I'd prefer to avoid migrating to Windows for various reasons, even if it is easier (not sure it is).

This is the second time I've tried this migration so I'd really like a little assistance now.

EDIT: As far as the complexity of my Jira server, there are quite a few projects in it but not many users and I primarily use it with just a handful of developers, so I shouldn't need more than the bare minimum requirements.  I had to move off of Bluehost because Jira was working but crashing almost every day because I have a 2 CPU + 2 GB RAM VPS there that's also hosting between 10 and 20 WordPress sites, so the fact that it worked there at all means it should be fine on its own in this new droplet.

[David Rhoderick · Accepted solution]

Rebuilt it on an Ubuntu droplet using the tutorial at  https://community.spiceworks.com/how_to/62855-hosted-jira-in-15-minutes-for-20-per-month-on-digital-ocean-updated and it is much better.

[Andy Heinzer]

Thanks for circling back to this and accepting your answer.  

[Andy Heinzer]

Hi,

Sorry to hear that you have been having so many crashes with Jira lately.  The fact that you are seeing a large number of GC allocation errors tends to mean that either JVM heap is inappropriately sized and/or the system is under resourced possibly.  However I don't really have enough info yet you tell you which it is.

I would recommend checking out the Jira Sizing Guide.  This is a helpful document to guide in you regards to the overall system resources as well as the resources we can expect Jira to use on it's own.  When you say Jira has 2GB, do you mean that the operating system has 2GB of ram?  Or that the JVM parameter of -Xmx is set to 2g or 2048m?   I would expect even small Jira sites to have at least 1 GB of memory in the JVM, but the system itself should have at least 8GB of memory for the overhead of other system processes.

We might also have to take other environmental factors into consideration here. Such as where the database server is?, what other applications are running on this same machine as Jira?, what plugins is Jira using?   All of these kinds of factors can influence the amount of system memory needed for the operating system, and plugins in Jira can also directly affect the JVM usage of memory as well.  

New Jira installations tend to have too small of a Xmx parameter for importing data from other sites.  If that happens, it could explain the gc allocation errors you have seen so far.  There are steps you can take to Increase the Jira application memory.  However I would be interested to know more about the system memory available before trying to increase this.

Also curious to learn...

1. What version of Jira is this? 
2. What version were you running before this migration?

[David Rhoderick]

Hey Andrew,

Thanks for getting back to me.  After so much struggling, your response is a bit late for this issue and I ended up fixing it by reinstalling following the tutorial to install it on Ubuntu instead of CentOS via the link I mentioned before (but using the version of Jira I had before and updating the JDBC to the latest version) but I think your response has a lot of interesting information.

Thanks for linking to the Sizing Guide, I will definitely check that out, especially if I'm having issues with my new installation because the old installation was crashing almost daily (sometimes not but making up for it by crashing two or three times other days).  I'm not sure what the old JVM memory was set to but the new one has now been set to 1024MB so I'll see if it's necessary to increase it but for now, the server it is on has 2GB total, so I should be able to increase it as necessary.  I'm sure the plugins add to the demand of the installation as well (mostly just using Tempo but I'm sure that has an impact).

I'm using the same server for database and application at the moment and did it before as well.  The old version had many WordPress installations on it and both Jira and the WordPress sites were failing regularly, which is why I needed to migrate the server.  Hopefully, now it will be fixed.

Just to answer questions, this is Jira Software version 7.12.3 and the old version was the exact same version (I saw it was important to match version, add-ons, and custom fields etc.).