Confluence Server: StuckThread after Upgrade to 7.0.1

[Trevor Downing]

We run Confluence Server on 2012 R2 and it was running smoothly until we upgraded from 6.15.8 to 7.0.1.

After the upgrade we encountered the following issues:

Resolved: The top bar was failing to load and in it's place was code. We followed an article about reloading plugins and it resolved the issue for us.

https://confluence.atlassian.com/confkb/how-to-reset-all-confluence-plugins-back-to-their-default-state-through-the-database-859509873.html

Unresolved: when Confluence was started back up and I was at the login screen, everything seemed to be running smoothly. I could browser our public articles and there would be no irregular delays loading the page. However, after I logged in, I noticed significant delay in navigating from page to page (over a minute each time).

I started looking around in the logs and found a few different logs but I was unable to ascertain what the connection between the logs are or what underlying issue is causing the delays.

Here are some of the logs:

From {service-name}-stderr.{today's-date}.log in confluence\logs\

org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [https-jsse-nio2-8443-exec-1] (id=[15]) has been active for [62,062] milliseconds (since [9/30/19 8:46 AM]) to serve the same request for [{base-url}] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.

org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [https-jsse-nio2-8443-exec-1] (id=[15]) was previously reported to be stuck but has completed. It was active for approximately [360,832] milliseconds.

From Confluence\Application Data\Confluence\Logs\atlassian-confluence.log:

ERROR [https-jsse-nio2-8443-exec-9] [renderer.internal.http.HttpClientFetcher] fetch Unable to perform a request to: {base-url}/rest/gadgets/1.0/g/messagebundle/en_US/gadget.common%2Cgadget.confluence
-- referer: {base-URL}/display/KB/{article-name} | url: /plugins/macrobrowser/browse-macros.action | traceId: ---- | userName: ----- | action: browse-macros
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

[oauth.serviceprovider.internal.AuthenticatorImpl] logOAuthProblem Problem encountered authenticating OAuth client for url "{base-URL}/rest/knowledge-base/1.0/search/KB", error was "consumer_key_unknown", with parameters "{oauth_problem=consumer_key_unknown}"

I'm unsure which of these error logs is most significant and I'm not sure where to go from here. I appreciate advice you have to offer!

[Trevor Downing]

Update on this: we rolled the server / DB back to a 6.x version that was working for us. I'm still searching for advice because we want to be able to upgrade, but we don't know what was causing the stuck thread.

Thank you for anyone who offers advice on this.

[Justin Parker]

I get those errors also and I am on Confluence 7.0.1 (upgraded from 6.x) and below is my env info. Not sure if you are running the same minus the Linux OS in VMware Fusion.

Have you tried running a diagnostic and look in the /var/atlassian/application-data/confluence/logs/atlassian-diagnostics.log file?

grep -i stuck /var/atlassian/application-data/confluence/logs/atlassian-confluence.log -c
11 counts

uname -a
Linux localhost.localdomain 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

rpm -qa | grep mysql

mysql-community-common-5.7.27-1.el7.x86_64
mysql-community-client-5.7.27-1.el7.x86_64
mysql-community-libs-compat-5.7.27-1.el7.x86_64
mysql-community-libs-5.7.27-1.el7.x86_64
mysql-community-server-5.7.27-1.el7.x86_64

cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

cat /proc/cpuinfo

rocessor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 94
model name : Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
stepping : 3
microcode : 0xcc
cpu MHz : 2591.775
cache size : 6144 KB

cat /proc/meminfo

MemTotal: 1863236 kB
MemFree: 68260 kB
MemAvailable: 22876 kB
Buffers: 1496 kB
Cached: 28068 kB
SwapCached: 539416 kB
Active: 1236028 kB
Inactive: 426012 kB
Active(anon): 1220520 kB
Inactive(anon): 412600 kB
Active(file): 15508 kB
Inactive(file): 13412 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 3905532 kB
SwapFree: 2399740 kB
Dirty: 28 kB
Writeback: 0 kB
AnonPages: 1606888 kB
Mapped: 14456 kB
Shmem: 660 kB
Slab: 67904 kB
SReclaimable: 29868 kB
SUnreclaim: 38036 kB
KernelStack: 8368 kB
PageTables: 9892 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4837148 kB
Committed_AS: 3218280 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 179620 kB
VmallocChunk: 34359310332 kB
HardwareCorrupted: 0 kB
AnonHugePages: 270336 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 75648 kB
DirectMap2M: 2021376 kB
DirectMap1G: 0 kB

[Trevor Downing]

Hi Parker,

Thank you for your response. I haven't ran the diagnostics, I'm not sure how and it is very difficult to navigate to anything inside the web-service right now because it won't load half the time and if it does it may take multiple minutes.

I tried to go to the troubleshooting tools section and there was a popup on the right side of the screen that said:

"Your URL doesn't match

Confluence's base URL is set to "{base-url}:{port}" but you are accessing Confluence from "{base-url}:{port}undefined"

It's the exact same url with the word "undefined" at the end. I don't know if this is relevant but I thought I would share it anyways.

In Troubleshooting and Support Tools it shows the Instance health checks and everything has a passing checkmark. I will try to find the diagnostics you mentioned running.

[Trevor Downing]

I was reading some more and I found the database latency for me is very high.

Under System Information it says: Database Latency: 48ms

I read this should be below 2ms so I'm looking for ways to troubleshoot this.

[Trevor Downing]

Follow up: The DB Latency metric usually appears low 0-10 but I occasionally find it higher like when I found it at 48ms.

I also found the following two entries in the logs so I'm wondering if this could be related:

02-Oct-2019 10:28:16.515 WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

02-Oct-2019 10:28:16.530 WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->TOKEN
HelperThread-#0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

02-Oct-2019 10:32:23.197 SEVERE [https-jsse-nio2-8443-exec-6] org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor.failed Socket accept failed
java.io.IOException: The specified network name is no longer available.

I'm wondering if these offer any insight about the issue I'm facing...
I ran a thread dump but I'm not sure how to interpret the results or what to look for.

Thank you for any help!

[Justin Parker]

What database and version are you running? What JDBC driver do you have running?

Is the database running on the same server as Confluence?

[Trevor Downing]

Microsoft SQL Server

12.00.5687

Driver version: 6.3.0.0

The DB is running on a different server than Confluence.