HTTP Error 502.3 - Bad Gateway

[Noe Masis]

I have been getting the following error on my confluence server:

HTTP Error 502.3 - Bad Gateway

I have restarted server, restarted services, made sure IIS is running but no luck.

[Andy Heinzer]

Hi Noe,

I understand that you're using Confluence server and seeing this error message.  However this error message comes from IIS.  Perhaps you're using this as a proxy for Confluence.  Unfortunately, it doesn't tell us enough to understand the cause of the problem, just that Confluence did not respond in time.

This could happen if Confluence is not started, or perhaps it crashed, or perhaps it's just too busy to respond to requests or that the configuration between Confluence and this proxy is misconfigured.

Is Confluence setup to run as a Windows Service?

If so, then I'd want to check the windows services to see what the current state of that service is. 

If the problems persist after Confluence has been started, then I would want to take a closer look at the atlassian-confluence.log file from this startup attempt time.  If there is some error that prevented Confluence from starting, that log should have some more details there that could help us understand why it's not working.

Regards,

Andy

[Noe Masis]

Confluence is running as a Windows Service and the state is running. I'm unable to access Confluence through localhost:8090 (without the proxy) so it is not the proxy. 

I checked out atlassian-confluence log after restarting the server and got this:

2020-04-14 09:16:06,587 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] earlyStartup Plugin system earlyStartup begun
2020-04-14 09:16:07,485 WARN [localhost-startStop-1] [org.twdata.pkgscanner.ExportPackageListBuilder] logDuplicateWarning Package Scanner found duplicates for package 'javax.annotation' with different versions. Files: javax.annotation-api-1.3.2.jar and jsr305-3.0.1.jar
'C:\Program Files\Atlassian\Confluence\confluence\WEB-INF\lib\javax.annotation-api-1.3.2.jar'
'C:\Program Files\Atlassian\Confluence\confluence\WEB-INF\lib\jsr305-3.0.1.jar'
2020-04-14 09:16:13,063 WARN [FelixStartLevel] [extender.internal.support.ExtenderConfiguration] addDefaultDependencyFactories Gemini Blueprint extensions bundle not present, annotation processing disabled.
2020-04-14 09:17:08,923 WARN [ThreadPoolAsyncTaskExecutor::Thread 32] [spring.scanner.util.ProductFilterUtil] detectProduct Couldn't detect product, will use ProductFilter.ALL
2020-04-14 09:18:38,920 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.15.2 [build 8100 based on commit hash 4410012ac87e845516b70bc69b6f7a893eabaa5a] - synchrony version 2.1.0-master-9d112c9d
2020-04-14 09:18:49,660 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
2020-04-14 09:18:59,316 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager
2020-04-14 09:19:32,316 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization completed in 42656 ms
2020-04-14 09:19:44,664 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] earlyStartup Plugin system earlyStartup begun
2020-04-14 09:19:45,915 WARN [localhost-startStop-1] [org.twdata.pkgscanner.ExportPackageListBuilder] logDuplicateWarning Package Scanner found duplicates for package 'javax.annotation' with different versions. Files: javax.annotation-api-1.3.2.jar and jsr305-3.0.1.jar
'C:\Program Files\Atlassian\Confluence\confluence\WEB-INF\lib\javax.annotation-api-1.3.2.jar'
'C:\Program Files\Atlassian\Confluence\confluence\WEB-INF\lib\jsr305-3.0.1.jar'
2020-04-14 09:19:52,306 WARN [FelixStartLevel] [extender.internal.support.ExtenderConfiguration] addDefaultDependencyFactories Gemini Blueprint extensions bundle not present, annotation processing disabled.

I also see alot of this too:

2020-04-14 09:11:49,625 WARN [C3P0PooledConnectionPoolManager[identityToken->2wlq94a9xifxde1unltyh|6d7ea39b]-AdminTaskTimer] [mchange.v2.async.ThreadPoolAsynchronousRunner] log Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@1c3ed4f8 (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
2020-04-14 09:11:49,625 WARN [C3P0PooledConnectionPoolManager[identityToken->2wlq94a9xifxde1unltyh|6d7ea39b]-AdminTaskTimer] [mchange.v2.async.ThreadPoolAsynchronousRunner] log Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@df52813 (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
2020-04-14 09:11:49,625 WARN [C3P0PooledConnectionPoolManager[identityToken->2wlq94a9xifxde1unltyh|6d7ea39b]-AdminTaskTimer] [mchange.v2.async.ThreadPoolAsynchronousRunner] log Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@62cccf42 (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
2020-04-14 09:11:51,407 WARN [C3P0PooledConnectionPoolManager[identityToken->2wlq94a9xifxde1unltyh|64c72df7]-AdminTaskTimer] [mchange.v2.async.ThreadPoolAsynchronousRunner] log Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@798ae35c (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
2020-04-14 09:11:51,407 WARN [C3P0PooledConnectionPoolManager[identityToken->2wlq94a9xifxde1unltyh|64c72df7]-AdminTaskTimer] [mchange.v2.async.ThreadPoolAsynchronousRunner] log Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@685bc590 (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
2020-04-14 09:11:51,407 WARN [C3P0PooledConnectionPoolManager[identityToken->2wlq94a9xifxde1unltyh|64c72df7]-AdminTaskTimer] [mchange.v2.async.ThreadPoolAsynchronousRunner] log Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@7b8fd8da (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
2020-04-14 09:13:09,750 WARN [C3P0PooledConnectionPoolManager[identityToken->2wlq94a9xifxde1unltyh|6d7ea39b]-AdminTaskTimer] [mchange.v2.async.ThreadPoolAsynchronousRunner] log com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1356da05 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2020-04-14 09:13:09,750 WARN [C3P0PooledConnectionPoolManager[identityToken->2wlq94a9xifxde1unltyh|6d7ea39b]-AdminTaskTimer] [mchange.v2.async.ThreadPoolAsynchronousRunner] log com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1356da05 -- APPARENT DEADLOCK!!! Complete Status:

[Noe Masis]

14-Apr-2020 11:45:58.520 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
14-Apr-2020 11:45:59.066 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
14-Apr-2020 11:45:59.129 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 4202 ms
14-Apr-2020 11:45:59.191 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]
14-Apr-2020 11:45:59.191 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/9.0.12
14-Apr-2020 11:46:04.696 INFO [localhost-startStop-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
14-Apr-2020 11:46:05.336 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@6bf48724]
14-Apr-2020 11:46:05.711 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'
14-Apr-2020 11:46:05.711 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started
14-Apr-2020 11:46:05.727 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Tue Apr 14 11:46:05 EDT 2020]; root of context hierarchy
14-Apr-2020 11:46:05.790 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: [class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig]
14-Apr-2020 11:46:06.587 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService 'defaultSockJsTaskScheduler'
14-Apr-2020 11:46:06.649 INFO [localhost-startStop-2] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path [/v1/bayeux-sync1] onto handler of type [class org.springframework.web.socket.server.support.WebSocketHttpRequestHandler]
14-Apr-2020 11:46:06.852 INFO [localhost-startStop-2] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler]
14-Apr-2020 11:46:07.321 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date [Tue Apr 14 11:46:05 EDT 2020]; root of context hierarchy
14-Apr-2020 11:46:07.415 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)
14-Apr-2020 11:46:07.415 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()
14-Apr-2020 11:46:07.555 INFO [localhost-startStop-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647
14-Apr-2020 11:46:07.587 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1876 ms
14-Apr-2020 11:56:15.687 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
14-Apr-2020 11:56:16.625 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
14-Apr-2020 11:56:18.187 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.
14-Apr-2020 11:56:18.187 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
14-Apr-2020 11:56:18.343 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
14-Apr-2020 11:56:18.531 WARNING [main] org.apache.catalina.core.StandardContext.setPath A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [${confluence.context.path}/synchrony-proxy] does not meet these criteria and has been changed to [/${confluence.context.path}/synchrony-proxy]
14-Apr-2020 11:56:18.531 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.

Getting these warnings on catalina log file. Any advise?

[Andy Heinzer]

Hi Noe,

I can see that you're getting an deadlock error message.  Given the location of that error, it looks like you might be affected by this problem Frequent APPARENT DEADLOCK messages appearing in the Confluence logs.  That KB has some steps you can take to make modifications to the <confluence-home>/confluence.cfg.xml

This file determines how Confluence connects to your database. Try these steps first and see if this helps to get Confluence started again.

If this doesn't help, then the next steps I would be interested to see would be to try to start Confluence with add-ons disabled.  Steps to do that are in Start and Stop Confluence: With add-ons disabled (for troubleshooting)....

Please let us know the results.

Andy

[Noe Masis]

Hey Andy,

Ended up being the RAM memory getting full. Thanks for your response.