Confluence on-prem server shutting down issue

[Vikramsinh Katkar]

Configured the confluence on-prem server and confluence db on EC2 instance using docker. Please see the below docker-compose.yml

version: '3'

services:
confluence:
image: atlassian/confluence:7.19.0
environment:
- DISABLE_NOTIFICATIONS=TRUE
- CONF_ARGS=-Datlassian.plugins.enable.wait=300
volumes:
- ./data:/opt/atlassian/confluence/data
ports:
- 8090:8090
environment:
- JVM_MINIMUM_MEMORY=1024m
- JVM_MAXIMUM_MEMORY=4096m
- JVM_RESERVED_CODE_CACHE_SIZE=512m
restart: always

confluencedb:
image: postgres:14
environment:
- POSTGRES_PASSWORD=secret
- POSTGRES_USER=confluence
- POSTGRES_DB=confluence
volumes:
- ./db:/var/lib/postgresql/data
ports:
- 5432:5432
restart: always

While running performance testing to insert 500K records(heavy operation) in confluence on-prem server using some script or when the confluence on-prem server is idle, I have observed the confluence on-prem server shutting down daily means when performing heavy operations on confluence on-prem server, the server always shuts down and gets the below error.

Out of memory issue

Exception in thread "I/O dispatcher 11" Exception in thread "I/O dispatcher 17" Exception in thread "I/O dispatcher 7" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
29-Oct-2022 00:08:58.709 SEVERE [Catalina-utility-4] org.apache.catalina.core.StandardServer.startPeriodicLifecycleEvent Error sending periodic event
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)

Despite increasing the heap memory of confluence on-prem server in docker-compose.yml file with - JVM_RESERVED_CODE_CACHE_SIZE=512m, the server is still shutting down.

DB connection pool issue

2-Nov-2022 04:48:43.442 SEVERE [http-nio-8090-exec-48] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is java.lang.IllegalStateException: DB connection pool is exhausted; obtaining a new DB connection for this request is disallowed.
at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:476)
at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doBegin(ConfluenceHibernateTransactionManager.java:41)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)

After booting up the containers everything works well but again When performing heavy operations on confluence on-prem server, the server shuts down.

It would be great if you could assist me with the following question to resolve the issue with Confluence on-prem server shutting down.

1. When performing heavy operations on confluence on-prem server and the server shuts down then how can we debug/check it?
2. What should be the system requirement to setup the confluence on-prem server and confluence db for running the heavy load?

[Vikramsinh Katkar]

@Atlassian Support  We are waiting for your response. Could you please look into it and guide me how can we resolve this issue.

[Tomasz Serafinski]

@Vikramsinh Katkar - were You able to figure out what was the problem?

[Jonas Khan]

@Tomasz Serafinski @Vikramsinh Katkar 

Could you please let us know how to find a solution to this? 

[Tomasz Serafinski]

Hi @Jonas Khan

in our case it was the system variables - this seems to work

# -- Defines any additional environment variables to be passed to the Confluence
# container. See https://hub.docker.com/r/atlassian/confluence for
# supported variables.
#
additionalEnvironmentVariables:
  - name: ATL_DB_POOLMAXSIZE
    value: '250'
  - name: ATL_DB_POOLMINSIZE
    value: '0'
  - name: ATL_DB_IDLETESTPERIOD
    value: '30000'
  - name: ATL_DB_VALIDATE
    value: "true"
  - name: CATALINA_OPTS
    value: "-Dorg.apache.tomcat.util.digester.PROPERTY_SOURCE=org.apache.tomcat.util.digester.EnvironmentPropertySource"