Relation "bandana" does not exist when using official Confluence docker image

[jayhding]

Environments

• Official Docker image atlassian/confluence-server:7.0.1
• Install via in-house helm charts to Google Kubernetes Engine
• GCE persistent disk is mounted as /var/atlassian/application-data/confluence/data/
• Google CloudSQL is used as database server, with PostgreSQL version 9.6
  • Access was via sqlproxy
  • Separate database and user were created. The confluence db user is the owner of the confluence database
• Evaluation license was used
• Following environment variables are set
  • CONFLUENCE_HOME: /var/atlassian/application-data/confluence/data/
  • ATL_DB_TYPE: postgresql 
  • ATL_JDBC_URL: jdbc:postgresql://***-gcloud-sqlproxy.sqlproxy:5432/confluence-**
  • ATL_JDBC_USER
  • ATL_JDBC_PASSWORD

Before Entering Setup

I can see from /var/atlassian/application-data/confluence/data/logs/atlassian-confluence.log and it looks good

2019-10-11 05:02:35,328 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.0.1 [build 8202 based on commit hash 6ec4be8a9be53dc4390b2cd3f9a4c5731ee93a77] - synchrony version 3.1.0-master-8becbfc8
2019-10-11 05:02:43,716 WARN [Catalina-utility-1] [com.atlassian.config.ApplicationConfig] setApplicationHome Application home does not exist. Creating directory: /var/atlassian/application-data/confluence/data/shared-home
2019-10-11 05:02:43,727 INFO [Catalina-utility-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$saveSetupConfigIntoSharedHome$9 Writing setup configuration into shared home...
2019-10-11 05:02:43,749 INFO [Catalina-utility-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$saveSetupConfigIntoSharedHome$9 Finished writing setup configuration into shared home
2019-10-11 05:02:47,459 INFO [Catalina-utility-1] [atlassian.plugin.manager.DefaultPluginManager] lambda$earlyStartup$3 Plugin system earlyStartup begun
2019-10-11 05:02:52,487 WARN [FelixStartLevel] [extender.internal.support.ExtenderConfiguration] addDefaultDependencyFactories Gemini Blueprint extensions bundle not present, annotation processing disabled.
2019-10-11 05:02:56,435 INFO [Catalina-utility-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system earlyStartup ended
2019-10-11 05:02:56,443 INFO [Catalina-utility-1] [atlassian.plugin.manager.DefaultPluginManager] lambda$lateStartup$4 Plugin system lateStartup begun
2019-10-11 05:02:59,912 INFO [Catalina-utility-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system lateStartup ended
2019-10-11 05:03:01,154 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve

Problem Description

After inputing license, the page returned 500 error and atlassian-confluence.log showed:

2019-10-11 05:11:18,704 ERROR [http-nio-8090-exec-6] [engine.jdbc.spi.SqlExceptionHelper] logExceptions ERROR: relation "bandana" does not exist
Position: 181
-- referer: https://xxxxxx/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 0096d582876d148d | userName: anon
ymous

2019-10-11 05:11:18,853 ERROR [http-nio-8090-exec-6] [atlassian.confluence.servlet.ConfluenceServletDispatcher] sendError Could not execute action
-- referer: https://xxxxxx/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 0096d582876d148d | userName: anonymous
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'i18NBean' defined in class path resource [i18NContext.xml]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.confluence.util.i18n.I18NBean]: Factory method 'getI18NBean' threw exception; nested exception is java.util.concurrent.CompletionException: org.springframework.dao.InvalidDataAccessResourceUsageException: could not extract ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.confluence.util.i18n.I18NBean]: Factory method 'getI18NBean' threw exception; nested exception is java.util.concurrent.CompletionException: org.springframework.dao.InvalidDataAccessResourceUsageException: could not extract ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622)
... 229 more

Caused by: java.util.concurrent.CompletionException: org.springframework.dao.InvalidDataAccessResourceUsageException: could not extract ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet
at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
at java.util.concurrent.CompletableFuture.uniHandleStage(Unknown Source)
at java.util.concurrent.CompletableFuture.handle(Unknown Source)
at java.util.concurrent.CompletableFuture.handle(Unknown Source)
at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)
at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.get(SynchronousExternalCache.java:236)
at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:78)
at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:71)
at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24)
at com.atlassian.confluence.setup.settings.DefaultSettingsManager.getGlobalSettings(DefaultSettingsManager.java:40)
at com.atlassian.confluence.languages.DefaultLocaleManager.getSiteDefaultLocale(DefaultLocaleManager.java:211)
at com.atlassian.confluence.languages.DefaultLocaleManager.getLocaleInfo(DefaultLocaleManager.java:113)
at com.atlassian.confluence.languages.DefaultLocaleManager.getLocale(DefaultLocaleManager.java:195)
at com.atlassian.confluence.util.i18n.UserI18NBeanFactory.getUserLocale(UserI18NBeanFactory.java:44)
at com.atlassian.confluence.util.i18n.UserI18NBeanFactory.getI18NBean(UserI18NBeanFactory.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
... 230 more

Caused by: org.springframework.dao.InvalidDataAccessResourceUsageException: could not extract ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet
at org.springframework.orm.hibernate5.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:230)
at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:391)
at org.springframework.orm.hibernate5.HibernateTemplate.execute(HibernateTemplate.java:336)
at com.atlassian.confluence.setup.bandana.persistence.dao.hibernate.HibernateConfluenceBandanaRecordDao.getRecord(HibernateConfluenceBandanaRecordDao.java:48)
at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.retrieve(ConfluenceDaoBandanaPersister.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy212.retrieve(Unknown Source)
at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.lambda$retrieve$0(ConfluenceCachingBandanaPersister.java:80)
at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.lambda$get$11(SynchronousExternalCache.java:239)
at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.lambda$foldResult$0(CompletionStageUtils.java:19)
... 251 more

Caused by: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:106)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:80)
at org.hibernate.loader.Loader.getResultSet(Loader.java:2123)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1911)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1887)
at org.hibernate.loader.Loader.doQuery(Loader.java:932)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
at org.hibernate.loader.Loader.doList(Loader.java:2615)
at org.hibernate.loader.Loader.doList(Loader.java:2598)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2430)
at org.hibernate.loader.Loader.list(Loader.java:2425)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:502)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:370)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:216)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1481)
at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1441)
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1410)
at com.atlassian.confluence.setup.bandana.persistence.dao.hibernate.HibernateConfluenceBandanaRecordDao.lambda$getRecord$0(HibernateConfluenceBandanaRecordDao.java:52)
at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:384)
... 269 more

Caused by: org.postgresql.util.PSQLException: ERROR: relation "bandana" does not exist
Position: 181
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2476)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2189)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:300)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:117)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:431)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:71)
... 286 more

2019-10-11 05:11:18,954 ERROR [http-nio-8090-exec-6] [ContainerBase.[Standalone].[localhost].[/]] log Unhandled exception occurred whilst decorating page

2019-10-11 05:11:23,141 ERROR [http-nio-8090-exec-6] [[Standalone].[localhost].[/].[jsp]] log Servlet.service() for servlet [jsp] threw exception
com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.

Preliminary Investigation

• The table bandana did exist and there were data in it

  

• /etc/hostname was resolvable

  # ping `cat /etc/hostname`
  PING confluence-dev-59955d5ddc-bmf2x (10.88.20.186) 56(84) bytes of data.
  64 bytes from confluence-dev-59955d5ddc-bmf2x (10.88.20.186): icmp_seq=1 ttl=64 time=0.028 ms
  64 bytes from confluence-dev-59955d5ddc-bmf2x (10.88.20.186): icmp_seq=2 ttl=64 time=0.022 ms
  64 bytes from confluence-dev-59955d5ddc-bmf2x (10.88.20.186): icmp_seq=3 ttl=64 time=0.031 ms
  ^C
  --- confluence-dev-59955d5ddc-bmf2x ping statistics ---
  3 packets transmitted, 3 received, 0% packet loss, time 2031ms

• Permission to local folders/files looks fine
  • The confluence user is owner of $CONFLUENCE_HOME

    ~# ls -l /var/atlassian/application-data/confluence/data/
    total 68
    drwxr-x--- 2 confluence confluence 4096 Oct 11 03:37 bundled-plugins
    -rw-r----- 1 confluence confluence 5772 Oct 11 03:40 confluence.cfg.xml
    drwxr-x--- 2 confluence confluence 4096 Oct 11 03:39 imgEffects
    drwxr-x--- 2 confluence confluence 4096 Oct 11 03:36 index
    drwxr-x--- 2 confluence confluence 4096 Oct 11 03:37 journal
    -rw-r----- 1 confluence confluence 0 Oct 11 03:37 lock
    drwxr-x--- 2 confluence confluence 4096 Oct 11 03:36 logs
    drwx------ 2 confluence confluence 16384 Oct 11 03:35 lost+found
    drwxr-x--- 2 confluence confluence 4096 Oct 11 03:37 plugins-cache
    drwxr-x--- 5 confluence confluence 4096 Oct 11 03:37 plugins-osgi-cache
    drwxr-x--- 2 confluence confluence 4096 Oct 11 03:37 plugins-temp
    drwxr-x--- 3 confluence confluence 4096 Oct 11 03:37 shared-home
    drwxr-x--- 2 confluence confluence 4096 Oct 11 03:36 temp
    drwxr-x--- 2 confluence confluence 4096 Oct 11 03:37 webresource-temp

1. • Because I used official docker image directly, according to https://bitbucket.org/atlassian-docker/docker-atlassian-confluence-server/src/84d6dee89eb2bdabe189be9e15b9369a02b444e6/Dockerfile#lines-38:43, only logs, temp and work folders under ${CONFLUENCE_INSTALL_DIR}, which is /opt/atlassian/confluence/ are owned by confluence use

     # ls -l /opt/atlassian/confluence/
     total 244
     drwxr-xr-x 3 root root 4096 Sep 10 01:41 bin
     -rw-r--r-- 1 root root 19540 Jul 4 15:20 BUILDING.txt
     drwxr-xr-x 1 root root 4096 Sep 10 01:40 conf
     drwxr-xr-x 1 root root 4096 Sep 10 01:41 confluence
     -rw-r--r-- 1 root root 5543 Jul 4 15:20 CONTRIBUTING.md
     drwxr-xr-x 2 root root 4096 Sep 10 01:40 lib
     -rw-r--r-- 1 root root 58153 Jul 4 15:20 LICENSE
     drwxr-xr-x 2 root root 69632 Sep 10 01:40 licenses
     drwxr-xr-x 1 confluence confluence 4096 Oct 11 03:35 logs
     -rw-r--r-- 1 root root 2401 Jul 4 15:20 NOTICE
     -rw-r--r-- 1 root root 2291 Sep 5 04:18 README.html
     -rw-r--r-- 1 root root 3334 Jul 4 15:20 README.md
     -rw-r--r-- 1 root root 1202 Sep 5 04:18 README.txt
     -rw-r--r-- 1 root root 7025 Jul 4 15:20 RELEASE-NOTES
     -rw-r--r-- 1 root root 16738 Jul 4 15:20 RUNNING.txt
     drwxr-xr-x 4 root root 4096 Sep 10 01:41 synchrony-proxy
     drwxr-xr-x 1 confluence confluence 4096 Oct 11 03:36 temp
     drwxr-xr-x 2 root root 4096 Sep 5 04:18 webapps
     drwxr-xr-x 1 confluence confluence 4096 Oct 11 03:35 work

Could anyone help to see what could be the solution? Thanks.

[Charlie Misonne · Accepted solution]

I'm running into the same issue. Were you able to solve it?

[jayhding]

@Charlie Misonne , eventually after several retry the problem did not happen again. During further installation did meet 504 because initial Confluence database setup took several minutes, while our Nginx has timeout as 60s. After bypassing the Nginx everything was working fine.