Confluence installed on Ubuntu after 2 days running, confluence cannot be connected anymore.

[Londa Liau]

Confluence version: 6.4.1

Confluence license: 10 persons with 1 year maintenance 

Ubuntu version: 16.04LTS

External database: No, use built-in data base 

Issue description: 

I cannot connect to confluence server anymore even I reboot ubuntu or restart confluence for couple times.

Any idea to make my confluence come back?

Thank you.

Londa

[Danyal Iqbal]

You need to check the confluence logs for errors.

[Londa Liau]

Dear @Danyal Iqbal

please let me know how to generate the log and where is the log.

Thank you.

Londa

[Danyal Iqbal]

The logs are generated automatically (as soon as you execute ./confluence-start.sh). see

https://confluence.atlassian.com/doc/working-with-confluence-logs-108364721.html

You might be trying to start multiple instances at once. I would killall java before testing any further, if I had the luxury of a dedicated machine.

  see https://confluence.atlassian.com/doc/start-and-stop-confluence-838416264.html# for notes on starting confluence.

[Londa Liau]

Dear @Danyal Iqbal

After run command "./start-confluence.sh" , I can see "java" process on "top" command. But still I cannot access confluence via "http://localhost:8090", as for the log file (catalina.out), I cannot find anywhere to upload for your analysis, please let me know how to upload log file, my confluence is still dead for no reason.

Here under is the log of terminal when typing "./start-confluence.sh",I did not see any error.

-----------------------------------------------------------------

wiki@InternaWiki:/opt/atlassian/confluence/bin$ sudo ./start-confluence.sh

To run Confluence in the foreground, start the server with start-confluence.sh -fg
executing using dedicated user: confluence
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /opt/atlassian/confluence/jre//bin/java
2018-02-13 14:52:12,805 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
Using CATALINA_BASE: /opt/atlassian/confluence
Using CATALINA_HOME: /opt/atlassian/confluence
Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp
Using JRE_HOME: /opt/atlassian/confluence/jre/
Using CLASSPATH: /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar
Using CATALINA_PID: /opt/atlassian/confluence/work/catalina.pid
Existing PID file found during start.
Removing/clearing stale PID file.
Tomcat started.

[Londa Liau]

One more thing I forgot to mention, I cannot find "atlassian-confluence.log" in my installation directory (what I found is Server startup logs:catalina.out)

[Londa Liau]

Can we have a online meeting to troubleshoot this issue?

Whole my team is looking forward comeback of this server.

If yes, please share available time , I will send out meeting invitation. 

Thank you.

[Danyal Iqbal]

hi @Londa Liau,

• log is the only source of truth :) Search for the confluence logs everywhere. Where is your installation directory? /opt? logs might be in /var/ (logically.)

find / -name "atlassian-confluence.log"

chown -R <confluence dir>

chmod -R 755 <confluence dir>

I could join you for a webex at 1700 Berlin Time :)

[Nic Brough -Adaptavist-]

The atlassian-confluence.log lives in the home directory, not the installation.  Try <confluence home>/logs

[Londa Liau]

Dear @Danyal Iqbal

Thank you.

Please join this meeting on 5PM (Berlin time)

https://cisco.webex.com/join/loliau

[Thomas Deiler]

Dear @Londa Liau,

have you made Confluence to a systemd service, before you have rebooted? If not, Confluence will not start automatically.

What's the error message of your browser?

So long

Thomas

[Londa Liau]

Dear @Thomas Deiler

I am afraid I do not make Confluence to a system service after I done the installation and configuration via "localist:8090"

Could you please share how to make Confluence to a system service ?

Also, I did not see any error message on browser, it's showing "connecting to sever" in status bar, and connection timeout to the end.

Something strange is , I try to manually start Confluence service by

"sudo /etc/init.d/confluence start" or sudo ./confluence-start.sh,

after typing either one command, I did see some message printed out like "Tomcat is created ....." but I still cannot see login page even "http://localhost:8090" on confluence server. 

What is proper steps to start confluence manually? and How to check if confluence is started?

Do I offer you confluence log to identify this issue? If yes, please let me know how to generate the log and where is the log.

Thank you.

Londa

[Thomas Deiler]

Dear @Londa Liau,

as root user, edit the text block for the location of your Confluence Installation:

cat << EOF | sudo tee /etc/systemd/system/confluence.service
[Unit]
Description=Atlassian Confluence Service
After=syslog.target network.target
[Service]
Type=forking
User=www-data
Group=www-data
ExecStart=<your path to confluence>/bin/startup.sh
ExecStop=<your path to confluence>/bin/shutdown.sh
[Install]
WantedBy=multi-user.target
EOF

Then copy paste it to your bash shell. This will create a start up file and to run Confluence as user 'www-data' (this is more save than 'root').

Register the new service:

systemctl daemon-reload

Enable the service to start on bootup:

systemctl enable confluence.service

 make sure, that the system user 'www-data' can access Confluence folder:

chown -R www-data:www-data <your path to confluence>

and also to Confluence Home:

chown -R www-data:www-data <your path to confluence HOME directory>

If your Confluence is not running, you can start it with:

systemctl start confluence

Done

So long

Thomas

[Londa Liau]

Dear @Thomas Deiler

Some error message occurs when enable confluence.service

wiki@InternaWiki:/opt/atlassian/confluence$ systemctl enable confluence.service
confluence.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install enable confluence
insserv: warning: script 'K01confluence' missing LSB tags and overrides
insserv: warning: script 'confluence' missing LSB tags and overrides
insserv: can not remove(../rc0.d/K02lightdm): Permission denied
insserv: can not symlink(../init.d/lightdm, ../rc0.d/K01lightdm): Permission denied
insserv: can not remove(../rc1.d/K02lightdm): Permission denied
insserv: can not symlink(../init.d/lightdm, ../rc1.d/K01lightdm): Permission denied
insserv: can not remove(../rc6.d/K02lightdm): Permission denied
insserv: can not symlink(../init.d/lightdm, ../rc6.d/K01lightdm): Permission denied
insserv: fopen(.depend.stop): Permission denied
update-rc.d: error: confluence Default-Start contains no runlevels, aborting.

Any idea?

Thanks

Londa

[Thomas Deiler]

Dear @Londa Liau,

looks like the registering of the service was not done as root. Are you performing the changes as user 'root'?

So long

Thomas

[Londa Liau]

Dear @Thomas Deiler

Sorry for my bad.

I performing the changes as 'root', but still error:

root@InternaWiki:/opt/atlassian/confluence/bin# systemctl enable confluence.service
Synchronizing state of confluence.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable confluence
insserv: warning: script 'K01confluence' missing LSB tags and overrides
insserv: warning: script 'confluence' missing LSB tags and overrides
update-rc.d: error: confluence Default-Start contains no runlevels, aborting.

[Londa Liau]

Dear @Thomas Deiler

I manually start confluence by "systemctl start confluence" command, when I check "http://localhost:8090", error message occurs on web as follows, any idea?

Thanks 

Londa 

HTTP Status 500 - java.lang.IllegalStateException: Spring Application context has not been set

type Exception report

message java.lang.IllegalStateException: Spring Application context has not been set

description The server encountered an internal error that prevented it from fulfilling this request.

exception

com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Spring Application context has not been set
com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)
com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
com.atlassian.confluence.plugin.servlet.filter.ServletFilterModuleContainerFilter.getServletModuleManager(ServletFilterModuleContainerFilter.java:20)
com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:62)
com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:72)
com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:49)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:23)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)

root cause

java.lang.IllegalStateException: Spring Application context has not been set
com.atlassian.spring.container.SpringContainerContext.getComponent(SpringContainerContext.java:48)
com.atlassian.spring.container.ContainerManager.getComponent(ContainerManager.java:33)
com.atlassian.confluence.util.LazyComponentReference$Accessor.get(LazyComponentReference.java:42)
com.atlassian.util.concurrent.Lazy$Strong.create(Lazy.java:85)
com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:325)
com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143)
com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
com.atlassian.confluence.plugin.servlet.filter.ServletFilterModuleContainerFilter.getServletModuleManager(ServletFilterModuleContainerFilter.java:20)
com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:62)
com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:72)
com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:49)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:23)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)
com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)

[Thomas Deiler]

Dear @Londa Liau,

than your Ubuntu 16.04 is not configured for systemd. It uses the old mechanism.

OK. Remove the /etc/systemd/system/confluence.service file.

Then follow this instructions.

Following annotations:

• Your system is Debian like.
• Start at Step 6.
• Don't create the users 'jira', user 'www-data' instead - that exists already.
• Adapt the script to your location and your JDK/JRE.
• Add in line 2 of the script the following.

### BEGIN INIT INFO
# Provides: jira
# Required-Start: $remote_fs $syslog $network
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 
# Short-Description: Atlassian Jira
### END INIT INFO

Then proceed with the instructions. To start jira: service jira start

So long

Thomas

[Londa Liau]

Dear @Thomas Deiler

Before I give it a try, I am wondering why jira matters in this issue?

I only need confluence, and never install jira before.

Thank you.

[Thomas Deiler]

Ah sorry - switching back and forth from one question to another, i mistakenly wrote Jira.

You can take the same script but name it different. Confluence is started the same way as Jira.

I appologizes

Thomas

[Thomas Deiler]

Dear @Londa Liau,

the error message could be an indicator, that the user 'www-data' has not read/write permissions.

As something seems weird, first, do you have the system user 'www-data' at all? Have a look in /etc/passwd.

Then check if the files in Confleunce dir are owned my 'www-data'.

So long

Thomas

[Londa Liau]

Dear @Thomas Deiler

Yes, the system does have user "www-data", here under is content of /etc/passwd

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin

Then, the confluence folder owns by "www-data" as belows

root@InternaWiki:/opt/atlassian# ll
total 12
drwxrwxrwx 3 root root 4096 二 7 09:19 ./
drwxr-xr-x 4 root root 4096 二 13 09:29 ../
drwxrwxrwx 14 www-data www-data 4096 二 7 09:20 confluence/

[Thomas Deiler]

And also all files within? And same for confluence HOME?

If you have meanwhile started confluence as 'root' the file permissions could be switched back to root, for some files.

[Londa Liau]

Dear @Thomas Deiler

@Danyal Iqbal helps me to trouble-shooting this issue online, he find out the root cause of issue (confluence server is dead ) is due to data base directory in installation folder disappear which might induced by I use built-in data base with trial license, then I purchase production license (10 persons) and only update the license but keep using built-in data base (H2 data base).

For now, it's critical to me for following 2 questions:

1. How to retrieve my previous data (space, uploaded files) from backup

   (I notice there's h2db.h2.db file in /var/atlassian/application-data/confluence/database), please share the steps or may we have online meeting in case the steps in complicated.

2. Could I keep using built-in data base for 10 persons license?

Thank you.

Londa
 

[Danyal Iqbal]

1. You can try to run the following to recover the h2 database-create a .SQL file .
2. java -cp .../lib/h2*.jar org.h2.tools.Recover ...
3. Users have reported limited success with a corrupt h2 database e.g in h2db corrupted. You could explore DBvisualizer/h2 console to recover the db.

4. If the recovery  works you can export xml per UI ... Migrating to another database.

5. There might be automatic backups in confluencedata/backups/ or /export
6. No, you cannot use the built-in database with a production license.

[Londa Liau]

Dear @Danyal Iqbal

1.After step2,  I saw there's new file "h2db.h2.sql" in /var/atlassian/application-data/confluence/database, what is the next steps ? (I still cannot see web UI for export)

2. Also, there're several files as belows in "/var/atlassian/application-data/confluence/backups"

backup-2018_02_08.zip
backup-2018_02_09.zip
backup-2018_02_10.zip
backup-2018_02_11.zip
backup-2018_02_12.zip

Are they suitable for database migration? 

If yes, what is the next steps?

I plan to install postgresql then import backup files retrieved from step1 or step2.

Thanks

Londa Liau

[Danyal Iqbal]

"h2db.h2.sql" is supposed to be like a db dump.  You can import it in your database directly. However, Users have reported limited success with it.

Since you have automatically generated backups, I would install the postgres db next, get the app working and than import the xml backup-backup-2018_02_12.zip through the UI. Put the XML file in the Confluence home directory of the new site (<CONFLUENCE-HOME-DIRECTORY>\restore) then choose Restore

[Londa Liau]

Dear @Danyal Iqbal

I installed postgresql , and add user/password of postgresql,

but when I input user/password in "configure databasde" web page then press "test connection" button, I always get incorrect username and password.

The instructions I follow is

https://confluence.atlassian.com/doc/database-setup-for-postgresql-173244522.html

The command I input is

create database confluence owner confuser encoding 'utf8';

ALTER USER confuser WITH PASSWORD 'confuser'

Could you please help to check what's wrong of my setting or may we have quick online meeting for this?

If yes, please share your available time. 

Thank you.

Londa Liau

[Danyal Iqbal]

grant db permission to the db.user with

GRANT ALL PRIVILEGES ON DATABASE <Database Name> TO <Role Name>

 how about 1700 berlin Time today?

[Londa Liau]

Dear @Danyal Iqbal

Some documentation on internet mentioned it's needed to download "database driver" , my installation is to run "sudo ./atlassian-confluence-6.4.2-x64.bin" in ubuntu server,

Do I need to download database driver?

Thanks .

Londa Liau

[Londa Liau]

Dear @Danyal Iqbal

Could we move meeting time a little earlier due to 5pm Berlin time 

is mid-night my time.

How about 4pm Berlin time?

Thank you

Londa 

[Danyal Iqbal]

Do I need to download database driver?

I am not sure, I think it is already there-postgres ist open source. I could imagine having to download the drivers for Oracle/MSSQL but not for postgres.  You should see missing driver exception in the logs if it is not there.

 Unfortunately, i am commited till 5pm today. How about 12 tomorow?

[Londa Liau]

Dear @Danyal Iqbal

Please forget question regarding database driver of postgresql,

In the link I shared , it mentioned database driver is only required for Oracle/MSSQL.

12PM  tomorrow Berlin time is good to me.

Here is webbed link:

https://cisco.webex.com/join/loliau

Talk to you then.

Thank you.

Londa Liau

[Londa Liau]

Dear @Danyal Iqbal

Eventually, I got this issue fixed.therefore I would like to cancel

today's meeting.

Thanks for your help.

Londa Liau

[Danyal Iqbal]

cheers mate !