Confluence crash/freeze: disk space error message in log

[Joe F]

Confluence crashed yesterday, I could not ssh in to restart the service initially, but the IP responded to a ping. After approx 45m I was able to ssh in to the VM and restart the confluence service (sudo service confluence stop/start) and confluence came backup.

When checking the log file /var/../../atlassian-confluence.log I see an entry (see below) about 'low free disk space'. I du'd and noted 85% use, then removed some old back-up files, making 64% use. The log file still presents the 'low disk space' message this morning.  Where should I be looking now to resolve this? Maybe it is a memory issue?

2020-05-01 08:52:03,883 ERROR [alert-dispatch:thread-1] [atlassian-monitor] log 2020-04-30T20:52:03.859Z Component 'Operating System' alerted 'Low free disk space' (details: {"directory":"/var/atlassian/application-data/confluence/shared-home","freeInMegabytes":4969,"totalInMegabytes":14506,"minimumInMegabytes":8192}, trigger: {"pluginKey": "not-detected"})
2020-05-01 08:52:37,817 WARN [Caesium-1-1] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-05-01 08:54:03,881 ERROR [alert-dispatch:thread-1] [confluence.alert-log] log 1588280043861 ; ERROR ; OS ; OS-1002 ; Low free disk space ; not-detected ; ; ; {"directory":"/var/atlassian/application-data/confluence","freeInMegabytes":4969,"totalInMegabytes":14506,"minimumInMegabytes":8192}
2020-05-01 08:54:03,882 ERROR [alert-dispatch:thread-1] [atlassian-monitor] log 2020-04-30T20:54:03.861Z Component 'Operating System' alerted 'Low free disk space' (details: {"directory":"/var/atlassian/application-data/confluence","freeInMegabytes":4969,"totalInMegabytes":14506,"minimumInMegabytes":8192}, trigger: {"pluginKey": "not-detected"})
2020-05-01 08:54:03,901 ERROR [alert-dispatch:thread-1] [confluence.alert-log] log 1588280043862 ; ERROR ; OS ; OS-1002 ; Low free disk space ; not-detected ; ; ; {"directory":"/var/atlassian/application-data/confluence/shared-home","freeInMegabytes":4969,"totalInMegabytes":14506,"minimumInMegabytes":8192}
2020-05-01 08:54:03,902 ERROR [alert-dispatch:thread-1] [atlassian-monitor] log 2020-04-30T20:54:03.862Z Component 'Operating System' alerted 'Low free disk space' (details: {"directory":"/var/atlassian/application-data/confluence/shared-home","freeInMegabytes":4969,"totalInMegabytes":14506,"minimumInMegabytes":8192}, trigger: {"pluginKey": "not-detected"})

[Brant Schroeder]

Joe,

  The error is accurate you have low disk space.  You will either need to clean up the disk and free up space or you will need to provision additional space.  Do you happen to have large log files and lots of them or a lot of backups that could still be removed?  How mush of your total disk space is being used?  What OS are you running on?

https://confluence.atlassian.com/confkb/confluence-diagnostics-low-free-disk-space-953667584.html

[Joe F]

Thanks Brant,

The OS is linux16. There are more old backups I can delete. I have removed some already and as stated du went from 84% to 64%. I can also remove some old logs I guess. I will do further work to increase the free space and monitor the log.

[Joe F]

this is df now: 57%, should be plenty do you think?

root@confluence-prod-02:/# df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.0G 0 2.0G 0% /dev
tmpfs 396M 16M 380M 4% /run
/dev/mapper/template--vg-root 15G 7.6G 5.9G 57% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/sda1 472M 161M 287M 36% /boot
tmpfs 396M 0 396M 0% /run/user/1002
tmpfs 396M 0 396M 0% /run/user/1000
root@confluence-prod-02:/#

[Brant Schroeder]

Are you still having the issue?

[Joe F]

Well, Confluence is not crashing. But the log still is recording low disk space as above.

I am looking at how to increase the VM hard drive and extend the partitions so we have more capacity/room on that drive.

I'm thinking it should be ok with what it has (57%) but I'm not sure. What do you think?

[Joe F]

I checked inode allocation as well, seems ok at 16%, but I have no info on what a bad count would be...

This link for the above: https://confluence.atlassian.com/confkb/not-enough-space-on-disk-errors-in-confluence-6-5-or-later-when-disk-space-is-available-939501396.html

[Brant Schroeder]

Yes 57% will be sufficient.  The issue with the logs is probably where your threshold is set.  

You can change the threshold of this alert using the following system property:

diagnostics.os.min.free.disk.space.megabytes

See Configuring System Properties for more information on setting system properties. 

[Joe F]

We upgraded post gres and jira about a month ago (both seperate VM's) so no recent changes to Confluence I know of. I'll do what you suggest and check the threshhold setting and get back to you. What might be a good threshold setting (75%? 80%?...) Thank you for replies...

[Brant Schroeder]

Joe,

 Confluence crashed or was unusable and had to be restarted.  I am not positive that the low disk space was the issue.  It really depends on a lot of things.  If you are stating that your disk space usage was only at 85% use, that possibly could be what caused the issue.  57% should be enough but it depends on how big your instance is and what is filling up your disk space.  I am assuming it is logs and daily backups.  You should look and see what your daily backups size is.  If it is large 2gigs or more than 80% is a bad idea for the amount of disk space you have.  You need to take in to consider the disk space you have an how quickly it can be filled up to determine when you want to be alerted.

[Joe F]

Thanks Brant,

I'm not certain the low disk space caused the crash either. I noticed the message when checking logs for a cause and saw the message then.

The atlassian confl... dir is 5.4Gb (/var/atlassian/application-data# du -hsc confluence/)

A tail of the log follows fyi. It shows low free disk space and low memory. Something weird is going on... Any other logs I should check?

root@confluence-prod-02:/var/atlassian/application-data/confluence/logs# tail atlassian-confluence.log

2020-05-06 11:33:48,064 ERROR [alert-dispatch:thread-1] [confluence.alert-log] log 1588721627691 ; ERROR ; OS ; OS-1002 ; Low free disk space ; not-detected ;  ;  ; {"directory":"/var/atlassian/application-data/confluence","freeInMegabytes":4508,"totalInMegabytes":14506,"minimumInMegabytes":8192}

2020-05-06 11:33:48,064 ERROR [alert-dispatch:thread-1] [atlassian-monitor] log 2020-05-05T23:33:47.691Z Component 'Operating System' alerted 'Low free disk space' (details: {"directory":"/var/atlassian/application-data/confluence","freeInMegabytes":4508,"totalInMegabytes":14506,"minimumInMegabytes":8192}, trigger: {"pluginKey": "not-detected"})

2020-05-06 11:33:48,066 ERROR [alert-dispatch:thread-1] [confluence.alert-log] log 1588721627692 ; ERROR ; OS ; OS-1002 ; Low free disk space ; not-detected ;  ;  ; {"directory":"/var/atlassian/application-data/confluence/shared-home","freeInMegabytes":4508,"totalInMegabytes":14506,"minimumInMegabytes":8192}

2020-05-06 11:33:48,066 ERROR [alert-dispatch:thread-1] [atlassian-monitor] log 2020-05-05T23:33:47.692Z Component 'Operating System' alerted 'Low free disk space' (details: {"directory":"/var/atlassian/application-data/confluence/shared-home","freeInMegabytes":4508,"totalInMegabytes":14506,"minimumInMegabytes":8192}, trigger: {"pluginKey": "not-detected"})

2020-05-06 11:35:47,708 WARN [alert-dispatch:thread-1] [confluence.alert-log] log 1588721747693 ; WARNING ; OS ; OS-1001 ; Low free memory ; not-detected ;  ;  ; {"freeInMegabytes":87,"totalInMegabytes":3951,"minimumInMegabytes":256}

2020-05-06 11:35:47,709 WARN [alert-dispatch:thread-1] [atlassian-monitor] log 2020-05-05T23:35:47.693Z Component 'Operating System' alerted 'Low free memory' (details: {"freeInMegabytes":87,"totalInMegabytes":3951,"minimumInMegabytes":256}, trigger: {"pluginKey": "not-detected"})

2020-05-06 11:35:47,717 ERROR [alert-dispatch:thread-1] [confluence.alert-log] log 1588721747694 ; ERROR ; OS ; OS-1002 ; Low free disk space ; not-detected ;  ;  ; {"directory":"/var/atlassian/application-data/confluence","freeInMegabytes":4508,"totalInMegabytes":14506,"minimumInMegabytes":8192}

2020-05-06 11:35:47,717 ERROR [alert-dispatch:thread-1] [atlassian-monitor] log 2020-05-05T23:35:47.694Z Component 'Operating System' alerted 'Low free disk space' (details: {"directory":"/var/atlassian/application-data/confluence","freeInMegabytes":4508,"totalInMegabytes":14506,"minimumInMegabytes":8192}, trigger: {"pluginKey": "not-detected"})

2020-05-06 11:35:47,724 ERROR [alert-dispatch:thread-1] [confluence.alert-log] log 1588721747694 ; ERROR ; OS ; OS-1002 ; Low free disk space ; not-detected ;  ;  ; {"directory":"/var/atlassian/application-data/confluence/shared-home","freeInMegabytes":4508,"totalInMegabytes":14506,"minimumInMegabytes":8192}

2020-05-06 11:35:47,724 ERROR [alert-dispatch:thread-1] [atlassian-monitor] log 2020-05-05T23:35:47.694Z Component 'Operating System' alerted 'Low free disk space' (details: {"directory":"/var/atlassian/application-data/confluence/shared-home","freeInMegabytes":4508,"totalInMegabytes":14506,"minimumInMegabytes":8192}, trigger: {"pluginKey": "not-detected"})

[Brant Schroeder]

If you are running out of memory (RAM) you will have issues for sure.  You should tune your instance following the steps here:https://confluence.atlassian.com/conf615/performance-tuning-967338634.html

[Joe F]

Well, that's what the log says, yet no users (we only have 10) report any issues. I did check the VM this morn and thought about adding more RAM but hung back on that, pending this kind of discussion. I'll check what the link suggests and maybe I'll go ahead tonight I think, double RAM to 8Gb anyway... and check the logs tomorrow.

And then I guess I allocate more HD space to the VM and extend the partition in Linux... I'm nervous.

Thanks again.