diy backup bug in scripts?

[Paul Smith]

Hi,

   I am trying to set up diy backup for my bitbucket server instance. I am a newbie, so please be gentle....

I have set up a test bitbucket instance with some users which appears to be working fine. I have created a diy-backup script as per the instructions and it appears to run, but the diagnostic output says Elasticsearch succeeded, but then backup failed; Note that I am not using any of the elasticsearch variables as it is a standard installation, so I am assuming that it shold be calling the no_op version of the Elasticsearch backups. (my example is based upon the postgres-rsync example script)

Running the command ./bitbucket-diy-backup.sh reports;

[common.ls /bitbucket-sh:17 -> source][bb] DEBUG: Using vars file ; './bitbucket.diy-backup.vars.sh'

(rsync stuff - works)

[common.sh:114 -> lock_bitbucket][bb] DEBUG: Running curl -L -s -f -u ******:****** http://localhost:7990/mvc/maintenance.lock
[common.sh:114 -> lock_bitbucket][bb] INFO: Bitbucket has been locked for maintenance. It can be unlocked with:

(etc.)

[bitbcket-diy-backup.sh:118 -> run_in_bg][bb] DEBUG: Started Elasticsearch backup (PID=2823)
[common.sh:120 -> backup_wait][bb] INFO: Waiting for Bitbucket to become ready to be backed up
[bitbcket-diy-backup.sh:0 -> main][bb] INFO: Backing up the database and filesystem in parallel
rm: cannot remove '/backup-root/bitbucket-db/': No such file or directory
[bitbucket-diy-backup.sh:123 -> run_in_bg][bb] DEBUG: Started Database backup (PID=2852)
[bitbucket-diy-backup.sh:124 -> run_in_bg][bb] DEBUG: Started Disk backup (PID=2858)
[disk-rsync.sh:23 -> perform_rsync_home_directory][bb] DEBUG: Running rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/shared/data/db.* --exclude=/search/data/ --exclude=/shared/search/data/  --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock --exclude=/shared/.lock /home/bbuser/bbdata/ /backup-root/bitbucket-home
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] DEBUG: Waiting for Databse backup (PID=2852)
sending incremental file list
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] DEBUG: Waiting for Disk backup (PID=2858)
analytics-logs/16588fa71a97d9104d1549704b342b16.090b4a5c-a7df-430f-afdf-80d2edc0eeac.atlassian-analytics.log

sent 10.76K bytes  received 151 bytes  21.81K bytes/sec
total size is 279.91K  speedup is 25.67
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] DEBUG: Disk backup finished successfully (PID=2858)
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] DEBUG: Waiting for Elasticsearch backup (PID=2823)
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] DEBUG: Elasticsearch backup finished successfully (PID=2823)
[bitbucket-diy-backup.sh:127 -> wait_for_bg_jobs][bb] ERROR: Database Backup failed with status 1 (PID=2823)
[common.sh:129 -> unlock_bitbucket][bb] DEBUG: running curl -L -s -f - u ******:****** http://loclahost:7990/mvc/maintenance/lock?token=214e9737d46b09adefda6668367458c7fa9abb3a
[bitbucket-diy-backup.sh:130 -> cleanup_incomplete_backup][bb] DEBUG: Cleaning up after failed backup
[bitbucket-diy-backup.sh:0 -> main][bb] ERROR: Backing up bitbucket failed
[common.sh:1 -> run_cleanup][bb] DEBUG: Running cleanup jobs...
(no further output!)


....but it doesnt cleanup properly either. I have to reboot the machine to get out of this one as even entering the correct backup token to unlock it still leaves the backup hung.

My major problem which I can;t get my head around is if the backup succedded why is it reporting an exit code of 1 and what can I do to fix it?

[Rajesh]

Hi  @Mehmet Fatih Karagoz   @Paul Smith

The DIY Backup for Bitbucket MYSQL instances was failing because the BITBUCKET_BACKUP_DB variable was a directory and not a file as expected by mysqldump command in database-mysql.sh

The below properties in bitbucket.diy-backup.vars.sh.example is common to all databases. Except MYSQL, other databases expect BITBUCKET_BACKUP_DB to be directory for backup. 

bitbucket.diy-backup.vars.sh.example

# The path to working folder for the backup
BITBUCKET_BACKUP_ROOT=
BITBUCKET_BACKUP_DB=${BITBUCKET_BACKUP_ROOT}/bitbucket-db/
BITBUCKET_BACKUP_HOME=${BITBUCKET_BACKUP_ROOT}/bitbucket-home/
BITBUCKET_BACKUP_DATA_STORES=${BITBUCKET_BACKUP_ROOT}/bitbucket-data-stores/

# The path to where the backup archives are stored
BITBUCKET_BACKUP_ARCHIVE_ROOT=

@Michał Ciania You nailed it with the right fix for backup, but it would have failed during restore.

Along with few other improvements, we have added the fix for both backup & restore in the database-mysql.sh file. Change of behaviour is no longer required in the bitbucket.diy-backup.vars.sh.example file. The fix has been fully tested for both backup & restore of Bitbucket Mysql instance and has been merged with the master branch. 

Please clone the latest master branch from Atlassian Labs - Bitbucket-DIY-Backup and use the DIY backup script for a seemless Backup & Restore of Bitbucket Mysql instance.

Thanks

[Daniel Alonso]

The problem is the pg_dump version you are using.

If you try the command pg_config, you will see the version of PostgreSQL:

In my case:

VERSION = PostgreSQL 9.2.24

In 9.2 pg_dump doesn't have the option '-d <database>' which is present on the file 'database-postgresql.sh':

run pg_dump -U "${POSTGRES_USERNAME}" -h "${POSTGRES_HOST}" --port=${POSTGRES_PORT} ${PG_PARALLEL} -Fd \
-d "${BITBUCKET_DB}" ${PG_SNAPSHOT_OPT} -f "${BITBUCKET_BACKUP_DB}"

Update your PostgreSQL and it will work now, you can try the pg_dump command directly and if it works, the backup script will work.

In my case, I need to install PostgreSQL 9.6 manually due restrictions with the customer network.

If you need help with it, please contact us on trundl.com

Daniel

[Michał Ciania]

Confirmed: What can say - exactly same here with MySQL. I try to use it to backup Bitbucket Docker instance. It's fine - all the files from named volume are processed, but it fails on database part.:

sent 13.74G bytes received 3.55M bytes 68.56M bytes/sec
total size is 13.72G speedup is 1.00
[common.sh:114 -> lock_bitbucket][9da8c79a4313] DEBUG: Running curl -L -s -f -u ******:****** http://atlassian-test:7990/mvc/maintenance/lock
[common.sh:114 -> lock_bitbucket][9da8c79a4313] INFO: Bitbucket has been locked for maintenance. It can be unlocked with:
[common.sh:114 -> lock_bitbucket][9da8c79a4313] INFO: curl -u ... -X DELETE -H 'Content-type:application/json' 'http://atlassian-test:7990/mvc/maintenance/lock?token=271fe3ec99a84a1997eff6e29cdac3664a9f2db4'
[common.sh:115 -> backup_start][9da8c79a4313] DEBUG: Running curl -L -s -f -u ******:****** http://atlassian-test:7990/mvc/admin/backups?external=true
[common.sh:115 -> backup_start][9da8c79a4313] INFO: Bitbucket server is now preparing for backup. If the backup task is cancelled, Bitbucket Server should be notified that backup was terminated by executing the following command:
[common.sh:115 -> backup_start][9da8c79a4313] INFO: curl -u ... -X POST -H 'Content-type:application/json' 'http://atlassian-test:7990/mvc/maintenance?token=c81a803cef12948ebb0ffe9863547c7d151e4f34'
[common.sh:115 -> backup_start][9da8c79a4313] INFO: This will also terminate the backup process in Bitbucket Server. Note that this will not unlock Bitbucket Server from maintenance mode.
[bitbucket.diy-backup.sh:118 -> run_in_bg][9da8c79a4313] DEBUG: Started Elasticsearch backup (PID=85)
[common.sh:120 -> backup_wait][9da8c79a4313] INFO: Waiting for Bitbucket to become ready to be backed up
[bitbucket.diy-backup.sh:0 -> main][9da8c79a4313] INFO: Backing up the database and filesystem in parallel
rm: can't remove '/var/cache/bitbucket-diy/bitbucket-db': No such file or directory
[bitbucket.diy-backup.sh:123 -> run_in_bg][9da8c79a4313] DEBUG: Started Database backup (PID=111)
[bitbucket.diy-backup.sh:124 -> run_in_bg][9da8c79a4313] DEBUG: Started Disk backup (PID=116)
[disk-rsync.sh:23 -> perform_rsync_home_directory][9da8c79a4313] DEBUG: Running rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/shared/data/db.* --exclude=/search/data/ --exclude=/shared/search/data/ --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock --exclude=/shared/.lock /mnt/server-data /var/cache/bitbucket-diy/bitbucket-home/
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] DEBUG: Waiting for Disk backup (PID=116)
sending incremental file list
server-data/caches/tag-index/
server-data/caches/tag-index/cache-entry14465606170308675900
server-data/export/
server-data/export/backup-backup-stash-20191211-085756-098Z.zip
server-data/log/atlassian-bitbucket-access.log
server-data/log/atlassian-bitbucket.log
server-data/log/audit/atlassian-bitbucket-audit.log

sent 7.40M bytes received 35.32K bytes 1.35M bytes/sec
total size is 13.72G speedup is 1,845.82
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] DEBUG: Disk backup finished successfully (PID=116)
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] DEBUG: Waiting for Database backup (PID=111)
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] DEBUG: Waiting for Elasticsearch backup (PID=85)
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] DEBUG: Elasticsearch backup finished successfully (PID=85)
[bitbucket.diy-backup.sh:127 -> wait_for_bg_jobs][9da8c79a4313] ERROR: Database backup failed with status 1 (PID=85)
[common.sh:129 -> unlock_bitbucket][9da8c79a4313] DEBUG: Running curl -L -s -f -u ******:****** http://atlassian-test:7990/mvc/maintenance/lock?token=271fe3ec99a84a1997eff6e29cdac3664a9f2db4
[bitbucket.diy-backup.sh:130 -> cleanup_incomplete_backup][9da8c79a4313] DEBUG: Cleaning up after failed backup
[bitbucket.diy-backup.sh:0 -> main][9da8c79a4313] ERROR: Backing up Bitbucket failed
[common.sh:1 -> run_cleanup][9da8c79a4313] DEBUG: Running cleanup jobs...

Also it fails with cleanup job and I need to recreate Bitbucket container.

[ajay.vanga@omnicell.com]

How were you able to resolve this issue. I ran into a similar issue while running the DIY script.

[Paul Smith]

Hi Ajay,

    I did start looking into this and started putting debug in to the script to figure out what was going on but I am overloaded with work and so never completed the task. As our team is only 4 member strong what I do is manually shut down the services and the manually backup the bitbucket filesystem and manually backup the Postgres database. 

One day I hope to get back on to trying to figure out the issue, but for the moment I can back up using this method. If I ever find the answer I will post it here.

[ajay.vanga@omnicell.com]

Thanks for your Reply Paul. I will post an answer if i get this script to work too.

~Ajay

[Michał Ciania]

Quick fix for this bug (but only for mysql). Not tested restoring, but it's an archive so manual restoring isn't an issue:

diff --git a/database-mysql.sh b/database-mysql.sh
index 8e4b29c..4b315cb 100644
--- a/database-mysql.sh
+++ b/database-mysql.sh
@@ -20,9 +20,9 @@ function prepare_backup_db {
}

function backup_db {
- rm -r "${BITBUCKET_BACKUP_DB}"
- run mysqldump ${MYSQL_HOST_CMD} ${MYSQL_BACKUP_OPTIONS} -u "${MYSQL_USERNAME}" -p"${MYSQL_PASSWORD}" \
- --databases "${BITBUCKET_DB}" > "${BITBUCKET_BACKUP_DB}"
+ mkdir -p "${BITBUCKET_BACKUP_DB}"
+ mysqldump ${MYSQL_HOST_CMD} ${MYSQL_BACKUP_OPTIONS} -u "${MYSQL_USERNAME}" -p"${MYSQL_PASSWORD}" \
+ --databases "${BITBUCKET_DB}" > "${BITBUCKET_BACKUP_DB}/${BITBUCKET_DB}.sql"
}

function prepare_restore_db {

[Mehmet Fatih Karagoz]

I see exactly the same behavior 6.8.1 version. Any solution is much appreciated.