Error after Bitbucket 7.18.3 upgrade.

[Khang Pham]

I am relatively new to administering Bitbucket and would like some input. I have just upgraded bitbucket from 6.10 to 7.18.3. The upgrade itself seems okay, however upon starting the bitbucket service, I noticed that there are 2 logs which caught my eye and wondering if these are of any concern:

1. /opt/atlassian/bitbucket/7.18.3/elasticsearch/bin/elasticsearch: line 43: bin/elasticsearch-keystore: Permission denied

Is anyone able to clarify what the elasticsearch-keystore does? I have checked the permissions on it and confirmed it has the same permissions as the user starting the bitbucket service. 

2. The current open files limit is set to less than 6192
Attempting to increase limit...
/opt/atlassian/bitbucket/7.18.3/bin/_start-webapp.sh: line 63: ulimit: open files: cannot modify limit: Operation not permitted

I believe this is used for admin to set resource usage, however not too sure what particular error indicates. 

Anyone able to explain the 2 errors and possible fix for them? Should these be of any concern?

I've tried looking for these errors in the KBs but not much luck. 

[Craig Steadman · Accepted solution]

Change into the folder and double check the rights on the files and also see if the will execute manually.  The line 43: bin/elasticsearch-keystore: Permission denied  error cleared up for me after give the files execute rights.

[Khang Pham]

The permissions are correctly assigned to the elasticsearch. After stopping the service and running it manually, it gave me a java error, but after starting the bb service again, the line 43: bin/elasticsearch-keystore: Permission denied  error is no longer appearing!

Thanks @Craig Steadman , this seems to have solved the first issue. Any idea on the second issue?

[Craig Steadman]

root can increase & decrease the ulimit setting ok

atlbitbucket for me can only reduce the setting

you can try adjusting the limits here

/etc/security/limits.conf

or you can tune sudo to allow atlbitbucket to run the ulimit command as root

and adjust the bin/_start-webapp.sh to run the command using sudo.

[Khang Pham]

@Craig Steadman 

Are you able to give me a example for the config of the /etc/security/limits.conf?

I did try and add the following lines in the limits.conf file:

#<domain> <type> <item> <value>
atlbitbucket soft nofile unlimited
atlbitbucket hard nofile unlimited

Restarted the bitbucket service but still getting the second error. 

[Craig Steadman]

contents should be something like

*          soft    nofile    100000

*          hard   nofile    100000

*  soft  core    0

*  hard code  0

*  hard maxlogins   10

you just need to set the default higher than what bitbucket needs

[Nic Brough -Adaptavist- · Accepted solution]

Elastic is the search system underneath bitbucket.  The file you're looking at holds keys for encrypted access.

The errors you are getting are happening because your server is not set up correctly.  The Linux user running the service does not have the right permissions. 

This usually happens when someone starts the service while being root, instead of starting it as the user dedicated to running bitbucket, then stops it, and restarts it as the right user.  The process run as root takes ownership of files and directories, making them inaccessible to the actual owner.

You should be able to fix this by stopping bitbucket and using chown -R user:group /opt/atlassian/bitbucket

For your second error, it might be a hangover from the first, but if it is not, then you'll need to increase the file limit for the user running bitbucket.  Edit the /etc/security/limits.conf file

[Khang Pham]

Hi Nic,

The permissions on Bitbucket installation are the default permissions set by the Bitbucket installer being atlbitbucket:atlbitbucket under /opt/atlassian/bitbucket. I have also confirmed the start-bitbucket.sh permissions is set as atlbitbucket:atlbitbucket.

When I ran the start script, I performed sudo -u atlbitbucket ./start-bitbucket.sh to start the service. This should've started the bitbucket service as atlbitbucket but unfortunately, the error still occurs once the service starts. Is there anything else I need to check for this?

[Khang Pham]

Any other ideas @Nic Brough -Adaptavist- I can check?

[Nic Brough -Adaptavist-]

Are you sure that command is running it as the right user?  Have the permissions or ownership of the files and directories changed after you ran it?

[Khang Pham]

Hi @Nic Brough -Adaptavist- 

Yes - the permissions for the whole /opt/atlassian/bitbucket/7.18.3/ is set as atlbitbucket:atlbitbucket (this was already set when the upgrade happen. I think the installer auto sets the permission to atlbitbucket user and group.)

When starting the service, I log in as myself (my own user with sudo privileges) and start the service by doing sudo -u atlbitbucket ./start-bitbucket.sh. Craig has helped me with the first error - Just needed to make elasticsearch executable. 

I am still getting the second error though. I did try and add the atlbitbucket user in the /etc/security/limits.conf file and set it as:

#<domain> <type> <item> <value>
atlbitbucket soft nofile unlimited
atlbitbucket hard nofile unlimited

but not much luck. Still getting the same error when starting the service. 

[Nic Brough -Adaptavist-]

Have you rebooted the server since changing the limts file?

[Khang Pham]

Ok, so after rebooting the server. I couldn't even log in. Turns out, you can't set it as unlimited, so I just set it as 6192 and that seemed to work. Thanks Nic. Looks like no more errors now!

[Craig Steadman]

Check the execute permissions on the files in elasticsearch/bin folder

chmod 750 /opt/atlassian/bitbucket/7.xx.x/elasticsearch/bin/*

note: set your version in the command string above

[Khang Pham]

Hi @Craig Steadman 

I just changed the permissions in the elasticsearch/bin folder as you suggested to all execute, and ran the service again. Unfortunately, still getting the same 2 errors.