Search dosen't work at Bitbucket

[netconomyadmin]

It looks like Elasticsearch dosent't work after upgrade from 5.3.1 to 5.9.1.

We have restart the Server often.

The config is the same.

Please help

[Lonnie]

Hello Team, When you upgraded your Bitbucket from  5.3.1 to 5.9.1. You also ended up upgrading your Elasticsearch too from 2.3 to 5.5. This means everything needs to be reindexed. Depending on the size of your environment it may take some time.

Having said that is could be that Elasticsearch isn't started or you are unable to connect to it. Go the Cog in the upper righthand corner > Administration > Server Settings. Find the Search section and click the test button to see if you are able to connect. If so then I suspect it what I mentioned above if not then we need to check if Elasticsearch is staring.

We can look for the process an see if it's running:

ps -ef | grep java

This will list all the java processes and you can see if one of them is Elasticsearch. If you don't see then you need to try and start it:

https://confluence.atlassian.com/bitbucketserver/starting-and-stopping-bitbucket-server-776640144.html

If it is started and you see the process then you need to make sure you can access the URL defined in the server settings. If on the server you can just paste the URL browser.

If your not I would try to telnet to the server @  port

telnet <bitbucketServer> 7992 

This should connect if not then your port is blocked either by a firewall or another process.

You can find out by running:

netstat -a 

and look for the port and see if it is listening

Hope this help, Feel free to come back with questions

Lonnie

[netconomyadmin]

Hello Lonnie,

thx for the fast answer.

I have found the problem, the elasticsearch service did not start.

But the elasticsearch service should be start automatically, when i run start-bitbucket.sh but it dosen't start.

I'm starting bitbucket like this:

systemctl start bitbucket.service

This is my systemctl entry for bitbucket:

#!/usr/bin/env bash

# Resolve any links in $0 to get the real path

[Unit]
Description=Atlassian Bitbucket Server Service
After=syslog.target network.target

[Service]
Type=forking
User=atlbitbucket
ExecStart=/opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1/bin/start-bitbucket.sh
ExecStop=/opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1/bin/stop-bitbucket.sh

[Install]
WantedBy=multi-user.target#

Do you have a idea why the bitbucket servcice starts but not the elasticsearch service?

Kind regards

Matthias

[InnoGames GmbH]

Hi,

We had the same problem.

In your Bitbucket_Home folder,  you can find the following folder:

log/search

and the file:  bitbucket_search.log

If you start Bitbucket, please watch this logfile, if you get the following message, you have the same problem as us:

[2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]


## Fix for max file descriptors :
https://www.elastic.co/guide/en/elasticsearch/reference/current/file-descriptors.html

Please change or add the following values:

bitbucket           soft    nofile          65536
bitbucket           hard    nofile          65536

in this file:
/etc/security/limits.conf

## Fix for max virtual memory areas
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode

sysctl -w vm.max_map_count=262144

and so that it will be loaded after the reboot.

paste the value:

vm.max_map_count=262144

into the following file :

/etc/sysctl.conf


Best regards

Felix

[netconomyadmin]

Hi Felix,

thx for the detailed answer, but this is not our problem. There are no logs in bitbucket_search.log if i start bitbucket.

If i start bitbucket with systemctl start bitbucket.service, only one processes start which i see if i type ps -ef | grep java.

atlbitb+ 20181     1 99 16:57 ?        00:00:09 /usr/lib/jvm/jdk-8-oracle-x64//bin/java -classpath /opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/var/atlassian/application-data/bitbucket -Dbitbucket.install=/opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1 -Xms512m -Xmx1g -XX:+UseG1GC -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.io.tmpdir=/var/atlassian/application-data/bitbucket/tmp -Djava.library.path=/opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1/lib/native;/var/atlassian/application-data/bitbucket/lib/native com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher start

If i start start-bitbucket.sh with the user atlbitbucket i get both processes and elasticsearch works.

atlbitb+ 14520 1 5 16:45 pts/0 00:00:27 /usr/lib/jvm/jdk-8-oracle-x64//bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss320k -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -Dpath.conf=/var/atlassian/application-data/bitbucket/shared/search -Des.path.home=/opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1/elasticsearch -cp /opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -p /var/atlassian/application-data/bitbucket/log/search/elasticsearch.pid -Epath.conf=/var/atlassian/application-data/bitbucket/shared/search

atlbitb+ 14537 1 30 16:45 pts/0 00:02:43 /usr/lib/jvm/jdk-8-oracle-x64//jre/bin/java -classpath /opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/var/atlassian/application-data/bitbucket -Dbitbucket.install=/opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1 -Xms512m -Xmx1g -XX:+UseG1GC -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.io.tmpdir=/var/atlassian/application-data/bitbucket/tmp -Djava.library.path=/opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1/lib/native;/var/atlassian/application-data/bitbucket/lib/native com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher start

Do you have a idea why the elasticsearch process didn't start when i start bitbucket over systemctl?

Kind regards

Matthias

[Lonnie]

So systemctl uses the start up script in /etc/init.d 

you need to make sure that script is set up to start both like the script in the installation/bin folder

This should point you in the right direction:

https://confluence.atlassian.com/bitbucketserver/starting-and-stopping-bitbucket-server-776640144.html#StartingandstoppingBitbucketServer-Start/stopBitbucketServerwhenrunningasaservice

[netconomyadmin]

I

I have install the script in /etc/systemd/system/
and this script only starts start-bitbucket.sh

The script looks like this:

#!/usr/bin/env bash

# Resolve any links in $0 to get the real path

[Unit]
Description=Atlassian Bitbucket Server Service
After=syslog.target network.target

[Service]
Type=forking
User=atlbitbucket
ExecStart=/opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1/bin/start-bitbucket .sh
ExecStop=/opt/atlassian/bitbucket/atlassian-bitbucket-5.9.1/bin/stop-bitbucket.s h

[Install]
WantedBy=multi-user.target

With older versions it works fine but now we have those problems and i don't know why.

It is possible to start elasticsearch manuell?

Kind regards

Matthias