elasticsearch not running with bitbucket v6.10

[Antonio Medač]

bitbucket version: 6.10.0.52

As seen from logs and troubleshooting, elasticsearch is not running:

[root@bitbucket ~]# ps -ef |grep [e]lastic 
[root@bitbucket ~]# tail /data/bitbucket/log/search/bitbucket_search.log 
[2020-02-05T16:17:05,811][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:17:35,811][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:18:05,813][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:18:35,814][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:05,815][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:35,816][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [5Aaqk0itQbeXaAzuqLKtog][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 2.6gb[13.3%], replicas will not be assigned to this node
[2020-02-05T16:19:58,488][INFO ][o.e.n.Node ] [bitbucket_bundled]stopping ...
[2020-02-05T16:20:00,670][INFO ][o.e.n.Node ] [bitbucket_bundled]stopped
[2020-02-05T16:20:00,670][INFO ][o.e.n.Node ] [bitbucket_bundled]closing ...
[2020-02-05T16:20:01,741][INFO ][o.e.n.Node ] [bitbucket_bundled]closed
[root@bitbucket ~]# telnet localhost 7992
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

from /data/bitbucket/log/atlassian-bitbucket.log:

2020-02-07 06:14:20,905 INFO [Caesium-1-2] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search 2020-02-07 06:14:20,907 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation. 2020-02-07 06:14:20,907 WARN [Caesium-1-2] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: e593ff34-b85c-4fb7-ab94-faca3409cfe4

 Also test from website is not working, see attached picture:

I tried curl commands, tried to reset password but can't do any of it since elasticsearch is not started.

What else could I try?

Best Regards, Antonio

[Antonio Medač · Accepted solution]

SOLVED: There was two issues:

1. wrong (root) permissions on elasticsearch folder: /opt/bitbucket/elasticsearch/

2. Broken index, solved by folowing rebuild index page.

[Kaspars Laizans]

When running 7.16 via systemd, I had an issue with missing JAVA_HOME env var. Then it would try to use builtin JRE, which was not present and fail. Only log showing this was journalctl service log, in atlassian_bitbucket.log and search/bitbucket_search.log nothing would be visible.

Added "Environment=JAVA_HOME=/path/to/java" in service definition and it managed to start up bundled elasticsearch instance

[Hitendra Chauhan]

This problem seems to be due to low disk space. Try increasing the disk space of the mount point on which data folder is located.

[Antonio Medač]

Hi,

thank you for response and help, but it is not disk space issue, it was just temporary there.... :(

[root@bitbucket buckler]# df -h /data/
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 15G 5.3G 74% /

As mentioned, connection doesn't work from web interface like I posted in screenshot above. It is not "refused", just failed to establish connection (which is normal because elastic search is not running).

I tried to reset password but it doesn't help (retreived values from $BITBUCKET_HOME/shared/search/buckler/buckler.yml file)

Here is what is shown in log, service status and evidence there is no two processes running as it should when you grep for bitbucket:

[root@bitbucket buckler]# ps -ef |grep [b]itb
tdsuser 15709 1 18 06:56 ? 00:16:15 /usr/lib/jvm/jre-11/bin/java -classpath /opt/bitbucket/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/data/bitbucket -Dbitbucket.install=/opt/bitbucket -Xms512m -Xmx1g -XX:+UseG1GC -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.io.tmpdir=/data/bitbucket/tmp -Djava.library.path=/opt/bitbucket/lib/native;/data/bitbucket/lib/native com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher start

[root@bitbucket buckler]# grep -i elasticsearch /var/log/bitbucket/atlassian-bitbucket.log
2020-02-10 06:57:30,689 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2020-02-10 06:57:45,800 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 06:58:45,820 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:00:45,834 WARN [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:04:45,849 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:12:45,980 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:22:45,993 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:32:46,006 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:42:46,019 WARN [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 07:52:46,038 WARN [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 08:02:46,052 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
2020-02-10 08:12:46,066 WARN [Caesium-1-3] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.
[root@bitbucket buckler]# systemctl status bitbucket
● bitbucket.service - Atlassian Bitbucket Server Service
Loaded: loaded (/usr/lib/systemd/system/bitbucket.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-02-10 06:56:15 UTC; 1h 25min ago
Process: 15562 ExecStop=/opt/bitbucket/bin/stop-bitbucket.sh (code=exited, status=0/SUCCESS)
Process: 15622 ExecStart=/opt/bitbucket/bin/start-bitbucket.sh (code=exited, status=0/SUCCESS)
Main PID: 15709 (java)
CGroup: /system.slice/bitbucket.service
└─15709 /usr/lib/jvm/jre-11/bin/java -classpath /opt/bitbucket/app -Datlassian.standalone=BITBUCKET -Dbitbucket.home=/data/bitbucket -Dbitbucket.install=/opt/bitbucket -Xms512m...

Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: - Access is allowed to 'others'. It is recommended that 'others' be denied
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: all access for security reasons.
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The recommended umask for Bitbucket is "u=,g=w,o=rwx" (0027) and can be
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: configured in _start-webapp.sh
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The current open files limit is set to less than 4096
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Attempting to increase limit...
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Limit increased to 4096 open files
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: Starting Bitbucket webapp at http://localhost:7990
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com start-bitbucket.sh[15622]: The Bitbucket webapp has been started.
Feb 10 06:56:15 bitbucket.shared.int.tds.tieto.com systemd[1]: Started Atlassian Bitbucket Server Service.

The main problem here is how to start elasticsearch?

Best Regards, //Antonio