Further ideas on 401 errors with elasticserarch

[Michael Lightfoot]

This question is in reference to Atlassian Documentation: Troubleshooting Connection Issues for Elasticsearch

Although this appears to be the issue we are having, the solution does not resolve our issue.

We followed the resolution outlined above but are still getting 401 errors and the username and password fields are empty in the admin UI, although the elasticserarch home URL is there.

There are no relevant entries in the bitbucket.properties file.

As part of the attempt to ensure things were starting correctly I changed from SysV init scripts to systemd.  systemctl status bitbucket shows both processes started.

Our environment is Ubuntu 16.04LTS and Bitbucket 4.8.2

[Kristy]

Hi Michael,

I would recommend opening a support ticket at support.atlassian.com. From there you will be able to provide Atlassian with logs and properties files which will help to debug the problem.

Kristy

[Michael Lightfoot]

We've done that and are waiting for a response.

I've also done a great deal of ferreting around in the bowels of Bitbucket and elasticsearch.

I can report that the problem is related to Bitbucket not connecting to the search:

2016-09-01 08:02:35,702 INFO  [I/O dispatcher 9] lig047 @S1KM40x482x71478x0 1vyzj3z 152.83.229.24,0:0:0:0:0:0:0:1 "POST /rest/search-admin/latest/test-connection HTTP/1.1" c.a.b.s.s.i.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed: HTTP response status code is 401

2016-09-01 08:03:07,592 INFO  [I/O dispatcher 13] lig047 @S1KM40x483x71568x0 1vyzj3z 152.83.229.24,0:0:0:0:0:0:0:1 "POST /rest/search-admin/latest/test-connection HTTP/1.1" c.a.b.s.s.i.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed: HTTP response status code is 401

2016-09-01 08:05:00,034 INFO  [Caesium-1-1]  c.a.b.s.i.i.jobs.StartupChecksJob Attempting provisioning in startup job

2016-09-01 08:05:00,037 INFO  [Caesium-1-1]  c.a.b.s.i.i.jobs.StartupChecksJob Running startup jobs for search

2016-09-01 08:05:00,041 ERROR [Caesium-1-1]  c.a.b.s.i.i.IndexingSynchronizationService Unable to synchronize the mapping in Elasticsearch

2016-09-01 08:05:00,041 WARN  [Caesium-1-1]  c.a.b.s.i.i.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.search.internal.indexing.jobs.StartupChecksJob  Job id: 91e10c10-712b-4486-b768-6f0592a24a4c

On our test system it all works.  Something in the upgrade process on production seems to have broken it.
 

[Kristy]

Hi Michael,

What version did you upgrade from? If it was earlier than 4.6 then the elasticsearch instance would not have been running, so the upgrade wouldn't have "broken" it in that sense. Another thing that might impact it is whether you are running a datacenter instance of Bitbucket or not.

The connection tester is coming back with a 401 which means that elasticsearch is set up with a password that Bitbucket does not know about.

I really recommend waiting for support's response to this, since they have access to all of the logs and details about your Bitbucket Home and Elasticsearch Home, and whether you are running with Data Center or not.

However, details on where to find the Elasticsearch password (configured by the Buckler plugin in Elasticsearch) can be found here: https://confluence.atlassian.com/display/BitbucketServer/Install+and+configure+a+remote+Elasticsearch+instance

[Michael Lightfoot]

Thanks Kristy.

I was not altogether accurate.  We upgraded first using a test database, then cut over to our production database which is when the problem occurred.

I will wait for Atlassian to look at our ticket.