How to rebuild corrupted Elasticsearch index

[John Lawlor]

I am using a remote Elasticsearch instance with Bitbucket. It looks like the index has become corrupt or something:

2019-03-11 10:18:09,398 ERROR [search-indexing:thread-1] c.a.b.i.s.i.e.DefaultIndexEventWorker Indexing - Failed for repository NSI/nsi (id: 1532) with error: Unexpected error occurred when requesting index state document (id=1532) the HTTP Status code is: 404
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Unexpected error occurred when requesting index state document (id=1532) the HTTP Status code is: 404
at com.atlassian.bitbucket.internal.search.indexing.indexer.IndexStateService.lambda$save$2(IndexStateService.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:511)
at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:466)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:244)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)

Does anyone know how to rebuild the index for a remote Elasticsearch instance?

[John Lawlor · Accepted solution]

You can check your index as follows:

http://<ip of elastic search instance>:9200/_cat/indices

Rebuild a corrupt index:

curl -u <admin user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' https://bitbucket.mycompany.com/rest/indexing/latest/sync

Once you do this if you check this URL again:

http://<ip of elastic search instance>:9200/_cat/indices

You should see the numbers on the right most column start to increase again:

yellow open bitbucket-project       qdfaTpJST6yQSygS-cnlyA 5 1    172 1131    52kb    52kb
yellow open bitbucket-search-v1     H0fGxz-RSB6hA_gA0lCbmw 5 1 152894 1623 715.9mb 715.9mb
yellow open bitbucket-index-state   d4fYg7BUR86kKAtJLtaPiw 5 1   1040  140 242.8kb 242.8kb
yellow open bitbucket-repository    Wu1f02P8Td6pWaBJIEDJhw 5 1   1131    0 404.5kb 404.5kb
yellow open bitbucket-index-version iedAk2sxQtW-qPgVLyoGaA 5 1      1    0   3.9kb   3.9kb

See also:

https://confluence.atlassian.com/bitbucketserverkb/elasticsearch-has-got-corrupted-index-861012994.html

[jetune@kube-cloud.com]

Thank's for your answer,

But I have not any Elasticsearch installed (I don't know if Bitbucket Install it for me), so that i can't invoke the verification URL

http://<ip of elastic search instance>:9200/_cat/indices

Anyway, whn I try to re-index with this URL template

curl -u <admin user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' https://bitbucket.mycompany.com/rest/indexing/latest/sync

I have this error

2020-04-02 11:01:10,624 INFO [http-nio-7990-exec-2] jetune @ISLY8Jx661x12x0 192.168.1.254,172.16.1.21 "POST /rest/indexing/latest/sync HTTP/1.0" c.a.b.i.s.c.c.DefaultClusterJobManager Attempting to schedule an ad-hoc for ElasticsearchSynchronizeJob
2020-04-02 11:01:10,625 INFO [http-nio-7990-exec-2] jetune @ISLY8Jx661x12x0 192.168.1.254,172.16.1.21 "POST /rest/indexing/latest/sync HTTP/1.0" c.a.b.i.s.c.c.DefaultClusterJobManager There are currently 2 to select from
2020-04-02 11:01:10,662 INFO [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Beginning indexing of all projects and repositories for search.
2020-04-02 11:01:10,671 INFO [Caesium-1-4] c.a.b.i.s.i.u.DefaultUpgradeService Executing upgrade task:[Update path and filename fields for file search]
2020-04-02 11:01:16,885 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV4Point0 Failed to recreate index
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:74)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.fastPath(OnSubscribeFromArray.java:76)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.request(OnSubscribeFromArray.java:58)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:32)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:24)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at rx.Observable.subscribe(Observable.java:10271)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:34)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV4Point0.execute(UpgradeTaskV4Point0.java:39)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob.run(ElasticsearchSynchronizeJob.java:51)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: null
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
... 1 common frames omitted
2020-04-02 11:01:16,886 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV4Point0 Failed to recreate indexes: ''[java.net.SocketTimeoutException]''
2020-04-02 11:01:16,887 ERROR [Caesium-1-4] c.a.b.i.s.i.u.DefaultUpgradeService Upgrade task failed:[Update path and filename fields for file search]
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Failed to recreate index: [java.net.SocketTimeoutException]
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV4Point0.execute(UpgradeTaskV4Point0.java:52)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob.run(ElasticsearchSynchronizeJob.java:51)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-04-02 11:01:16,887 ERROR [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Upgrading index failed, unable to continue, operator intervention is required.
2020-04-02 11:01:16,888 WARN [Caesium-1-4] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob Job id: 6e6125e9-926c-40f4-b087-f45889ed2072

I've try to stop bitbucket server, delete after backup the directory

<bitbucket-home>/shared/search/data/nodes

and restart bitbucket service, I get the same error when it try to build index...