Re: Elasticsearche error after upgrade

[Danzel Washington]

Hello! I will not create a separate subject. I have exactly the same problem, only I don't know when it started. I came to a new firm and the bitbucket server here has a problem with elasticsearch a long time. I tried deleting the contents of the /data/ folder and tried executing curl from this article (https://confluence.atlassian.com/bitbucketserverkb/rebuild-the-bundled-elasticsearch-indexes-for-bitbucket-server-833988382.html), nothing helps. atlassian-bitbucket.log contains the following entries

2019-10-11 06:17:49,727 INFO [https-jsse-nio-6443-exec-5] admin @5D64KPx377x170x0 10.22.2.107 "POST /rest/indexing/latest/sync HTTP/1.1" c.a.b.i.s.c.c.DefaultClusterJobManager Attempting to schedule an ad-hoc for ElasticsearchSynchronizeJob
2019-10-11 06:17:49,727 INFO [https-jsse-nio-6443-exec-5] admin @5D64KPx377x170x0 10.22.2.107 "POST /rest/indexing/latest/sync HTTP/1.1" c.a.b.i.s.c.c.DefaultClusterJobManager There are currently 2 to select from
2019-10-11 06:17:54,778 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV3Point0 Failed to recreate index
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:74)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.fastPath(OnSubscribeFromArray.java:76)
at rx.internal.operators.OnSubscribeFromArray$FromArrayProducer.request(OnSubscribeFromArray.java:58)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:32)
at rx.internal.operators.OnSubscribeFromArray.call(OnSubscribeFromArray.java:24)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:75)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at rx.Observable.subscribe(Observable.java:10271)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:34)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV3Point0.execute(UpgradeTaskV3Point0.java:39)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob.run(ElasticsearchSynchronizeJob.java:51)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: null
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
... 1 common frames omitted
2019-10-11 06:17:54,778 ERROR [Caesium-1-4] c.a.b.i.s.i.u.UpgradeTaskV3Point0 Failed to recreate indexes: ''[java.net.SocketTimeoutException]''
2019-10-11 06:17:54,778 ERROR [Caesium-1-4] c.a.b.i.s.i.u.DefaultUpgradeService Upgrade task failed:[Create indexes for each mapping, this is required for ES 6.x support.]
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Failed to recreate index: [java.net.SocketTimeoutException]
at com.atlassian.bitbucket.internal.search.indexing.upgrade.UpgradeTaskV3Point0.execute(UpgradeTaskV3Point0.java:52)
at com.atlassian.bitbucket.internal.search.indexing.upgrade.DefaultUpgradeService.upgrade(DefaultUpgradeService.java:45)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:112)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:84)
at com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob.run(ElasticsearchSynchronizeJob.java:51)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:81)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.lang.Thread.run(Thread.java:748)
2019-10-11 06:17:54,779 ERROR [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Upgrading index failed, unable to continue, operator intervention is required.
2019-10-11 06:17:54,779 WARN [Caesium-1-4] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.ElasticsearchSynchronizeJob Job id: 4bfd826a-9a29-43ed-9c0c-2abda506f4a0

And in bitbucket_search.log everything normal

[2019-10-11T06:07:27,013][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-index-state/chgpoeVfQwK9n1RfOU5J7A] deleting index
[2019-10-11T06:07:27,102][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-search/eq7EFIlEQGOlw2kxQOuQlg] deleting index
[2019-10-11T06:07:27,192][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-project/QopFWJBUST2kTYPYDq9F5Q] deleting index
[2019-10-11T06:07:27,292][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-repository] creating index, cause [api], templates [], shards [5]/[1], mappings [repository]
[2019-10-11T06:07:28,059][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-project] creating index, cause [api], templates [], shards [5]/[1], mappings [project]
[2019-10-11T06:07:28,782][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-index-state] creating index, cause [api], templates [], shards [5]/[1], mappings [indexstate]
[2019-10-11T06:07:28,847][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-search] creating index, cause [api], templates [], shards [5]/[1], mappings [file]
[2019-10-11T06:08:32,311][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-repository/BqWxHO0MSlWSeAUgrNtryw] deleting index
[2019-10-11T06:08:32,424][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-project/PJ4LhwXTTH-5fWrqr9Kz9w] deleting index
[2019-10-11T06:08:32,553][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-index-state/PpQSeln-R8WbMHrXc2ic2w] deleting index
[2019-10-11T06:08:32,653][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-search/I8nCq2skSD-NBXXh3zj9Hw] deleting index
[2019-10-11T06:08:32,755][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-repository] creating index, cause [api], templates [], shards [5]/[1], mappings [repository]
[2019-10-11T06:08:33,750][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-index-state] creating index, cause [api], templates [], shards [5]/[1], mappings [indexstate]
[2019-10-11T06:08:34,705][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-project] creating index, cause [api], templates [], shards [5]/[1], mappings [project]
[2019-10-11T06:08:34,785][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-search] creating index, cause [api], templates [], shards [5]/[1], mappings [file]
[2019-10-11T06:10:37,767][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-repository/954VJoPmTSuvjcmYHS-Kpw] deleting index
[2019-10-11T06:10:37,894][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-project/IO_WozjPS-28ASmdD6Rl_w] deleting index
[2019-10-11T06:10:37,998][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-index-state/pBmJ997wQGG76aJJxAct5w] deleting index
[2019-10-11T06:10:38,098][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-search/VY_Q-DomR7qfllvaAMef2w] deleting index
[2019-10-11T06:10:38,205][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-repository] creating index, cause [api], templates [], shards [5]/[1], mappings [repository]
[2019-10-11T06:10:38,977][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-project] creating index, cause [api], templates [], shards [5]/[1], mappings [project]
[2019-10-11T06:10:39,397][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-index-state] creating index, cause [api], templates [], shards [5]/[1], mappings [indexstate]
[2019-10-11T06:10:39,893][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-search] creating index, cause [api], templates [], shards [5]/[1], mappings [file]
[2019-10-11T06:14:38,534][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-repository/m0nMk9H4TbmkJau6OgGSDg] deleting index
[2019-10-11T06:14:38,614][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-index-state/xlErrMpcRG-OjTP4R_aZMQ] deleting index
[2019-10-11T06:14:38,693][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-project/6NEq4RP1Q8y_Cz6972ReCA] deleting index
[2019-10-11T06:14:38,774][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-search/jjMgUuqgR1aCZL2t0QfGLg] deleting index
[2019-10-11T06:14:38,859][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-repository] creating index, cause [api], templates [], shards [5]/[1], mappings [repository]
[2019-10-11T06:14:39,638][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-index-state] creating index, cause [api], templates [], shards [5]/[1], mappings [indexstate]
[2019-10-11T06:14:40,017][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-project] creating index, cause [api], templates [], shards [5]/[1], mappings [project]
[2019-10-11T06:14:40,388][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-search] creating index, cause [api], templates [], shards [5]/[1], mappings [file]
[2019-10-11T06:15:43,885][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-repository/oAjB9dD8QhC3PSCUw_hUZA] deleting index
[2019-10-11T06:15:43,998][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-project/kYtrgYpjQZWd6OfaxwFugA] deleting index
[2019-10-11T06:15:44,092][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-index-state/rKlzaBAySU23l9GWAWPR-g] deleting index
[2019-10-11T06:15:44,229][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-search/PqWo_Qq_StKPljQTNHiDZA] deleting index
[2019-10-11T06:15:44,321][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-repository] creating index, cause [api], templates [], shards [5]/[1], mappings [repository]
[2019-10-11T06:15:45,069][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-project] creating index, cause [api], templates [], shards [5]/[1], mappings [project]
[2019-10-11T06:15:45,853][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-index-state] creating index, cause [api], templates [], shards [5]/[1], mappings [indexstate]
[2019-10-11T06:15:45,893][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-search] creating index, cause [api], templates [], shards [5]/[1], mappings [file]
[2019-10-11T06:17:49,339][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-repository/hx42f4n7RbOAZo-r_FhgMA] deleting index
[2019-10-11T06:17:49,445][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-project/GezdTFxRTI2kAfQM-gXQhA] deleting index
[2019-10-11T06:17:49,567][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-index-state/MzenSNG4QwSIZBWuvQ7YfQ] deleting index
[2019-10-11T06:17:49,665][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-search/ziXcCgtgQDqoQqnAbaD8dg] deleting index
[2019-10-11T06:17:49,778][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-repository] creating index, cause [api], templates [], shards [5]/[1], mappings [repository]
[2019-10-11T06:17:50,506][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-project] creating index, cause [api], templates [], shards [5]/[1], mappings [project]
[2019-10-11T06:17:51,354][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-index-state] creating index, cause [api], templates [], shards [5]/[1], mappings [indexstate]
[2019-10-11T06:17:51,401][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-search] creating index, cause [api], templates [], shards [5]/[1], mappings [file]
[2019-10-11T06:21:54,803][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-repository/zVgBhCxkQpiD8oJ-l1HPYg] deleting index
[2019-10-11T06:21:54,935][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-project/M0eKufYkRH2BTjgggJe1nA] deleting index
[2019-10-11T06:21:55,015][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-index-state/rMWxFnxhS-S9dDdT9r_tSw] deleting index
[2019-10-11T06:21:55,091][INFO ][o.e.c.m.MetaDataDeleteIndexService] [bitbucket_bundled][bitbucket-search/HNUiDOGKRFmYNMmCHAJG5g] deleting index
[2019-10-11T06:21:55,198][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-repository] creating index, cause [api], templates [], shards [5]/[1], mappings [repository]
[2019-10-11T06:21:55,779][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-project] creating index, cause [api], templates [], shards [5]/[1], mappings [project]
[2019-10-11T06:21:56,653][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-index-state] creating index, cause [api], templates [], shards [5]/[1], mappings [indexstate]
[2019-10-11T06:21:56,747][INFO ][o.e.c.m.MetaDataCreateIndexService] [bitbucket_bundled][bitbucket-search] creating index, cause [api], templates [], shards [5]/[1], mappings [file]