Confluence 5.8.18 crashing after schedulerError Job

[Frederik Frey]

Hello,

since some weeks I'm getting the following error for like 100 times and then the confluence instance is crashing and rebooting (sometimes).

ERROR [scheduler_Worker-3] [org.quartz.core.ErrorLogger] schedulerError Job (DEFAULT.IndexQueueFlusher threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: com.atlassian.bonnie.LuceneException: Error refreshing index searcher]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:234)
	at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool.lambda$runInThread$152(ConfluenceQuartzThreadPool.java:19)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: com.atlassian.bonnie.LuceneException: Error refreshing index searcher
	at com.atlassian.bonnie.LuceneConnection.commitAndRefreshSearcher(LuceneConnection.java:561)
	at com.atlassian.bonnie.LuceneConnection.withBatchUpdate(LuceneConnection.java:532)
	at com.atlassian.confluence.search.lucene.DefaultConfluenceIndexManager.processTasks(DefaultConfluenceIndexManager.java:199)
	at com.atlassian.confluence.search.lucene.DefaultConfluenceIndexManager.flushQueue(DefaultConfluenceIndexManager.java:144)
	at sun.reflect.GeneratedMethodAccessor649.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at com.sun.proxy.$Proxy69.flushQueue(Unknown Source)
	at com.atlassian.confluence.search.lucene.IndexQueueFlusher.executeJob(IndexQueueFlusher.java:32)
	at com.atlassian.confluence.setup.quartz.AbstractClusterAwareQuartzJobBean.surroundJobExecutionWithLogging(AbstractClusterAwareQuartzJobBean.java:75)
	at com.atlassian.confluence.setup.quartz.AbstractClusterAwareQuartzJobBean.executeInternal(AbstractClusterAwareQuartzJobBean.java:56)
	at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
	at com.atlassian.scheduler.quartz1.Quartz1JobFactory$ClassLoaderProtectingWrappedJob.execute(Quartz1JobFactory.java:62)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:223)
	... 2 more
Caused by: java.io.IOException: Map failed
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
	at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:283)
	at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:228)
	at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:195)
	at org.apache.lucene.store.MMapDirectory.createSlicer(MMapDirectory.java:203)
	at org.apache.lucene.store.CompoundFileDirectory.<init>(CompoundFileDirectory.java:102)
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:116)
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:56)
	at org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121)
	at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:269)
	at org.apache.lucene.index.IndexWriter.applyAllDeletes(IndexWriter.java:3036)
	at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3027)
	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2757)
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2897)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2872)
	at com.atlassian.bonnie.LuceneConnection.commitAndRefreshSearcher(LuceneConnection.java:556)
	... 21 more
Caused by: java.lang.OutOfMemoryError: Map failed
	at sun.nio.ch.FileChannelImpl.map0(Native Method)
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
	... 36 more
2017-06-01 11:49:31,253 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 5.8.18

Sometimes it is not starting again, that's why we recognized this as a problem...

Any idea where this is coming from?
Thanks for your help!

[AnnWorley]

This article is for JIRA but as the issue is with Java memory and the OS, it applies to Confluence as well:

JIRA crashes with java.io.IOException: Map fail

[Frederik Frey]

Thanks for your reply, I have already added this to my startup script and it is still occuring...