What to do when it states java.lang.OutOfMemoryError: unable to create new native thread

[Frank Troglauer]

I am getting this error when Bitbucket tries to start up. 

2020-05-27 17:14:54,931 INFO [main] o.s.boot.SpringApplication
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.2.4.RELEASE)

2020-05-27 17:14:55,289 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on ip-172-31-33-69 with PID 7105 (/opt/atlassian/bitbucket/7.1.1/app/WEB-INF/classes started by atlbitbucket in /opt/atlassian/bitbucket/7.1.1/bin)
2020-05-27 17:14:55,294 INFO [main] c.a.b.i.b.BitbucketServerApplication The following profiles are active: nio
2020-05-27 17:14:56,890 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 7.1.1 (3342f37 built on Wed Apr 08 05:20:52 UTC 2020)
2020-05-27 17:14:56,890 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: AdoptOpenJDK OpenJDK 64-Bit Server VM 1.8.0_242-b08
2020-05-27 17:14:59,890 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 8.34 seconds (JVM running for 10.985)
2020-05-27 17:15:05,752 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
2020-05-27 17:15:13,143 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket/shared
2020-05-27 17:15:15,712 WARN [spring-startup] com.hazelcast.instance.Node [172.31.33.69]:5701 [atlbitbucket] [3.12.6] No join method is enabled! Starting standalone.
2020-05-27 17:15:36,732 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:55695
2020-05-27 17:15:39,828 ERROR [hz.hazelcast.scheduled.thread-] com.hazelcast.spi.ExecutionService [172.31.33.69]:5701 [atlbitbucket] [3.12.6] Failed to execute java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@28930c5e
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at com.hazelcast.util.executor.CachedExecutorServiceDelegate.addNewWorkerIfRequired(CachedExecutorServiceDelegate.java:146)
at com.hazelcast.util.executor.CachedExecutorServiceDelegate.execute(CachedExecutorServiceDelegate.java:116)
at com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator.run(DelegateAndSkipOnConcurrentExecutionDecorator.java:52)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
... 1 frame trimmed
2020-05-27 17:18:54,489 INFO [main] o.s.boot.SpringApplication

This is on an ubuntu instance. I am trying to run jstack to help track the issue but jstack doesn't work.  
((This is the article https://blog.fastthread.io/2016/06/06/how-to-take-thread-dumps-7-options/  ))

jstack -l 1488 jstackoutput.txt
Attaching to core jstackoutput.txt from executable 1488, please wait...
Error attaching to core file: Can't attach to the core file

Not sure what to do to track the issue. Would appreciate any help here.

The previous message was in the /var/atlassian/application-data/bitbucket/log/atlassian-bitbuck-2020-05-27.log file. The below is from the atlassian-bitbucket.log file from the same location. 

2020-05-28 03:27:14,206 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [pluginProvidedAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [goGetFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,233 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,240 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,241 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,241 WARN [http-nio-7990-exec-2] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,692 WARN [http-nio-7990-exec-3] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,692 WARN [http-nio-7990-exec-3] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,808 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [pluginProvidedAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [goGetFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2020-05-28 03:27:14,809 WARN [http-nio-7990-exec-4] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available

The below is from launcher.log

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f007ee70000, 65536, 1) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/hs_err_pid10669.log
#
# Compiler replay data is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/replay_pid10669.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000101133000, 131072, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 131072 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/hs_err_pid7105.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f1c6b6fd000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/atlassian/bitbucket/7.1.1/bin/hs_err_pid7433.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f1c695ea000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 139760004671232 also had an error]
2020-05-27 17:27:01
Full thread dump OpenJDK 64-Bit Server VM (25.242-b08 mixed mode):

"SpringContextShutdownHook" #29 prio=5 os_prio=0 tid=0x00007fce4004b800 nid=0x1ed7 in Object.wait() [0x00007fce293e9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000c0def1f0> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1326)
at com.atlassian.johnson.spring.lifecycle.LifecycleDispatcherServlet.destroy(LifecycleDispatcherServlet.java:55)
at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1291)
- locked <0x00000000c0fbec38> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapper.stopInternal(StandardWrapper.java:1592)
- locked <0x00000000c0fbec38> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0fbec38> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5389)
- locked <0x00000000c0afc2f8> (a org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedContext)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afc2f8> (a org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedContext)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
- locked <0x00000000c0afc678> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afc678> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)

at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
- locked <0x00000000c0afcb50> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afcb50> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:473)
- locked <0x00000000c0afcb50> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c0afd268> (a org.apache.catalina.core.StandardService)
at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:992)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
- locked <0x00000000c04973e0> (a org.apache.catalina.core.StandardServer)
at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:478)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.stopTomcat(TomcatWebServer.java:269)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.stop(TomcatWebServer.java:324)
- locked <0x00000000c0921960> (a java.lang.Object)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.stopAndReleaseWebServer(ServletWebServerApplicationContext.java:306)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onClose(ServletWebServerApplicationContext.java:172)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1035)
at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:948)
- locked <0x00000000c0925d50> (a java.lang.Object)

"SIGTERM handler" #37 daemon prio=9 os_prio=0 tid=0x00007fce2c002800 nid=0x1ed4 in Object.wait() [0x00007fce29bec000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000c0fa4e30> (a org.springframework.context.support.AbstractApplicationContext$1)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000000c0423428> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
Picked up _JAVA_OPTIONS: -Xmx2048m
Picked up _JAVA_OPTIONS: -Xmx2048m
Picked up _JAVA_OPTIONS: -Xmx2048m
Picked up _JAVA_OPTIONS: -Xmx2048m

Here is a screenshot showing memory usage using top command. 

https://prnt.sc/sp55h0

[Christian Glockner]

Hi Frank,

If it's unable to create a new native thread it means the system is actually out of physical RAM and not out of heap, so running jstack won't help.

How much RAM do you have on the machine and what is the size of the heap that you have configured?

Cheers,

Christian

Premier Support Engineer

Atlassian

[Frank Troglauer]

We increased the instance size to one with double the memory and the issue remained. The fix was to manually assign the project directory in the bitbucket settings file. Afterword it seems to be working mostly, though we are having issues getting a sql server installed for it to use. If it isn't one thing it is something else. Thanks!