JIRA Freezes with DB deadlock

[Devu Heda]

Hi recently moved to JIRA 446 and JIRA freezes because of the following deadlocks ...one another observation ...JIRA 402 worked with 100 Active DB Instances in our system and now we had to bump it 300 as we were running out of DB Connections leading me to beleive JIRA is having issues with DB Connection Pool handling

Any thoughts around this and how this can be resolved ... working with support as well ...but wanted to reach out to the community

Thanks

{code}

"StreamsCompletionService::thread-2232" prio=10 tid=0x00002aaabca3b000 nid=0x756c waiting for monitor entry [0x0000000059240000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:300)
- waiting to lock <0x000000058001b568> (a org.apache.commons.dbcp.PoolableConnectionFactory)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:816)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:153)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:150)
at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:53)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.trackConnection(DBCPConnectionFactory.java:149)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:69)
at org.ofbiz.core.entity.ConnectionFactory.tryGenericConnectionSources(ConnectionFactory.java:69)
at org.ofbiz.core.entity.transaction.JNDIFactory.getConnection(JNDIFactory.java:146)
at org.ofbiz.core.entity.TransactionFactory.getConnection(TransactionFactory.java:101)
at org.ofbiz.core.entity.ConnectionFactory.getConnection(ConnectionFactory.java:59)
at org.ofbiz.core.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:367)
at org.ofbiz.core.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:515)
at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:1034)
at org.ofbiz.core.entity.GenericHelperDAO.findListIteratorByCondition(GenericHelperDAO.java:175)
at org.ofbiz.core.entity.GenericDelegator.findListIteratorByCondition(GenericDelegator.java:1001)
at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.atlassian.multitenant.impl.MultiTenantComponentFactoryImpl$AbstractMultiTenantAwareInvocationHandler.invokeInternal(MultiTenantComponentFactoryImpl.java:181)
at com.atlassian.multitenant.impl.MultiTenantComponentFactoryImpl$MultiTenantAwareMethodInterceptor.intercept(MultiTenantComponentFactoryImpl.java:230)
at org.ofbiz.core.entity.GenericDelegator$$EnhancerByCGLIB$$7d4fdd03.findListIteratorByCondition(<generated>)
at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findListIteratorByCondition(DefaultOfBizDelegator.java:464)
at com.atlassian.jira.issue.changehistory.DefaultChangeHistoryManager.doFindUserHistory(DefaultChangeHistoryManager.java:345)
at com.atlassian.jira.issue.changehistory.DefaultChangeHistoryManager.findUserHistory(DefaultChangeHistoryManager.java:259)
at sun.reflect.GeneratedMethodAccessor620.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

{code}

[Devu Heda · Accepted solution]

Thanks to Jamie...the java melody plugin is awesome ... we found it was issue with the HTTP threads and not actually DB threads ...

As suggested by Jamie we also reduced the number of threads

Thanks!!

[JamieA]

300 is too many, 100 is too many... you have a plugin that is leaking database connections.

If you install the java melody plugin, it will show you all the database connections, along with stacktraces from what created the connection. Using this, you can find out which plugin (it probably is a plugin) is creating them.

[Devu Heda]

Thanks Jamie,

I have installed the plugin to try out on my dev instance ....any idea what are the log4j properties to configure tracing for this plugin