Errors related to LocalQCache when production database copied to development

[Peter Macdonald]

We have a two node DataCenter production environment running 7.13.1. In order to do some testing of JIRA 8.1.0 in development, I had the database copied to our development environment which is also running 7.13.1. However, when I started this to confirm everything was running okay, I am getting a swag of error messages (192 of them) being log every 6 seconds. The first 159 messages are: (this is restricted by the 20,000 limit of chars)

{code}

2019-05-15 16:07:17,092 localq-reader-0 ERROR [c.a.j.c.distribution.localq.LocalQCacheOpReader] Abandoning sending: LocalQCacheOp{cacheName='com.qotilabs.jira.rich-filters-plugin:RichFiltersManagerCachingImpl.rich-filters-cache', action=REMOVE_ALL, key=null, value=null, creationTimeInMillis=1557900244210} from cache replication queue: [queueId=queue_myprodnode2_5_a8c20a6e2f238665a766a1905dead885, queuePath=/var/atlassian/application-data/jira/localq/queue_myprodnode2_5_a8c20a6e2f238665a766a1905dead885], failuresCount: 10/10. Removing from queue. Error: java.rmi.UnknownHostException: Unknown host: myprodnode2.domainname.com; nested exception is:
java.net.UnknownHostException: myprodnode2.domainname.com
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$UnrecoverableFailure: java.rmi.UnknownHostException: Unknown host: myprodnode2.domainname.com; nested exception is:
java.net.UnknownHostException: myprodnode2.domainname.com
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:88)
at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.UnknownHostException: Unknown host: myprodnode2.domainname.com; nested exception is:
java.net.UnknownHostException: myprodnode2.domainname.com
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
... 6 more
Caused by: java.net.UnknownHostException: myprodnode2.domainname.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocketWithTimeout(ConfigurableRMIClientSocketFactory.java:97)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:87)
... 15 more
2019-05-15 16:07:17,092 localq-reader-1 ERROR [c.a.j.c.distribution.localq.LocalQCacheOpReader] Abandoning sending: LocalQCacheOp{cacheName='com.qotilabs.jira.rich-filters-plugin:RichFiltersManagerCachingImpl.rich-filters-cache', action=REMOVE_ALL, key=null, value=null, creationTimeInMillis=1557900244210} from cache replication queue: [queueId=queue_myprodnode1_5_aec385c61e827fe258aaddcaba973e02, queuePath=/var/atlassian/application-data/jira/localq/queue_myprodnode1_5_aec385c61e827fe258aaddcaba973e02], failuresCount: 10/10. Removing from queue. Error: java.rmi.UnknownHostException: Unknown host: myprodnode1.comainname.com; nested exception is:
java.net.UnknownHostException: myprodnode1.comainname.com
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$UnrecoverableFailure: java.rmi.UnknownHostException: Unknown host: myprodnode1.comainname.com; nested exception is:
java.net.UnknownHostException: myprodnode1.comainname.com
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:88)
at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.UnknownHostException: Unknown host: myprodnode1.comainname.com; nested exception is:
java.net.UnknownHostException: myprodnode1.comainname.com
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
... 6 more
Caused by: java.net.UnknownHostException: myprodnode1.comainname.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocketWithTimeout(ConfigurableRMIClientSocketFactory.java:97)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:87)
... 15 more
2019-05-15 16:07:17,095 localq-reader-0 WARN [c.a.j.c.distribution.localq.LocalQCacheOpReader] Runtime exception: UnrecoverableFailure occurred when processing: LocalQCacheOp{cacheName='com.qotilabs.jira.rich-filters-plugin:RichFiltersManagerCachingImpl.rich-filters-cache', action=REMOVE_ALL, key=null, value=null, creationTimeInMillis=1557900244263} from cache replication queue: [queueId=queue_myprodnode2_5_a8c20a6e2f238665a766a1905dead885, queuePath=/var/atlassian/application-data/jira/localq/queue_myprodnode2_5_a8c20a6e2f238665a766a1905dead885], failuresCount: 1/10, error: java.rmi.UnknownHostException: Unknown host: myprodnode2.domainname.com; nested exception is:
java.net.UnknownHostException: myprodnode2.domainname.com
2019-05-15 16:07:17,095 localq-reader-1 WARN [c.a.j.c.distribution.localq.LocalQCacheOpReader] Runtime exception: UnrecoverableFailure occurred when processing: LocalQCacheOp{cacheName='com.qotilabs.jira.rich-filters-plugin:RichFiltersManagerCachingImpl.rich-filters-cache', action=REMOVE_ALL, key=null, value=null, creationTimeInMillis=1557900244263} from cache replication queue: [queueId=queue_myprodnode1_5_aec385c61e827fe258aaddcaba973e02, queuePath=/var/atlassian/application-data/jira/localq/queue_myprodnode1_5_aec385c61e827fe258aaddcaba973e02], failuresCount: 1/10, error: java.rmi.UnknownHostException: Unknown host: myprodnode1.comainname.com; nested exception is:
java.net.UnknownHostException: myprodnode1.comainname.com
2019-05-15 16:07:17,566 localq-reader-5 ERROR [c.a.j.c.distribution.localq.LocalQCacheOpReader] Abandoning sending: LocalQCacheOp{cacheName='UpmRoleBasedMetadataCache', action=REMOVE_ALL, key=null, value=null, creationTimeInMillis=1557900272503} from cache replication queue: [queueId=queue_myprodnode2_4_a8c20a6e2f238665a766a1905dead885, queuePath=/var/atlassian/application-data/jira/localq/queue_myprodnode2_4_a8c20a6e2f238665a766a1905dead885], failuresCount: 10/10. Removing from queue. Error: java.rmi.UnknownHostException: Unknown host: myprodnode2.domainname.com; nested exception is:
java.net.UnknownHostException: myprodnode2.domainname.com
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$UnrecoverableFailure: java.rmi.UnknownHostException: Unknown host: myprodnode2.domainname.com; nested exception is:
java.net.UnknownHostException: myprodnode2.domainname.com
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:88)
at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.UnknownHostException: Unknown host: myprodnode2.domainname.com; nested exception is:
java.net.UnknownHostException: myprodnode2.domainname.com
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
... 6 more
Caused by: java.net.UnknownHostException: myprodnode2.domainname.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocketWithTimeout(ConfigurableRMIClientSocketFactory.java:97)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:87)
... 15 more
2019-05-15 16:07:17,569 localq-reader-5 WARN [c.a.j.c.distribution.localq.LocalQCacheOpReader] Runtime exception: UnrecoverableFailure occurred when processing: LocalQCacheOp{cacheName='HostLicenseCache', action=REMOVE, key=com.atlassian.cache.impl.ReferenceKey@0, value=null, creationTimeInMillis=1557900272510} from cache replication queue: [queueId=queue_myprodnode2_4_a8c20a6e2f238665a766a1905dead885, queuePath=/var/atlassian/application-data/jira/localq/queue_myprodnode2_4_a8c20a6e2f238665a766a1905dead885], failuresCount: 1/10, error: java.rmi.UnknownHostException: Unknown host: myprodnode2.domainname.com; nested exception is:
java.net.UnknownHostException: myprodnode2.domainname.com
2019-05-15 16:07:17,577 localq-reader-13 ERROR [c.a.j.c.distribution.localq.LocalQCacheOpReader] Abandoning sending: LocalQCacheOp{cacheName='UpmRoleBasedMetadataCache', action=REMOVE_ALL, key=null, value=null, creationTimeInMillis=1557900272503} from cache replication queue: [queueId=queue_myprodnode1_4_aec385c61e827fe258aaddcaba973e02, queuePath=/var/atlassian/application-data/jira/localq/queue_myprodnode1_4_aec385c61e827fe258aaddcaba973e02], failuresCount: 10/10. Removing from queue. Error: java.rmi.UnknownHostException: Unknown host: myprodnode1.comainname.com; nested exception is:
java.net.UnknownHostException: myprodnode1.comainname.com
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$UnrecoverableFailure: java.rmi.UnknownHostException: Unknown host: myprodnode1.comainname.com; nested exception is:
java.net.UnknownHostException: myprodnode1.comainname.com
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:88)
at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.UnknownHostException: Unknown host: myprodnode1.comainname.com; nested exception is:
java.net.UnknownHostException: myprodnode1.comainname.com
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
... 6 more
Caused by: java.net.UnknownHostException: myprodnode1.comainname.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocketWithTimeout(ConfigurableRMIClientSocketFactory.java:97)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:87)
... 15 more
2019-05-15 16:07:17,580 localq-reader-13 WARN [c.a.j.c.distribution.localq.LocalQCacheOpReader] Runtime exception: UnrecoverableFailure occurred when processing: LocalQCacheOp{cacheName='HostLicenseCache', action=REMOVE, key=com.atlassian.cache.impl.ReferenceKey@0, value=null, creationTimeInMillis=1557900272510} from cache replication queue: [queueId=queue_myprodnode1_4_aec385c61e827fe258aaddcaba973e02, queuePath=/var/atlassian/application-data/jira/localq/queue_myprodnode1_4_aec385c61e827fe258aaddcaba973e02], failuresCount: 1/10, error: java.rmi.UnknownHostException: Unknown host: myprodnode1.comainname.com; nested exception is:
java.net.UnknownHostException: myprodnode1.comainname.com
2019-05-15 16:07:18,096 localq-reader-15 ERROR [c.a.j.c.distribution.localq.LocalQCacheOpReader] Abandoning sending: LocalQCacheOp{cacheName='com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUserCache.userCache', action=PUT, key={1,59182072-77097643699}, value=59182072-77097643699:1, creationTimeInMillis=1557900262479} from cache replication queue: [queueId=queue_myprodnode1_7_aec385c61e827fe258aaddcaba973e02, queuePath=/var/atlassian/application-data/jira/localq/queue_myprodnode1_7_aec385c61e827fe258aaddcaba973e02], failuresCount: 10/10. Removing from queue. Error: java.rmi.UnknownHostException: Unknown host: myprodnode1.comainname.com; nested exception is:
java.net.UnknownHostException: myprodnode1.comainname.com
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$UnrecoverableFailure: java.rmi.UnknownHostException: Unknown host: myprodnode1.comainname.com; nested exception is:
java.net.UnknownHostException: myprodnode1.comainname.com
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:88)
at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:84)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.UnknownHostException: Unknown host: myprodnode1.comainname.com; nested exception is:
java.net.UnknownHostException: myprodnode1.comainname.com
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.lookupRemoteCachePeer(BasicRMICachePeerProvider.java:64)
at com.atlassian.jira.cluster.distribution.localq.rmi.BasicRMICachePeerProvider.create(BasicRMICachePeerProvider.java:39)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.getCachePeerFor(CachingRMICachePeerManager.java:58)
at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:91)
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:63)
... 6 more
Caused by: java.net.UnknownHostException: myprodnode1.comainname.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocketWithTimeout(ConfigurableRMIClientSocketFactory.java:97)
at net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory.createSocket(ConfigurableRMIClientSocketFactory.java:87)
... 15 more

{code}

How do I resolve this in our development system so I can concentrate on the JIRA 8.1.0 testing?

[Shankar Asam {Appfire}]

Hi,

I just want to add my notes here as I resolved this issue recently. The issue caused because of this below exception (from your logs)

 java.net.UnknownHostException: myprodnode2.domainname.com

java.net.UnknownHostException: myprodnode1.comainname.com

Make sure to resolve your hostnames of the nodes in the cluster. The easy way is to put the IPaddress and hostnames in  /etc/hosts of all your nodes and then restart the JIRA service. For us, the cluster cache replication and RMI related errors were gone.

Thanks