JIRA sending one email notification at a time

[Dan Hartnack]

We recently upgraded to JIRA 5.0 from 4.1. After doing this we noticed that when someone creates/updates an issue, JIRA sends out the notification emails one at a time in intervals of about 15 seconds. We did nothing different when setting up 5.0 with mail handlers and the SMTP mail server settings. We did migrate at this time to a new exchange server. We have tried many things to JIRA to send out the emails in bulk but nothing is working. Any help would be appreciated.

[Radu Dumitriu]

Can you check this and see if it applies to your situation ?

http://technet.microsoft.com/en-us/library/bb232205.aspx

I think it's message throttling that is causing you all the trouble ....

[Dan Hartnack]

No, we can confirm that this issue has nothing to do with Exchange. We can see the emails being generated from JIRA in real time coming into exchange in approx. 15 second intervals which exchange is handling and sending out immediatly as it should. We need JIRA to be sending the notifications in bulk to exchange which is not happening.

[Dan Hartnack]

I see in this in the logs:

2012-07-19 14:10:09,924 ERROR [scheduler_Worker-3] [atlassian.crowd.directory.DbCachingDirectoryPoller] pollChanges Error occurred while refreshing the cache for directory [ 13959170 ].
com.atlassian.crowd.exception.OperationFailedException: Unable to synchronise directory: duplicate groups with name 'Exchange Domain Servers'
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseMemberships(AbstractCacheRefresher.java:133)
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseAll(AbstractCacheRefresher.java:44)
at com.atlassian.crowd.directory.ldap.cache.UsnChangedCacheRefresher.synchroniseAll(UsnChangedCacheRefresher.java:223)
at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:621)
at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:63)
at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:50)
at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobBean.executeInternal(DirectoryPollerJobBean.java:29)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool$1.run(ConfluenceQuartzThreadPool.java:20)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
2012-07-19 14:12:04,437 WARN [StreamsCompletionService::thread-8443] [atlassian.streams.internal.ActivityProviderConnectionMonitorImpl$ActivityMonitorJob] call Could not reach provider Release Management CI Instance; it will be omitted from the stream. The connection will be retried in 5 minutes

Could this be causing a problem?

[Radu Dumitriu]

Man, this is something completely different.

https://confluence.atlassian.com/display/JIRAKB/%27Unable+to+synchronise+directory%27+Due+to+Duplicate+Groups+in+an+External+Directory

[Dan Hartnack]

com.atlassian.confluence.setup.quartz.AbstractClusterAwareQuartzJobBean.executeInternal(AbstractClusterAwareQuartzJobBean.java:46)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool$1.run(ConfluenceQuartzThreadPool.java:20)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: javax.mail.AuthenticationFailedException
at javax.mail.Service.connect(Service.java:319)
at javax.mail.Service.connect(Service.java:169)
at javax.mail.Service.connect(Service.java:118)
at javax.mail.Transport.send0(Transport.java:188)
at javax.mail.Transport.send(Transport.java:118)
at alt.javax.mail.TransportImpl.send(TransportImpl.java:18)
at com.atlassian.mail.server.impl.SMTPMailServerImpl.send(SMTPMailServerImpl.java:177)
... 17 more
2012-07-19 12:37:04,437 WARN [StreamsCompletionService::thread-8386] [atlassian.streams.internal.ActivityProviderConnectionMonitorImpl$ActivityMonitorJob] call Could not reach provider Release Management CI Instance; it will be omitted from the stream. The connection will be retried in 5 minutes

[Dan Hartnack]

I did some more digging and found this error:

2012-07-19 12:34:15,575 ERROR [scheduler_Worker-2] [atlassian.core.task.AbstractErrorQueuedTaskQueue] handleException com.atlassian.mail.MailException: javax.mail.AuthenticationFailedException
com.atlassian.mail.MailException: javax.mail.AuthenticationFailedException
at com.atlassian.mail.server.impl.SMTPMailServerImpl.send(SMTPMailServerImpl.java:191)
at com.atlassian.confluence.jmx.JmxSMTPMailServer.send(JmxSMTPMailServer.java:71)
at com.atlassian.confluence.mail.template.AbstractMailNotificationQueueItem.send(AbstractMailNotificationQueueItem.java:128)
at com.atlassian.confluence.mail.template.PreRenderedMailNotificationQueueItem.send(PreRenderedMailNotificationQueueItem.java:122)
at com.atlassian.confluence.mail.template.AbstractMailNotificationQueueItem.execute(AbstractMailNotificationQueueItem.java:99)
at com.atlassian.core.task.AbstractErrorQueuedTaskQueue$TaskDecorator.execute(AbstractErrorQueuedTaskQueue.java:107)
at com.atlassian.core.task.AbstractTaskQueue.flush(AbstractTaskQueue.java:45)
at com.atlassian.core.task.AbstractErrorQueuedTaskQueue.flush(AbstractErrorQueuedTaskQueue.java:37)
at com.atlassian.quartz.jobs.TaskQueueFlushJob.doExecute(TaskQueueFlushJob.java:32)
at com.atlassian.quartz.jobs.AbstractJob.executeInternal(AbstractJob.java:86)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at com.atlassian.confluence.setup.quartz.DelegatingClusterAwareQuartzJobBean.executeJob(DelegatingClusterAwareQuartzJobBean.java:16)
at com.atlassian.confluence.setup.quartz.AbstractClusterAwareQuartzJobBean.surroundJobExecutionWithLogging(AbstractClusterAwareQuartzJobBean.java:63)
at

[Dan Hartnack]

This is becoming a very crippling issue for us. Sometimes when there is a high vlume of tickets being updated/created some users may not reviece the notification email for 20+ mins which is critical. Any other suggestions?

[Radu Dumitriu]

https://confluence.atlassian.com/display/JIRAKB/Troubleshooting+Mail+and+Mail+Handlers

User -Dmail.debug to see what's happening. IMHO, because you have AuthenticationFailedException, it is something linked to your new exchange server configuration.

Edit: https://confluence.atlassian.com/display/JIRAKB/Authenticate+Failed+Error+when+Connecting+to+Exchange

[Dan Hartnack]

We disabled authentication to the exchange server and I see no difference. JIRA is still holding the emails and pushing them out one at a time. I am going to set the logs to debug this afternoon and see if I can dig up some more information.

[Dattatray Borude]

Can we see notification sent date/time?

SQL?

[Andy Brook [Plugin People]]

JIRA doesn't track outbound notifications so no. It would result in millions of notification entries, JIRA dropping 'inbound' receipt records recently, it was a common cause of complaint taking up GB in big instances.

JEMH has some support for recording outbound notifications, but is meant for targetted projects, to avoid the problems of space above.

[J. Caldwell]

You mention looking at exchange, but have you been checking your SMTP server or is exchange performing that function as well? When you upgraded, did you switch in new hardware?

[Dan Hartnack]

When we upgraded we did switch in new hardware. People have metioned message throttling but we have never heard about that so what are some recommendations for setting those policies. Our exchange only accepts the email we give it and JIRA is the one making the email and using exchange as the sender.

[Jason Hensler]

Check your Jira mail settings for TLS and make sure it matches your config in the exchance server. Also the Atlassian docs surgest looking for outofmemory errors is jira isn't sending out emails.

[MattS]

Could it be that the default throttling is now in place on Exchange, and wasn't before? I'd recommend using a staging JIRA instance to see whether JIRA is trying to send the emails at long intervals, or whether it is trying and getting lots of 'not ready' responses from Exchange.

[Dan Hartnack]

What is a good start to change the throttling to? We have never touched (or heard of) throttling before this.

[Dan Hartnack]

This message is also repeating when i look at the catalina logs:

2012-05-22 16:09:42,806 IssueIndexer:thread-5 WARN administrator 951x198x1 xdausq 192.168.201.136 /secure/admin/XmlRestore.jspa [atlassian.jira.web.FieldVisibilityManagerImpl] Issue with id '141210' and key 'CENSUS-6142' has a null project, returning true for isFieldHidden check.

[Jason Hensler]

The above error seems to indicate that the cencus-6142 issue is not or does not have an assigned project. The Project field for that issue in the db is probably set to null.

[Dan Hartnack]

No I do not. I have attached the logs to this.

[Renjith Pillai]

Are you noticing any other exceptions in the logs? Also assuming that your service properties are not customized to have a higher timeout.

[Dan Hartnack]

For a little more information. The Flush Mail Queue button does not work when clicked and I checked the logs to see if JIRA was having any OutOfMemory issues and I did not see that error.