Confluence hang when trying to sync user directory

[Alexander Perrin]

Problem 

Sometimes when a user tries to log in to confluence the system gets stuck loading for a while. There seems to be a problem with synchronising users from the active directory. 

Firstly it tries to perform an incremental synchronisation of the cache with the active directory, and try to add 100 users (who have seemingly been added recently):

2017-06-29 09:09:39,563 INFO [Caesium-1-2] [atlassian.crowd.directory.DbCachingRemoteDirectory] synchroniseCache INCREMENTAL synchronisation for directory [ 76480513 ] starting

...

2017-06-29 09:09:43,161 INFO [Caesium-1-2] [atlassian.crowd.directory.DbCachingRemoteChangeOperations] addUsers adding [ 100 ] users

Then it will throw a duplicate key error for  a user (alias xxx.xxx)

2017-06-29 09:09:43,246 ERROR [Caesium-1-2] [engine.jdbc.spi.SqlExceptionHelper] logExceptions Violation of UNIQUE KEY constraint 'cwd_user_name_dir_id'. Cannot insert duplicate key in object 'dbo.cwd_user'. The duplicate key value is (xxx.xxx, 0513).

2017-06-29 09:09:43,248 WARN [Caesium-1-2] [persistence.hibernate.batch.AbstractBatchProcessor] processBatch batch failed falling back to individual processing

java.lang.RuntimeException: could not flush session

It can't seem to process users in a batch because of single user xxx.xxx? And so it tries to process indivduals - but confluence does not support individual processing.

2017-06-29 09:09:43,250 ERROR [Caesium-1-2] [persistence.hibernate.batch.AbstractBatchProcessor] processIndividual Could not process class com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup: com.atlassian.crowd.util.persistence.hibernate.batch.TransactionGroup@7faa647e[primaryObject=com.atlassian.crowd.model.user.InternalUser@7c8e78d8[id=130457012,name=yyy.yyy,createdDate=Thu Jun 29 09:09:43 BST 2017,updatedDate=Thu Jun 29 09:09:43 BST

java.lang.RuntimeException: Confluence does not support individual processing

Then notifies that those users cannot be processed:

2017-06-29 09:09:43,314 ERROR [Caesium-1-2] [crowd.embedded.hibernate2.HibernateUserDao] addAll The following users could not be processed:

list of the users

Then confluence goes unresponsive for a few minutes (see time stamps) where confluence is essentially stuck in loading and then this is reported:

2017-06-29 09:12:55,713 WARN [http-nio-8443-exec-3] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1498723405374, threadId=226, threadName='http-nio-8443-exec-3', userId='alex.Perrin', type='web-request', summary='/'}

 -- url: / | traceId: 8ff5ec5656cd6ef0 | userName: alex.Perrin

Then it seems to work fine for a while until it tries to do this again.

Any help/comments much appreciated!

[Alexander Perrin · Accepted solution]

Update - This was actually an issue with Active Directory - we had two duplicates of that particular user. Deleting one of them solved the issue. However it seems that the Confluence hang is unrelated to the user issue, and it still occurs.

[AnnWorley]

Glad to hear that fixing the user in AD fixed the sync issue.

I understand Confluence still hangs, is that still during the LDAP sync or it hangs periodically and the LDAP sync is healthy now?

This article has some good advice for troubleshooting hangs/crashes: Troubleshooting Confluence hanging or crashing

I look forward to hearing what you find.

[AnnWorley]

Please let me know what version of Confluence you are running. There have been changes to the way user management works over different versions.

As you noticed, the unique constraint error you cited is telling us that some record is already in the cwd_user table and the sync process is trying to add it again.

• If you create a new LDAP User directory (Confluence Admin>User Directories) with the same settings it will probably work fine, as it will create a new cache.
• After you are sure the new directory is working (test by disabling the old one and logging in with LDAP credentials) you can remove the current User Directory.
• If you are using "read-only with local groups" and added a lot of users to local groups, these memberships will not exist in the new directory, so it's not an optimal solution in that case.

Another strategy that sometimes works in similar cases is to temporarily turn off incremental sync (edit the User Directory and uncheck it) and run a full sync.

Note: If you have Follow Referrals checked in the User Directory it could be causing the unexpected 100 users every time you sync as it will refer the sync to look for users on other AD servers than the one you designated in the settings.

[Alexander Perrin]

Thanks for the reply Ann!

So this problem is actually ocurring in our test environment in which we upgraded our test instance, as a test for this operation in production. The problem started ocurring after this upgrade. 

We upgraded from version 5.4.4 to 6.2.1. I had to change the collation of the database as per https://confluence.atlassian.com/confkb/how-to-fix-the-collation-of-a-microsoft-sql-server-confluence-database-687216725.html. Not sure if this could have caused this problem somehow?

I've re-created the user directory multiple times, and deleted the old directories, and still get the same issue...

I've also tried turning off incremental sync, but with no improvement. Follow referals is not checked.

We have the same settings as in our production environment, which works fine.

Another peculiar feature is that user, if I click "Edit details" on her profile in confluence, her name has a + symbol after it, which is not present in the production version... Could that plus symbol be causing Confluence to try and add the user again perhaps?

[Mayuresh Sakharape]

Hi @AnnWorley,

We are facing same issue. We are upgrading to Confluence 6.6.8 from 5.10.8

We are having 2 active directories and some users are present in both. Please suggest the solution to get rid of these errors.

Regards,

Mayuresh