Bitbucket fails to authenticate with Jira

[Paul_Schwaegerle]

I have a Bitbucket, Jira and Bamboo servers.  The Jira server holds the user database.  Bamboo has no issue authenticating with the Jira Server.  Bitbucket does.  The only user I can get to login to Bitbucket with is with mine which the user is a local admin and i also have the same user on Jira as a admin so, not sure which one is really being used.  When somebody logs in with thier Jira login, they get "The remote authentication server is not available. Please try again later."  There is no issue logging into Jira or Bamboo. 

There are no proxies or firewalls enabled.  All three server are set up with whitelists with the other servers.  All the application links show connected between the servers.  When I go into the use directories and test the settings, I get a successful test with Bamboo but a failure under bitbucket.  bellow is the log from the failure.  I rebooted, upgraded been though everything 4 and 5 time with no luck. 

I did go into User Directories for BitBucket and was able to succesfully test a new connection to Jira but when I go to save it it fails: 

Connection test failed. Response from the server:
com.atlassian.crowd.exception.InvalidAuthenticationException: Application failed to authenticate

The form also wants to keep auto filling in the Proxy Username only.  Not sure why.

Please help.

2019-08-06 12:27:55,065 ERROR [Caesium-1-4] c.a.c.d.DbCachingDirectoryPoller Error occurred while refreshing the cache for directory [ 360449 ].
com.atlassian.crowd.exception.OperationFailedException: com.atlassian.crowd.exception.ApplicationPermissionException: Forbidden (403) Encountered a "403 - Forbidden" error while loading this page. client.forbidden.exception Go to Jira home
at com.atlassian.crowd.directory.RemoteCrowdDirectory.searchUsers(RemoteCrowdDirectory.java:334)
at com.atlassian.crowd.directory.ldap.cache.RemoteDirectoryCacheRefresher.findAllRemoteUsers(RemoteDirectoryCacheRefresher.java:54)
at com.atlassian.crowd.directory.ldap.cache.RemoteDirectoryCacheRefresher.synchroniseAllUsers(RemoteDirectoryCacheRefresher.java:90)
at com.atlassian.crowd.directory.synchronisation.cache.AbstractCacheRefresher.synchroniseAll(AbstractCacheRefresher.java:45)
at com.atlassian.crowd.directory.ldap.cache.EventTokenChangedCacheRefresher.synchroniseAll(EventTokenChangedCacheRefresher.java:62)
at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1022)
at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.lambda$synchronise$0(DirectorySynchroniserImpl.java:80)
at com.atlassian.crowd.audit.NoOpAuditLogContext.withAuditLogSource(NoOpAuditLogContext.java:17)
at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:78)
at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:45)
at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobRunner.runJob(DirectoryPollerJobRunner.java:92)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.lang.Thread.run(Unknown Source)
... 12 frames trimmed
Caused by: com.atlassian.crowd.exception.ApplicationPermissionException: Forbidden (403) Encountered a "403 - Forbidden" error while loading this page. client.forbidden.exception Go to Jira home
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.throwError(RestExecutor.java:531)
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:367)
at com.atlassian.crowd.integration.rest.service.RestCrowdClient.searchUsers(RestCrowdClient.java:514)
at com.atlassian.crowd.directory.RemoteCrowdDirectory.searchUsers(RemoteCrowdDirectory.java:328)

[Paul_Schwaegerle]

Still not sure how Bitbucket got into this state as nothing as changed on it.  I did figure out that some of the fields in the Configure Atlassian Crowd Server in the User Directories are being auto filled in without my knowledge like proxy username and application password every time the page refreshed.  So, I would fill in the information, have a successful test then fail on the save.  Once I figured this out, I was able to enter in a correct password after I tested and all was good.

[Paul_Schwaegerle]

For no reason, Bitbucket is back to what I have listed above.  When I try and test the server settings, I am getting a 403 error.  I even changed the password for Bitbucket to talk to Jira with and still getting the same thing.