Bitbucket Failed to start with HTTPS

[Rethink]

the application working fine over HTTP but after importing the SSL Certificate and adding the following properties it failed to start:

server.port=8443
server.secure=true
server.scheme=https
server.ssl.enabled=true
server.ssl.client-auth=want
server.ssl.protocol=TLS
server.ssl.key-store=C:\Atlassian\ApplicationData\Bitbucket\shared\config\ssl-keystore
server.ssl.key-store-password=P@ssw0rd
server.ssl.key-password=P@ssw0rd

please find the log:

2019-01-02 15:07:35,046 INFO  [main]  c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on Bitbucket with PID 3332 (C:\Atlassian\Bitbucket\5.16.0\app\WEB-INF\classes started by atlbitbucket in C:\Atlassian\Bitbucket\5.16.0\app)
2019-01-02 15:07:35,046 INFO  [main]  c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
2019-01-02 15:07:48,484 INFO  [main]  c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 5.16.0 (c03f0a7 built on Fri Nov 16 08:10:50 GST 2018)
2019-01-02 15:07:48,499 INFO  [main]  c.a.b.i.boot.log.BuildInfoLogger JVM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0_172-b11
2019-01-02 15:08:05,718 ERROR [main]  o.a.catalina.core.StandardService Failed to start connector [Connector[HTTP/1.1-8443]]
org.apache.catalina.LifecycleException: Failed to start component [Connector[HTTP/1.1-8443]]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.addPreviouslyRemovedConnectors(TomcatEmbeddedServletContainer.java:265)
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.start(TomcatEmbeddedServletContainer.java:208)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.startEmbeddedServletContainer(EmbeddedWebApplicationContext.java:297)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:145)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:546)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
    at com.atlassian.bitbucket.internal.boot.BitbucketServerApplication.start(BitbucketServerApplication.java:247)
    at com.atlassian.bitbucket.internal.boot.BitbucketServerApplication.main(BitbucketServerApplication.java:82)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
    at com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher.start(BitbucketServerLauncher.java:151)
    at com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher.main(BitbucketServerLauncher.java:99)
    ... 5 frames trimmed
Caused by: org.apache.catalina.LifecycleException: Protocol handler start failed
    at org.apache.catalina.connector.Connector.startInternal(Connector.java:1020)
    ... 16 common frames omitted
Caused by: java.lang.IllegalArgumentException: C:\Atlassian\Bitbucket\5.16.0\app\AtlassianApplicationDataBitbucketsharedconfigssl-keystore (The system cannot find the file specified)
    at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:115)
    ... 16 common frames omitted
Caused by: java.io.FileNotFoundException: C:\Atlassian\Bitbucket\5.16.0\app\AtlassianApplicationDataBitbucketsharedconfigssl-keystore (The system cannot find the file specified)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at java.io.FileInputStream.<init>(FileInputStream.java:93)
    at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
    at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
    ... 16 common frames omitted
2019-01-02 15:08:40,515 WARN  [spring-startup]  c.a.stash.internal.home.HomeLock Failed to write process information into the lock file
java.nio.channels.ClosedByInterruptException: null
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
    at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
    at com.atlassian.stash.internal.home.HomeLock.acquireLock(HomeLock.java:127)
    at com.atlassian.stash.internal.home.HomeLock.lock(HomeLock.java:98)
    at com.atlassian.stash.internal.home.HomeLockAcquirer.lock(HomeLockAcquirer.java:58)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:151)
    at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:525)
    at javax.servlet.GenericServlet.init(GenericServlet.java:158)
    at java.lang.Thread.run(Thread.java:748)
    ... 29 frames trimmed
2019-01-02 15:08:40,515 INFO  [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory C:\Atlassian\ApplicationData\Bitbucket
2019-01-02 15:08:40,562 WARN  [spring-startup]  c.a.s.internal.home.HomeValidator Failed to read .version; it will be recreated
java.nio.channels.ClosedByInterruptException: null
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.read1(BufferedReader.java:210)
    at java.io.BufferedReader.read(BufferedReader.java:286)
    at java.io.Reader.read(Reader.java:140)
    at java.util.Properties$LineReader.readLine(Properties.java:434)
    at java.util.Properties.load0(Properties.java:353)
    at java.util.Properties.load(Properties.java:317)
    at com.atlassian.bitbucket.util.PropertiesUtils.load(PropertiesUtils.java:60)
    at com.atlassian.stash.internal.home.HomeValidator.maybeUpdateLegacyVersion(HomeValidator.java:212)
    at com.atlassian.stash.internal.home.HomeValidator.validate(HomeValidator.java:56)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:151)
    at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:525)
    at javax.servlet.GenericServlet.init(GenericServlet.java:158)
    at java.lang.Thread.run(Thread.java:748)
    ... 27 frames trimmed
2019-01-02 15:08:40,562 WARN  [spring-startup]  c.a.s.internal.home.HomeValidator Failed to write .version
java.nio.channels.ClosedByInterruptException: null
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
    at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
    at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
    at java.nio.channels.Channels.writeFully(Channels.java:101)
    at java.nio.channels.Channels.access$000(Channels.java:61)
    at java.nio.channels.Channels$1.write(Channels.java:174)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
    at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
    at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
    at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
    at java.io.BufferedWriter.flush(BufferedWriter.java:254)
    at java.util.Properties.store0(Properties.java:844)
    at java.util.Properties.store(Properties.java:771)
    at com.atlassian.bitbucket.util.PropertiesUtils.store(PropertiesUtils.java:119)
    at com.atlassian.stash.internal.home.HomeValidator.maybeUpdateLegacyVersion(HomeValidator.java:234)
    at com.atlassian.stash.internal.home.HomeValidator.validate(HomeValidator.java:56)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:151)
    at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:525)
    at javax.servlet.GenericServlet.init(GenericServlet.java:158)
    at java.lang.Thread.run(Thread.java:748)
    ... 27 frames trimmed
2019-01-02 15:08:40,577 WARN  [spring-startup]  o.s.w.c.s.XmlWebApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'homeValidator' defined in class path resource [critical-context.xml]: Invocation of init method failed; nested exception is com.atlassian.stash.internal.home.HomeValidationFailedException: The home directory cannot be validated for use by Bitbucket 5.16.0
2019-01-02 15:08:40,577 INFO  [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Releasing lock on C:\Atlassian\ApplicationData\Bitbucket
2019-01-02 15:08:40,593 ERROR [spring-startup]  c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'homeValidator' defined in class path resource [critical-context.xml]: Invocation of init method failed; nested exception is com.atlassian.stash.internal.home.HomeValidationFailedException: The home directory cannot be validated for use by Bitbucket 5.16.0
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1631)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:151)
    at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:525)
    at javax.servlet.GenericServlet.init(GenericServlet.java:158)
    at java.lang.Thread.run(Thread.java:748)
    ... 20 frames trimmed
Caused by: com.atlassian.stash.internal.home.HomeValidationFailedException: The home directory cannot be validated for use by Bitbucket 5.16.0
    at com.atlassian.stash.internal.home.HomeValidator.validate(HomeValidator.java:85)
    ... 6 common frames omitted
Caused by: java.nio.channels.ClosedByInterruptException: null
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.read1(BufferedReader.java:210)
    at java.io.BufferedReader.read(BufferedReader.java:286)
    at java.io.Reader.read(Reader.java:140)
    at java.util.Properties$LineReader.readLine(Properties.java:434)
    at java.util.Properties.load0(Properties.java:353)
    at java.util.Properties.load(Properties.java:317)
    at com.atlassian.bitbucket.util.PropertiesUtils.load(PropertiesUtils.java:60)
    at com.atlassian.bitbucket.util.PropertiesUtils.loadIfPresent(PropertiesUtils.java:100)
    at com.atlassian.stash.internal.home.HomeProperties.load(HomeProperties.java:63)
    at com.atlassian.stash.internal.home.HomeValidator.validate(HomeValidator.java:59)
    ... 6 common frames omitted
2019-01-02 15:08:42,421 WARN  [localhost-startStop-1]  o.a.c.loader.WebappClassLoaderBase The web application [ROOT] appears to have started a thread named [OracleTimeoutPollingThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
2019-01-02 15:08:42,421 WARN  [localhost-startStop-1]  o.a.c.loader.WebappClassLoaderBase The web application [ROOT] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
2019-01-02 15:08:42,421 WARN  [localhost-startStop-1]  o.a.c.loader.WebappClassLoaderBase The web application [ROOT] appears to have started a thread named [InterruptTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
2019-01-02 15:08:42,546 ERROR [main]  o.s.b.d.LoggingFailureAnalysisReporter

***************************
APPLICATION FAILED TO START
***************************

Description:

The Tomcat connector configured to listen on port 8443 failed to start. The port may already be in use or the connector may be misconfigured.

Action:

Verify the connector's configuration, identify and stop any process that's listening on port 8443, or configure this application to listen on another port.

2019-01-02 15:08:42,562 INFO  [main]  c.a.b.i.boot.log.BuildInfoLogger Bitbucket 5.16.0 has shut down

[Daniel Santos · Accepted solution]

Hi @rethink_it,

It looks like Spring Boot is not correctly parsing the path for your keystore file or the file is not in the right location:

Caused by: java.lang.IllegalArgumentException:

C:\Atlassian\Bitbucket\5.16.0\app\AtlassianApplicationDataBitbucketsharedconfigssl-keystore (The system cannot find the file specified)
 at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:115)

Moving forward:

1. Make sure the file ssl-keystore is at
   

   C:\Atlassian\ApplicationData\Bitbucket\shared\config

2. Change the following line
   

   server.ssl.key-store=C:\Atlassian\ApplicationData\Bitbucket\shared\config\ssl-keystore

   to
   

   server.ssl.key-store=C:\\Atlassian\\ApplicationData\\Bitbucket\\shared\\config\\ssl-keystore

   or just comment it out (since it is already pointing to the default location Bitbucket will use to search for that file)
   If you decide to comment it out, it should look like:
   

   #server.ssl.key-store=C:\Atlassian\ApplicationData\Bitbucket\shared\config\ssl-keystore

3. Restart Bitbucket

I hope that you help you to move forward.

[Rethink]

thanks Daniel the app is working now but I received this error

A fatal error has occurred

The following problem occurred, which prevents Atlassian Bitbucket from starting:

• The database, as currently configured, is not accessible.
  Login failed for user 'sa'. ClientConnectionId:81dbc9a3-27cc-4bba-9e44-80b0021420e1

[Rethink]

ERROR [spring-startup]  com.zaxxer.hikari.pool.HikariPool bitbucket - Exception during pool initialization.
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'sa'. ClientConnectionId:d80e096c-100a-4e09-b40d-ca6a7cd97c82

[Rethink]

 I have solved the DB issue thanks Daniel

[Rethink]

I need your help , how to redirect the HTTP request to HTTPS .

[Daniel Santos]

If you have a proxy in front of Bitbucket you will need to do the redirect on the proxy side. If you have the SSL port configured on Bitbucket side, let's say in port 8443 with no other connector configured, then you would need something like this in your bitbucket.properties file:

server.require-ssl=true
server.additional-connector.1.port=7990
server.additional-connector.1.redirect-port=8443

This would create a connector 7990 and redirect its requests to 8443.

Reference: Migrate server.xml customizations to bitbucket.properties (at Bitbucket Server secured with SSL and additional connector to redirect HTTP requests section)

[Rethink]

thanks Daniel for your valuable help , it's working fine.

[Michael Wolfe]

I'm using Apache to proxy the port numbers.  I'm using an apache site config that is similar to what I have configured for Jira and Confluence.  Could you elaborate at little more on what you would need to put in bitbucket.properties?  I have tried using 7443 as the port but the service fails to start and complains that 7443 is already in use.  When checking to see if that port was in use, I didn't see it active when I do an netstat -an | grep 7443.  I tried using the standard config from here.

<VirtualHost bitbucket-test.domain.com:80>
  ServerName bitbucket-test.domain.com
  ProxyRequests Off
  ProxyPreserveHost On
  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>
  ProxyPass / http://bitbucket-test.domain.com:7990/
  ProxyPassReverse / http://bitbucket-test.domain.com:7990/
</VirtualHost>

<VirtualHost bitbucket-test.domain.com:443>
  ServerName bitbucket-test.domain.com
  SSLEngine on
  SSLCertificateFile      /etc/ca-certificates/bitbuckettestssl.cer
  SSLCertificateKeyFile   /etc/ca-certificates/bitbuckettestssl.key
  SSLCertificateChainFile /etc/ca-certificates/rootCA.crt
  ProxyRequests Off
  ProxyPreserveHost On
  <Proxy *>
    Order deny,allow
    Allow from all
</Proxy>
  SSLProxyEngine On
  ProxyPass / https://bitbucket-test.domain.com:7443/
  ProxyPassReverse / https://bitbucket-test.domain.com:7443/
</VirtualHost>

Here is my bitbucket.properties

jdbc.driver=org.postgresql.Driver
jdbc.url=jdbc:postgresql://localhost:5432/bitbucketdb
jdbc.user=bitbucketdbuser
jdbc.password=superSecurePassword

# Bitbuckt SSL Settings

server.port=7443
server.require-ssl=true
server.ssl.enabled=true
server.ssl.key-store=$JAVA_HOME/lib/security/cacerts
server.ssl.key-alias=bitbucket-test
server.ssl.key-store-type=${keystore.type:jks}
server.ssl.key-store-password=changeit