Local Home requires POSIX when deploying Bitbucket on AKS using azurefile csi.

[Thiago de Oliveira Santos]

I am deploying Bitbucket to AKS following the Atlassian installation guide: Install Atlassian DC Helm Charts. 

For Local Home I am dynamically creating both pv and pvc using the default storage class azurefile-csi from Dynamically create azure files pvs by using the built in storage classes.

For local home I am passing the follow values on values.yaml

localHome:

# Dynamic provisioning of local-home using the K8s Storage Classes
#
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic
# https://atlassian.github.io/data-center-helm-charts/examples/storage/aws/LOCAL_STORAGE/
#
persistentVolumeClaim:

# -- If 'true', then a 'PersistentVolume' and 'PersistentVolumeClaim' will be dynamically
# created for each pod based on the 'StorageClassName' supplied below.
#
create: true

# -- Specify the name of the 'StorageClass' that should be used for the local-home
# volume claim.
#
storageClassName: "azurefile-csi"

# -- Specifies the standard K8s resource requests and/or limits for the local-home
# volume claims.
#
resources:
requests:
storage: 1Gi

I also tried to create a claim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-local-home
namespace: bitbucket
spec:
accessModes:
- ReadWriteMany
storageClassName: "azurefile-csi" 
resources:
requests:
storage: 1Gi

and passed it as:

localHome:

# Dynamic provisioning of local-home using the K8s Storage Classes
#
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic
# https://atlassian.github.io/data-center-helm-charts/examples/storage/aws/LOCAL_STORAGE/
#
persistentVolumeClaim:

# -- If 'true', then a 'PersistentVolume' and 'PersistentVolumeClaim' will be dynamically
# created for each pod based on the 'StorageClassName' supplied below.
#
create: false

...

# -- Static provisioning of local-home using K8s PVs and PVCs
#
# NOTE: Due to the ephemeral nature of pods this approach to provisioning volumes for
# pods is not recommended. Dynamic provisioning described above is the prescribed
# approach.
#
# When 'persistentVolumeClaim.create' is 'false', then this value can be used to define
# a standard K8s volume that will be used for the local-home volume(s). If not defined,
# then an 'emptyDir' volume is utilised. Having provisioned a 'PersistentVolume', specify
# the bound 'persistentVolumeClaim.claimName' for the 'customVolume' object.
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#static
#
customVolume:
 persistentVolumeClaim:
 claimName: "pvc-local-home"

Both resulted in the same error.

After deploying the helm chart, the pod with Bitbucket starts but never gets to a ready state due to the following error:

2024-01-08 17:52:21,922 ERROR [spring-startup]  c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started

org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshHealthcheck' defined in URL [jar:file:/opt/atlassian/bitbucket/app/WEB-INF/lib/bitbucket-git-mesh-8.9.8.jar!/com/atlassian/stash/internal/scm/git/mesh/MeshHealthcheck.class]:

 Unsatisfied dependency expressed through constructor parameter 0; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshClient' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'meshClient' parameter 2; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'routePropagatingClientInterceptor' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'routePropagatingClientInterceptor' parameter 0; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'meshRouter' parameter 5; 

 nested exception is org.springframework.beans.factory.BeanCreationException:

 Error creating bean with name 'sidecarManager': 

 Invocation of init method failed; 

 nested exception is java.nio.file.FileSystemException: 

 /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem:

 Operation not permitted

 at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:801)

 at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:929)

 at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591)

 at javax.servlet.GenericServlet.init(GenericServlet.java:143)

 at java.base/java.lang.Thread.run(Thread.java:829)

 ... 18 frames trimmed

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshClient' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshClient' parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'routePropagatingClientInterceptor' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'routePropagatingClientInterceptor' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshRouter' parameter 5; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 ... 6 common frames omitted

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'routePropagatingClientInterceptor' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'routePropagatingClientInterceptor' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshRouter' parameter 5; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 ... 6 common frames omitted

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshRouter' parameter 5; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 ... 6 common frames omitted

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160)

 ... 5 common frames omitted

Caused by: java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)

 at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)

 at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)

 at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:254)

 at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:276)

 at java.base/java.nio.file.Files.setPosixFilePermissions(Files.java:2080)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.write(KeyUtils.java:382)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.writePublicKey(KeyUtils.java:260)

 at com.atlassian.stash.internal.scm.git.mesh.MeshLauncher.start(MeshLauncher.java:121)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.startSidecar(DefaultSidecarManager.java:272)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.internalStart(DefaultSidecarManager.java:223)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.start(DefaultSidecarManager.java:178)

 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 ... 5 common frames omitted

When checking the stack trace I realized that the error:

Caused by: java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

is being thrown when executing:

at java.base/java.nio.file.Files.setPosixFilePermissions(Files.java:2080)

at com.atlassian.bitbucket.mesh.util.KeyUtils.write(KeyUtils.java:382)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.writePublicKey(KeyUtils.java:260)

 at com.atlassian.stash.internal.scm.git.mesh.MeshLauncher.start(MeshLauncher.java:121)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.startSidecar(DefaultSidecarManager.java:272)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.internalStart(DefaultSidecarManager.java:223)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.start(DefaultSidecarManager.java:178)

It seems to be an issue when trying to set POSIX.

Azure files CSI is not compliant with POSIX, for that an NFS File Share has to be used.

When running the helm install using emptyDir for local home the issue does not happen.

I couldn't find any mention of POSIX in the section 5 Configure Persistent Storage of the installation guide.

Am I doing something wrong or a volume compliant with POSIX is needed in order to run Bitbucket properly on Kubernetes?

[Nic Brough -Adaptavist-]

This is broadly what it says it is - you need to move to using a standard supportable file system

NFS can work with older Windows file systems like NTFS and FAT, and all Unix file systems like ext1-4, btr and so-on, because these are all Unix compatible, and can be made to behave as though they are POSIX compliant.