PIPE microsoft/azure-aks-deploy:1.0.1 - ERROR: [Errno 13] Permission denied: '.kube'

[Chris Buehrle]

I'm using Bitbucket pipelines and this pipe started failing on 6/18. The last known good run was on 6/17 and had been working without issue for over a month. This looks like there is a permissions issue on the mspipes/azure-aks-deploy:1.0.1 image. I'm able to execute the deployment from a local instance using the same service principal so I don't think this has anything to do with my AKS instance or the service principal. Has anyone else reported recent issues with this pipe?

INFO: retrieve the kube config via the azure cliaz aks get-credentials --resource-group test-resource-group --name arunAKSCluster1 --file .kube/kubeconfig-test-resource-group-arunAKSCluster1 --overwrite-existingERROR: [Errno 13] Permission denied: '.kube'Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/knack/cli.py", line 206, in invoke cmd_result = self.invocation.execute(args) File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in execute raise ex File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 386, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 379, in _run_job six.reraise(*sys.exc_info()) File "/usr/local/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 356, in _run_job result = cmd_copy(params) File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 171, in __call__ return self.handler(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 441, in default_command_handler return op(**command_args) File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/acs/custom.py", line 1622, in aks_get_credentials _print_or_merge_credentials(path, kubeconfig, overwrite_existing) File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/acs/custom.py", line 2246, in _print_or_merge_credentials os.makedirs(directory) File "/usr/local/lib/python3.6/os.py", line 220, in makedirs mkdir(name, mode)PermissionError: [Errno 13] Permission denied: '.kube'✖ Unable to retrieve the kubernetes config file from the cluster using az aks get credentials!

[Raul Gomis · Accepted solution]

Hi @Chris Buehrle and @Jon Poploskie ,

Sorry for any inconvenience caused! Rolling out a feature, we introduced a bug that affected permissions when pipes create files / directories. I believe this might the cause of this as the Microsoft microsoft/azure-aks-deploy:1.0.1 does store the kube context temporarily in the .kube directory.

We rolled it back on 19th June 6:32 PM (AEST). 

Please, let me know if you have any more questions or observe other weird behavior.

Regards,

Raul 

[Jon Poploskie · Accepted solution]

We just started experiencing the same issue yesterday with same symptoms: stable for some time, runs correctly w/ sp locally. v1.0.1.

[Jon Poploskie]

FYI this seems to be resolved for us now. I added the "Azure Kubernetes Service Cluster Admin Role" to the service principal, ran the pipeline again and it worked. Then we removed that role assignment (so it just has the "Azure Kubernetes Service Cluster User Role"), and it's continuing to work... I'm not sure if that actually had an impact, or if something changed on the Azure backend somewhere and they've fixed it so the admin role assignment was a red herring???

[Chris Buehrle]

I just tried again without making any changes to my service principal and am getting the same failure. I'll try adding that same role and see if that fixes anything. The strange thing is that I'm able to deploy using the same principal from my local machine without any issue.

[Chris Buehrle]

I've added that role and am still getting a permissions error but it appears to be an issue with creating the '.kube' directory on the container that's running the az aks commands not with the service principal permissions.

File "/usr/local/lib/python3.6/os.py", line 220, in makedirs mkdir(name, mode)

PermissionError: [Errno 13] Permission denied: '.kube'

[Jon Poploskie]

Sorry that didn't work for you, good luck! If we see anything else that might help you I'll let you know.

[Chris Buehrle]

Okay, I re-ran my entire pipeline instead of just the failed deployment and it passed this time. This still seems fishy but I appreciate the feedback @Jon Poploskie

[Jon Poploskie]

Ah yeah, good call: I did re-run the entire pipeline when I tried it. Glad you tried that step and it worked for you. And yes completely agree that it's fishy...