Git Submodule Authentication via https

[Stephan Krusche]

Hi,

I'd like to build an application in Bamboo which is stored in a git repository that depends on a git submodule. Both repositories are stored in Stash, required https authentication via username and paswsword and our build user has access rights.

If I configure the git repository in Bamboo's build plan, there is the option to enable submodules. Unfortunately the submodule clone does not succeed because of an authentication exception:

Submodule 'framework' (https://repobruegge.in.tum.de:8443/scm/FRAMEWORK/framework.git) registered for path 'framework'
24-Jun-2013 17:29:01    Cloning into 'framework'...
24-Jun-2013 17:29:01    fatal: Authentication failed
24-Jun-2013 17:29:01    Clone of 'https://repobruegge.in.tum.de:8443/scm/FRAMEWORK/framework.git' into submodule path 'framework' failed
24-Jun-2013 17:29:01            at com.atlassian.bamboo.executor.RetryingTaskExecutor.rerun(RetryingTaskExecutor.java:132)
24-Jun-2013 17:29:01            at com.atlassian.bamboo.executor.RetryingTaskExecutor.runTask(RetryingTaskExecutor.java:88)
24-Jun-2013 17:29:01            at com.atlassian.bamboo.executor.RetryingTaskExecutor.retry(RetryingTaskExecutor.java:191)
24-Jun-2013 17:29:01            at com.atlassian.bamboo.executor.RetryingTaskExecutor.retry(RetryingTaskExecutor.java:176)
24-Jun-2013 17:29:01            at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.execute(VcsCheckoutTask.java:109)
24-Jun-2013 17:29:01            at com.atlassian.bamboo.task.TaskExecutorImpl.executeTasks(TaskExecutorImpl.java:188)
24-Jun-2013 17:29:01            at com.atlassian.bamboo.task.TaskExecutorImpl.executePreparationTasks(TaskExecutorImpl.java:69)
24-Jun-2013 17:29:01            at com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask.call(PrepareBuildTask.java:69)
24-Jun-2013 17:29:01            at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:206)
24-Jun-2013 17:29:01            at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:103)
24-Jun-2013 17:29:01            at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:111)
24-Jun-2013 17:29:01            at com.atlassian.bamboo.build.pipeline.concurrent.NamedThreadFactory$2.run(NamedThreadFactory.java:52)
24-Jun-2013 17:29:01            at java.lang.Thread.run(Thread.java:680)
24-Jun-2013 17:29:01    Caused by: com.atlassian.bamboo.repository.RepositoryException: Checkout to revision 4d58bfaf90508c753a0538bf06f8d8fc777f5d43 has failed.command /Applications/Xcode.app/Contents/Developer/usr/bin/git submodule update --init --recursive failed with code 1. Working directory was [/opt/bamboo-agent-home/xml-data/build-dir/GAME-GAME-CHECKOUTBUILD]., stderr:

I assume the problem is, that for the submodule, Bamboo does not automtically take the same authentication type (we use Username/Password) as for the normal repository. In our submodule file we do not include the username so that all participating developers are able to clone the git repository including the submodule with SourceTree (it works without any problems with SourceTree)

[submodule "framework"]
    path = framework
    url = https://repobruegge.in.tum.de:8443/scm/FRAMEWORK/framework.git

It is also not possible to specify any authentication options for the submodule.

Is there any solution or workaround for my problem?


Thanks
Stephan

[Ivan Maduro · Accepted solution]

Hi Stephan,

You're running into this bug BAM-11369, indeed checking out the Submodule isn't working because the authentication isn't actually going through the main Bamboo instance as a proxy. To workaround this you will need to set up authentication via SSH Keys.

You will need to generate an SSH key, store it on the agent (under the Bamboo agents home/.ssh dir)

And set up the appropriate .ssh/config for it. This will allow Git to authenticate against the submodules server and check out your Submodule correctly.

Cheers,

Ivan.

[Matthias Linhuber]

Is there no other solution than SSH keys?

[Jack Davis]

I'm a little confused at this answer because apparently HTTPS works alright according to the referenced bug. However this answer suggests that HTTPS doesn't work at all and that you've got to use SSH.

Help us out here Atlassian, does HTTPS work? Or do we have to use SSH?

[Przemek Bruski]

HTTP should work if you embed username/password into the submodule url, which the OP doesn't want to do. If you can't embed credentials, your only option is SSH.

[Jack Davis]

If you can't use SSH in ondemand, does that mean you cannot use this workaround?

[Przemek Bruski]

On OnDemand, you can only use the HTTP with embedded credentials (I think they should work).

[Jack Davis]

but if you don't want to embed credentials you're out of luck? Would a netrc file work?

[Przemek Bruski]

I think it should work. For submodules, any standard credentials provisioning method supported natively by Git will work.

[Jack Davis]

Should I place it in the bamboo user's home directory? (Windows)

[Jack Davis]

I just tried putting it in /Users/Bamboo and it didn't authenticate in bamboo.

However when I put it in the administrator's account and clone from gitbash it works.

I'm assuming the %HOME% directory of the Bamboo user is different, do you know what it is?

[Jack Davis]

Apologies, the /Users/Bamboo directory does work. I am now able to authenticate over HTTPS without embedding credentials.

[John Sposato]

Here's what we did:

1. Setup a user (bamboouser in our case) and gave Observer rights to projects with submodules

2. Added this user/password to the Source Repositories section of the plan setup

3. In our .gitmodules, used the https: Stash link instead of ssh:, and ran git submodule sync

Ran the build and the submodules were successfully checked out.

[Jerry Morrow]

I've been successfully clone git repos without manual authentication using the ssh key feature.  That's been good.  I've had an issue where the submodule update command does not use the ssh key, and I still need to type username/password.  This is a problem for scripted builds (either shell script, buildbot, etc.).

Has anyone else had this issue?  I've already changed my .gitmodules such that the submodule is identified by ssh, not https.  I thought that would be the golden ticket, but it wasn't.

I don't want to throw in the towel. 

[Todd Morrison]

Atlasssian Team,

Please note John Sposato's response above allows submodules to be fetched.

Please add this to your site's project documention for git support on Bamboo.

Thanks!

Todd

Here's what we did:
1. Setup a user (bamboouser in our case) and gave Observer rights to projects with submodules
2. Added this user/password to the Source Repositories section of the plan setup
3. In our .gitmodules, used the https: Stash link instead of ssh:, and ran git submodule sync
 
Ran the build and the submodules were successfully checked out.

[Caleb Tote]

For the record, I tried embedding my credentials into my https connection string, and received the same error. Our repo is hardcoded to not allow ssh (or rather, only allow https) - so we're pretty stuck.

[Kevin Dixon]

This is a terrible issue, but in the mean time, I have had success using

http://gitcredentialstore.codeplex.com/

on our windows build machines to pre-cache credentials for the submodules.

[John Sposato]

Done, but I still need the information for the workaround. Anyone?

[John Sposato]

Ivan,

What needs to be in the config for SSH? I am attempting to get this to work. We have a project that uses submodules from GitHub (which are fine) and Stash (get second auth prompt and it eventually fails).

Is there not a document on Atlassian's site to explain how to do this?

[Jack Davis]

John please vote on this bug BAM-11369 so we can get it fixed.