Speed up sharing of artifacts

[Cameron Ferguson]

We have noticed in our bamboo setup that larger artifacts take a VERY long time to share. All of our agents and the actual bamboo server are on a gigabit network and are very high end computers (dual hex core xeons hyperthreaded, 24 gigs of ram, etc)

Publishing an artifact of 250mb doesn't take any time at all.. but when an agent needs that artifact, the "preparing artifact for use" section takes roughly 3-4 minutes for the artifact to become available.

I've manually tested a secure copy from "agent to agent", "agent to server" and "server to agent", and it only takes about 3-5 seconds for the data to be transfered... Is there some option that can cause this? I am not sure why this takes so long for bamboo to "prepare the artifact for use"

I'd appreciate it!

Thanks!

[Przemek Bruski · Accepted solution]

Hmm, are you using standalone distribution (Jetty) or Tomcat? Also, what OS is that? There are known transfer performance problems with Win2008 on Tomcat.

With that much of a difference between publish and fetch, your question qualifies for support.atlassian.com, because it looks like a bug.

[Cameron Ferguson]

I just found it: jetty/7.2.1.v20101111

[Cameron Ferguson]

We use a few different OS types...

Windows XP 64 bit

CentOS 5.5 64 bit

CentOS 5.3 64 bit

RedHat 6.0 64 bit

Each of these OS's have the same slowdown... though it seems that WindowsXP takes maybe an extra minute over the Linux distros we use.

I am not sure what standalone distribution we are using... where would I find that?

[Cameron Ferguson]

Let me also add something that may be the reason this is happening now that I think about it.

We compile about 11 different versions of a few of our products (different compilers, different OS, different architectures, etc). We had an initial design issue with bamboo and our perforce server where when we had all 11 of these different build jobs pull, we would have errors with perforce that so much was being pulled at once by the same user from the same location. To solve this issue, we basically designed our first stage to pull our repository from perforce, and then share that as an artifact (The copy pattern was just ** ). We do not zip this artifact up or tar it or anything, we just share the repository itself and all files.

In the 2nd stage, our build stage, we would depend upon this artifact. This way for an entire build, we pull the repository once, and then share it with other jobs.

For one of our repositories that takes quite a long time, it's roughly 3600 files that total up around 250mb. Could the reason for the slow up be because there are so many files that are shared and not necessarily the size?

[Cameron Ferguson]

Zipping up the files as a single file to be shared seems to have sped up the process... our build times prior to this change were about 10 minutes... they are now down to 5 with the zip feature. Seems bamboo really doesn't like sharing thousands of files as a single artifact.