Elastic Agent stuck "Pending" on Custom Elastic Image

[Karol Sójko]

Ok so I think I've got everything pretty much running how I want - the only thing is the Agent seems to be stuck when I start an instance.

So here's what I did:

1. I've created an instance based on Atlassian maintained AMI (ami-5be8f232) and customised it with provisioning handled from Vagrant+Puppet. So I ended up with an customised dev environment based on an Atlassian AMI
2. I've created an image out of that so I could reproduce the instances with that
3. Now every time I try to start an instance based on the new image the instance starts perfectly but the agent is stuck "pending".
4. When I `tail -f /home/bamboo/bamboo-elastic-agent.out` it seems to be stuck on starting the tunnel server. Here's the output:

2014-06-11 10:29:49,620 INFO [main] [S3Utils] Found 1592 files in /opt/bamboo-elastic-agent

2014-06-11 10:29:49,620 INFO [main] [S3Utils] Generating the list of files to fetch from S3...

2014-06-11 10:29:49,645 INFO [main] [S3Utils] Generating the list of files to remove...

2014-06-11 10:29:49,651 INFO [main] [S3Utils] Removing 1174 files from /opt/bamboo-elastic-agent

2014-06-11 10:29:51,995 INFO [main] [S3Utils] Fetching 171 files to /opt/bamboo-elastic-agent

2014-06-11 10:30:05,991 INFO [main] [S3Utils] Fetched 135 MB from S3

2014-06-11 10:30:05,996 INFO [main] [ElasticAgentInstaller] Starting [java, -server, -Xms32m, -Xmx256m, -XX:MaxPermSize=128m, -XX:+HeapDumpOnOutOfMemoryError, -Dimagefiles.version=3.3, -Dbamboo.agent.installDir=/opt/bamboo-elastic-agent, -cp, /opt/bamboo-elastic-agent/boot/annotations-13.0.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-core-agent-bootstrap-5.5-OD-04-0004.jar:/opt/bamboo-elastic-agent/boot/gson-2.2.2-atlassian-1.jar:/opt/bamboo-elastic-agent/boot/commons-io-2.4.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-5.5-OD-04-0004.jar:/opt/bamboo-elastic-agent/boot/jackson-core-2.1.1.jar:/opt/bamboo-elastic-agent/boot/atlassian-tunnel-0.20.jar:/opt/bamboo-elastic-agent/boot/stax-api-1.0-2.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-api-agent-bootstrap-5.5-OD-04-0004.jar:/opt/bamboo-elastic-agent/boot/commons-httpclient-3.0.1.jar:/opt/bamboo-elastic-agent/boot/atlassian-util-concurrent-2.4.1.jar:/opt/bamboo-elastic-agent/boot/joda-time-2.3.jar:/opt/bamboo-elastic-agent/boot/log4j-1.2.15.jar:/opt/bamboo-elastic-agent/boot/commons-codec-1.4.jar:/opt/bamboo-elastic-agent/boot/commons-lang-2.6.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-bootstrap-5.5-OD-04-0004.jar:/opt/bamboo-elastic-agent/boot/jackson-databind-2.1.1.jar:/opt/bamboo-elastic-agent/boot/fugue-1.1.jar:/opt/bamboo-elastic-agent/boot/guava-10.0.1.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-1.7.1.jar:/opt/bamboo-elastic-agent/boot/httpclient-4.2.5.jar:/opt/bamboo-elastic-agent/boot/commons-logging-1.0.4.jar:/opt/bamboo-elastic-agent/boot/atlassian-aws-1.0.59.jar:/opt/bamboo-elastic-agent/boot/jackson-annotations-2.1.1.jar:/opt/bamboo-elastic-agent/boot/bcprov-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/bcpkix-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/atlassian-annotations-0.4.jar:/opt/bamboo-elastic-agent/boot/jsr305-1.3.9.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-shared-5.5-OD-04-0004.jar:/opt/bamboo-elastic-agent/boot/httpcore-4.2.5.jar:, com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap]

2014-06-11 10:30:06,209 INFO [main] [ElasticAgentBootstrap] Starting Agent Bootstrap using Java 1.6.0_45 from Sun Microsystems Inc.

2014-06-11 10:30:06,446 INFO [main] [ElasticAgentBootstrap] Using tunnelling. Registering 'httpt' and 'httpst' protocols.

2014-06-11 10:30:06,471 INFO [main] [ElasticAgentBootstrap] Starting tunnel server

2014-06-11 11:00:06,472 ERROR [main] [ElasticAgentBootstrap] Fatal exception.

java.util.concurrent.TimeoutException: The server has failed to contact the agent within 30 minutes. Assuming that the server is unreachable and terminating the agent.

at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.startTunnelServer(ElasticAgentBootstrap.java:159)

at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.run(ElasticAgentBootstrap.java:88)

at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.main(ElasticAgentBootstrap.java:56)

[Karol Sójko · Accepted solution]

yeah I was able to ssh to it via Vagrant tool-set. But Bamboo wasn't able to access it. In the end I have prepared the environment manually based on an Atlassian AMI and now it's accessible.

[Przemek Bruski]

Is the tunnel port (26224) on your EC2 instance reachable from outside?

[Fabian Knorr]

Hi Karol,

this looks like a network error. Are you running the Elastic Agents in a VPC? If so, are routing-tables and firewall correctly configured? I mean both, at AWS and at you own LAN.

[Karol Sójko]

No, the Elastic Agents are not run on a VPC