Fedora 17 Elastic Image Minor Problem

[Matt Clarkson]

I have a small problem with my Fedora 17 OnDemand elastic agent.

I have followed all the instructions on the confluence website and have successfully set up a Fedora 17 elastic bamboo image.

However, I have one minor issue that I cannot get my head around. Hopefully someone with far greater Linux insight can shed some light.

The following line in rc.local does nothing on my Fedora 17 box:

su -c bamboo-elastic-agent - bamboo &

There is some issue with the & doing something wierd. If I remove it and move the ampersand to the /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent script:

java -server -Xms32m -Xmx512m -XX:MaxPermSize=256m -cp $BAMBOO_CLASSPATH com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap 2>&1 | tee -a ~/bamboo-elastic-agent.out &

(Notice it has been added on the end!)

Everything is fine then...

I have tried lots and lots of debugging (4 hours worth) trying to work out why that line doesn't execute correctly in rc.local.

The following line works:

su -c "echo hello > ~/test.log" - bamboo &

it correctly creates the test.log file in the /home/bamboo location. However, if I move the same thing into a script:

/usr/bin/bamboo-test-script:

#!/bin/sh
echo hello

The following line will do nothing:

su -c "bamboo-test-script > ~/test.log" - bamboo &

But the following will work fine:

su -c "bamboo-test-script > ~/test.log" - bamboo

Do you guys have any idea what is going on? This is literally the only thing stopping me from making various distro elastic images :(

[Matt Clarkson · Accepted solution]

This is now solved in the newer bamboo agent scripts with OnDemand Bamboo 4.1

[Matt Clarkson]

The following are a temporary workaround (for anyone reading this at a later date):

[bamboo@localhost ~]$ cat /etc/rc.d/rc.local 
#!/bin/sh
/opt/bamboo-elastic-agent/bin/testIfStartedFromBamboo.sh >/tmp/testIfStartedFromBamboo.log 2>&1
if [ $? -eq 0 ]; then
	/opt/bamboo-elastic-agent/bin/setupEphemeralStorageStructure.sh >> /tmp/setupEphemeralStorageStructure.log
	/opt/bamboo-elastic-agent/bin/restoreEbsSnapshotFromUserData.sh >> /tmp/setupEbsSnapshot.log
	/opt/bamboo-elastic-agent/bin/testVersions.sh >/tmp/testVersions.log 2>&1
	if [ $? -eq 0 ]; then
# matt.clarkson@gmail.com
# Usually the following line is used for the bamboo elastic agent.
# However, using the ampersand does not let the script run on F17
# We have moved the ampersand into the script
# There is a thread going with Atlassian for a fix:
# https://answers.atlassian.com/questions/61970/fedora-17-elastic-image-minor-problem
#		su -c bamboo-elastic-agent - bamboo &
		su -c bamboo-elastic-agent - bamboo
	else
		echo "Bamboo agent not started..."        
	fi

	echo "*/5 * * * * root /opt/bamboo-elastic-agent/bin/checkElasticAgent.sh" > /etc/cron.d/elasticagentcheck
# This file doesn't exist on F17 so we use the system control
#	/etc/init.d/crond restart
	/bin/systemctl restart crond.service
else
	echo "This instance will be shut down immediately."
	/sbin/telinit -t 0 0
fi
[bamboo@localhost ~]$ cat /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent 
#!/bin/bash
trap "exit" ERR
BAMBOO_HOME=`dirname $0`/..
unset BAMBOO_CLASSPATH
for JAR in `find $BAMBOO_HOME/lib`
do
  if [[ -n $BAMBOO_CLASSPATH ]]
  then
    BAMBOO_CLASSPATH=$BAMBOO_CLASSPATH:$JAR
  else
    BAMBOO_CLASSPATH=$JAR
  fi
done
# matt.clarkson@gmail.com
# The /etc/rc.d/rc.local script that comes with
# the install doesn't seem to work on F17 so we have moved
# the ampersand here
java -server -Xms32m -Xmx512m -XX:MaxPermSize=256m -cp $BAMBOO_CLASSPATH com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap 2>&1 | tee -a ~/bamboo-elastic-agent.out &

[Przemek Bruski]

Which shell are you using? I.e. what does

ls -l /bin/sh

show?

Also:

#!bin/sh

should be:

#!/bin/sh

[Matt Clarkson]

Sorry missed out the forward slash. it did have it when i was testing. Fedora uses bash . Thats what /bin/sh is softlinked too.

[Przemek Bruski]

Does

su -c "bamboo-test-script > ~/test.log" - bamboo </dev/null >/dev/null 2>&1 &

work?

This is literally the only thing stopping me from making various distro elastic images :(

I don't think so ;-)

[Przemek Bruski]

OK, now try:

su -c bamboo-elastic-agent - bamboo </dev/null >/tmp/bamboo-elastic-agent.out 2>&1 &

Let's see if there was any interesting output

[Matt Clarkson]

That didn't start up the bamboo agent - but the script ran.

The following

#su -c bamboo-elastic-agent - bamboo &
	su -c "echo matt > ~/matt1.log" - bamboo </dev/null >/dev/null 2>&1 &
	su -c "echo matt > ~/matt2.log" - bamboo </dev/null >/dev/null 2>&1
	su -c "matt-test > ~/matt3.log" - bamboo </dev/null >/dev/null 2>&1 &
	su -c "matt-test > ~/matt4.log" - bamboo </dev/null >/dev/null 2>&1
	su -c "matt-test > ~/matt5.log" - bamboo &
	su -c "matt-test > ~/matt6.log" - bamboo 
	su -c "bamboo-test-script > ~/test.log" - bamboo </dev/null >/dev/null 2>&1 &

Where /usr/bin/matt-test is

[bamboo@localhost ~]$ cat /usr/bin/matt-test && ls -asl /usr/bin/matt-test 

#!/bin/sh

echo matt

4 -rwxr-xr-x. 1 root root 20 Jun 14 03:50 /usr/bin/matt-test

Creates:

[bamboo@localhost ~]$ ls -asl *.log && cat *.log && ps -ef | grep java
4 -rw-rw-r--. 1 bamboo bamboo 5 Jun 14 03:54 matt1.log
4 -rw-rw-r--. 1 bamboo bamboo 5 Jun 14 03:54 matt2.log
4 -rw-rw-r--. 1 bamboo bamboo 5 Jun 14 03:54 matt3.log
4 -rw-rw-r--. 1 bamboo bamboo 5 Jun 14 03:54 matt4.log
4 -rw-rw-r--. 1 bamboo bamboo 5 Jun 14 03:54 matt5.log
4 -rw-rw-r--. 1 bamboo bamboo 5 Jun 14 03:54 matt6.log
0 -rw-rw-r--. 1 bamboo bamboo 0 Jun 14 03:54 test.log
matt
matt
matt
matt
matt
matt
bamboo     879   856  0 03:57 pts/0    00:00:00 grep --color=auto java

This is literally the only thing stopping me from making various distro elastic images :(

I don't think so ;-)

Ha, I can imagine there will be a lot more stumbling blocks along the way ;-)

[Matt Clarkson]

Thanks, I'll try that in a minute. I can't reproduce the previous post at the moment.

I'm starting up the instance, then rebooting through the EC2 console but no /home/bamboo/test.log at the moment.

[Matt Clarkson]

That outputs nothing:

[bamboo@localhost ~]$ ls /tmp
bamboo-elastic-agent.out  hsperfdata_root  setupEbsSnapshot.log  setupEphemeralStorageStructure.log  testIfStartedFromBamboo.log  testVersions.log
[bamboo@localhost ~]$ cat /tmp/bamboo-elastic-agent.out

I've noticed something strange. I now have two test files. One is /usr/bin/matt

[root@localhost bamboo]# ls /usr/bin/matt-test -l
-rwxr-xr-x. 1 root root 21 Jun 14 04:33 /usr/bin/matt-test
[root@localhost bamboo]# cat /usr/bin/matt-test 
#!/bin/sh
echo hello

And the other is /opt/bamboo-elastic-agent/bin/bamboo-test-script

[root@localhost bamboo]# ls -l /opt/bamboo-elastic-agent/bin/bamboo-test-script 
-rwxr-xr-x. 1 root root 450 Jun 14 04:31 /opt/bamboo-elastic-agent/bin/bamboo-test-script
[root@localhost bamboo]# cat /opt/bamboo-elastic-agent/bin/bamboo-test-script 
#!/bin/bash
echo Winning...

With the following /etc/rc.d/rc.local

[root@localhost bamboo]# cat /etc/rc.d/rc.local 
#!/bin/sh
[snip]
	#su -c bamboo-elastic-agent - bamboo &
	su -c "echo matt > ~/matt1.log" - bamboo </dev/null >/dev/null 2>&1 &
	su -c "echo matt > ~/matt2.log" - bamboo </dev/null >/dev/null 2>&1 
	su -c "matt-test > ~/matt3.log" - bamboo </dev/null >/dev/null 2>&1 &
	su -c "matt-test > ~/matt4.log" - bamboo </dev/null >/dev/null 2>&1
	su -c 'set > ~/set.log' - bamboo </dev/null >/dev/null 2>&1
	su -c "bamboo-test-script > ~/test.log" - bamboo </dev/null >/dev/null 2>&1 &
	su -c bamboo-elastic-agent - bamboo </dev/null >/tmp/bamboo-elastic-agent.out 2>&1 &
[snip]

The following gets created:

[root@localhost bamboo]# ls && cat matt?.log
matt1.log  matt2.log  matt3.log  matt4.log  set.log
matt
matt
hello
hello

Note that there is no test.log that should contain "Winning..."

The PATH in set.log is:

PATH=/opt/bamboo-elastic-agent/bin:/opt/ec2-api-tools/bin:/usr/lib/jvm/java-openjdk/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/bamboo/.local/bin:/home/bamboo/bin

Which is correct for the system I am on.

[Matt Clarkson]

The Fedora image I started with was form here:

http://fedoraproject.org/wiki/Cloud_images

They are very minimal images, could I be missing something that might be needed in the scripts?

[Przemek Bruski]

Hmm, try using an absolute path when staring the bamboo startup script and put something like touch /tmp/asd as the first line of that script to see whether it runs at all.

[Matt Clarkson]

I added the follow to /etc/rc.d/rc.local:

su -c /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent - bamboo </dev/null >/tmp/bamboo-elastic-agent.out 2>&1 &

And the following

[root@ip-10-243-149-134 bamboo]# cat /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent 
#!/bin/bash
touch /tmp/bamboo-elastic-agent-script-ran
trap "exit" ERR
[snip]

And /tmp looks like this:

[bamboo@ip-10-190-95-16 ~]$ ls /tmp/
checkElasticAgent.log  hsperfdata_root  setupEbsSnapshot.log  setupEphemeralStorageStructure.log  testIfStartedFromBamboo.log  testVersions.log

I am so confused how that line isn't doing anything in rc.local... :'-(

I noticed that Fedora 17 doesn't have /etc/init.d/crond:

[root@ip-10-243-149-134 bamboo]# ls /etc/init.d/ -asl
total 40
 4 drwxr-xr-x.  2 root root  4096 May 16 19:57 .
 4 drwxr-xr-x. 10 root root  4096 Jun 14 07:44 ..
20 -rw-r--r--.  1 root root 19914 Mar 16 10:26 functions
 4 -rwxr-xr-x.  1 root root  2989 Mar 16 10:26 netconsole
 8 -rwxr-xr-x.  1 root root  7199 Mar 16 10:26 network

But I have the service:

[root@ip-10-243-149-134 bamboo]# /sbin/service crond status
Redirecting to /bin/systemctl  status crond.service
crond.service - Command Scheduler
	  Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled)
	  Active: active (running) since Thu, 14 Jun 2012 07:37:52 -0400; 11min ago
	Main PID: 343 (crond)
	  CGroup: name=systemd:/system/crond.service
		  └ 343 /usr/sbin/crond -n

Jun 14 07:37:53 localhost.localdomain /usr/sbin/crond[343]: (CRON) INFO (running with inotify support)

Also I can boot the AMI in the EC2 console and it doesn't terminate, so the following in /etc/rc.d/rc.local should run.

else
	echo "This instance will be shut down immediately."
	/sbin/telinit -t 0 0
fi

[Matt Clarkson]

Forgot to metion, this is with OnDemand, so using bamboo-elastic-agent-3.0.4