How to configure pipeline to send prod and staging builds to different aws ec2 with code deploy?

[Muhammad Shahid]

Hi Team,

We have two ec2 instances one is for production and 2nd one is for development/staging.  Similarly we have two branches, master and develop. Master's build is in used at production ec2 instance and develop is for staging server.

I have configured bitbucket-pipeline.yml with branches flow.  But how I can control appsec.yml or any other feature to manage flow of builds to copy over just to relevant ec2 instances.   At AWS codedeploy I can configure deployment group and when there would be commit either at master or develop branch, that build will be pushed to codedeploy deployment process and it will push code to deployment group where both servers are in same deployment group.

So question is how and where we can control build deployment to respective instance only with feature bitbucket to aws codedeploy.

[Oleksandr Kyrdan]

Hi @Muhammad Shahid ,

if you need AWS Code Deploy in your Bitbucket Pipelines you can use pipe aws-code-deploy 

Also, more available pipes you can discover on the Bitbucket Pipes page.

Cheers,
Alex

[Muhammad Shahid]

.

[Jeroen De Raedt]

Ah right I see. 

I suspect that the python script you use has hardcoded in it to use a certain environment variable to configure the AWS deployment group? 

In the step that does a deployment, you could add a command to your script in bitbucket-pipelines.yml that overwrites the value of your deployment group variable.

For example: 

pipelines: 
 branches: 
 develop: 
 - step:
 name: Run Build and Deploy to Staging Server
 image: node:6
 script:
 - npm install
 - npm run build 
 - export DEPLOYMENT_GROUP="STAGING"
 - codedeploy_deploy.py # run the deployment script 
 master:
 - step:
 name: Run Build and Deploy to Prod Server
 image: node:6
 script:
 - npm install
 - npm run build
 - export DEPLOYMENT_GROUP="PRODUCTION"
 - python codedeploy_deploy.py

[BillyP]

Jeroen, 

It looks like this will deploy to both groups at the same time.  Is that correct?

[Muhammad Shahid]

No, That can be controller through bitbucket-pipeline.yml branch flow. Actual thing was at code deploy deployment and now if we can over ride environment variables through deployment script that would be super cool. Will give it a try. @Jeroen De Raedt can you please share any possible deployment script?  or edit this one :

https://bitbucket.org/awslabs/aws-codedeploy-bitbucket-pipelines-python/src/73b7c31b0a72a038ea0a9b46e457392c45ce76da/codedeploy_deploy.py?at=master&fileviewer=file-view-default

[networkredux]

Muhammad Shahid / Billy Poggi /Jeroen De Raedt

Could you please share the current status on this. Will this  export command overrides Deployment group environment variables?

Do we need to change anything on the python script. I'm stuck here on this. Kindly advise. 

Thanks in advance

[Muhammad Shahid]

@networkreduxNot yet, busy with other tasks and still not sure variables of python script may override environmental variables of pipelines.  Please advise if you go through with it. Thanks

[Aswin Satish]

Hi all,

We are following this same procedure for deployment, except that the deployment goes to same server, but to different paths in 2 appsec.yml, one for master and other development branch. The issue is that when commit occurs in any branch, the files in other branch are getting removed. For eg: if we commit to master, all the files that are already in development branch get deleted.

Can anyone suggest a solution? 

[Aswin Satish]

Any one, please

[Aswin Satish]

@Muhammad Shahid

Do you have any ideas regarding my issue?

[Graham Gatus]

@Aswin Satish One solution is to use 2 appspec files, one per branch:

appspec-master.yml:

version: 0.0
os: linux

files:
  - source: ./
    destination: /opt/master-path

appspec-develop.yml:

version: 0.0
os: linux

files:
  - source: ./
    destination: /opt/develop-path

In your pipelines config, depending on the branch, you could assemble your CodeDeploy artifact using either file Appspec file, e.g

pipelines:
   default:
      - step:
        script:
          - run-tests.sh
   branches:
     master:
        - step:
            script:
              - mkdir application
              - mv appspec-master.yml application/appspec.yml
              - <add other files to the application directory>
              - >
                aws deploy push --application-name <your app> \
                  --source application \
                  --s3-location s3://<your bucket>/<revision key>
     develop:
         - step:
             script: 
               - mkdir application
               - mv appspec-master.yml application/appspec.yml
               - <add other files to the application directory>
               - > 
                 aws deploy push --application-name <your app> \
                   --source application \
                   --s3-location s3://<your bucket>/<revision key>

There is a $BITBUCKET_BRANCH environment variable that you could also use to generate a appspec.yml file dynamically. You could use it to generate an appspec.yml file where the files destination is the branch name. That way, each branch would get deployed to its own path on the EC2 instance.

[Aswin Satish]

@Graham Gatus Thanks for your reply. 

I'm using different appspec files for master and production branches:

Master:

version: 0.0
os: linux
files: 
   - source: /
     destination: /home/ec2-user/

Production:

version: 0.0
os: linux
files: 
   - source: /
     destination: /var/www/

My pipeline file for master:

pipelines: 
  branches: 
    master:
      - step:
          image: python:3.5.1 
          script: 
            - apt-get update # required to install zip 
            - apt-get install -y zip # required for packaging up the application 
            - pip install boto3==1.3.0 # required for codedeploy_deploy.py 
            - zip -r /tmp/artifact.zip * # package up the application for deployment 
            - python codedeploy_deploy.py # run the deployment script

For production:

pipelines: 
  branches: 
    production:
      - step:
          image: python:3.5.1 
          script: 
            - apt-get update # required to install zip 
            - apt-get install -y zip # required for packaging up the application 
            - pip install boto3==1.3.0 # required for codedeploy_deploy.py 
            - zip -r /tmp/artifact.zip * # package up the application for deployment 
            - python codedeploy_deploy.py # run the deployment script

And I'm using a codedeploy file which contains the following variables:

APPLICATION_NAME: Name of codedeploy application

S3_BUCKET: Name of s3 bucket

DEPLOYMENT_GROUP_NAME: Codedeploy group name

DEPLOYMENT_CONFIG: CodeDeployDefault.OneAtATime

AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION

AWS_ACCESS_KEY_ID

I need my code in bitbucket to be copied to ​/home/ec2-user when any push or merge happens in master branch and to /var/www/ when any push or merge happens in production branch. 

Everything is working fine, however after a push or merge in one branch, on another push to other branch overwrite files in first branch

[Aswin Satish]

Any ideas ?

[Aswin Satish]

Still waiting for a reply..Can anyone help, please

[tjrchampionxyz]

@Aswin SatishI know this is more than a little late, but if you're deploying a staging/production version of your app to the the same EC2 instance, in order for CodeDeploy to not delete the previously deployed app (lets say develop) you should create two CodeDeploy applications in the AWS console and then define the application_name and deployment_group in your bitbucket-piplines.yaml file for each branch.

Hopefully this is useful for anyone else who visits this page in 2019.. It worked for me and I'm now able to deploy to both staging/prod on one EC2 instance.

[BillyP]

Hi Muhammad, 

I was reading up on your issue and I wanted to learn more about the situation.  In doing so, I found this page.

https://hackernoon.com/deploy-to-ec2-with-aws-codedeploy-from-bitbucket-pipelines-4f403e96d50c

How do you currently select which system to deploy to? dev/ staging

-Billy

[Muhammad Shahid]

Thank you Billy, I have check out this post and my setup is 100% relevant to this post.  This setup is good for one deployment group. For example in DG1 (CodeDeploy) group if I put both servers production and staging and when pipelines will run and push code/build at that point how should I control deployment to just staging or just to production?

This is my bitbucker-pipeline.yml

pipelines: branches: develop: - step: name: Run Build and Deploy to Staging Server image: node:6 script: npm install - npm run build  - codedeploy_deploy.py # run the deployment script  branches: master: - step: name: Run Build and Deploy to Prod Server image: node:6 script: - npm install - npm run build  - python codedeploy_deploy.py

appsec.yml

version: 0.0os: linuxfiles: - source: / destination: /var/www/apphooks: BeforeInstall: - location: scripts/install_dependencies timeout: 300 runas: root - location: scripts/start_server timeout: 300 runas: root ApplicationStop: - location: scripts/stop_server timeout: 300 runas: root

[Jeroen De Raedt]

Am I correct in assuming that the real difference is that you want to use different appsec.yml files for staging and production? 

If that is the case, 1 solution could be to create 2 directories in your repository (/staging and /production) and in each you store the appropriate version of the appsec.yml file. When doing deployments, you change directories to the appropriate folder and then execute the codedeploy script. 

[Muhammad Shahid]

Thanks for reply, basically issue is not handle appsec.yml.  We can have similar path for staging and production server.  Issue is at AWS deployment group.  There we can select deployment group with bunch of ec2 instances.  But how we can deploy to just staging and production instances separately.  At pipeline we can define environmental variables but for singal deployment group. So how to deploy to staging and just for production deployment groups.