Deploy to AWS Beanstalk fails: remove environment from a VPC

[danbrellis]

I am able to successfully deploy to my dev environment from my CLI with `eb deploy`.

I have set up a pipeline, which successfully zips the app and uploads it, but Beanstalk fails to deploy.

Relevant Beanstalk Events:

ERROR Failed to deploy application.
ERROR Missing value for required parameter: InstanceType
ERROR Missing value for required parameter: InstanceTypeFamily
ERROR You cannot remove an environment from a VPC. Launch a new environment outside the VPC.
ERROR "option_settings" in one of the configuration files failed validation. More details to follow.

In my .ebextensions I have option_settings:

option_settings: 
  aws:autoscaling:launchconfiguration:
    InstanceType: t3.medium
    SecurityGroups: sg-1111111111
    IamInstanceProfile: arn:aws:iam::1111111111:instance-profile/my-profile-IAMEBInstanceProfile-AAAAAAAAAAAAA
  aws:autoscaling:asg:
    MinSize: '2'
    MaxSize: '5'
  aws:ec2:vpc:
    VPCId: "vpc-1111111111"
    Subnets: "subnet-1111111111,subnet-1111111111"
    ELBSubnets: "subnet-1111111111,subnet-1111111111"
  aws:elbv2:listener:443:
    ListenerEnabled: 'true'
    Protocol: HTTPS
    SSLCertificateArns: arn:aws:acm:us-east-1:1111111111:certificate/1111111111

Here's the pipeline:

image: atlassian/default-image:2

pipelines:
  branches:
    master:
      - step:
          name: "Zip Project Files"
          script:
            - zip -r application.zip *
          artifacts:
            - application.zip
      - step:
          name: "Deploy to Production"
          deployment: production
          script:
            - pipe: atlassian/aws-elasticbeanstalk-deploy:0.6.7
              variables:
                AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
                AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
                AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
                APPLICATION_NAME: $APPLICATION_NAME
                ENVIRONMENT_NAME: $ENVIRONMENT_NAME
                ZIP_FILE: "application.zip"
                S3_BUCKET: $S3_BUCKET

$APPLICATION_NAME and $ENVIRONMENT_NAME match what's shown in AWS console.

I can provide output from Bitbucket pipeline log if helpful.

If anyone has insight into why the error is occurring/how to fix it. Not sure if I'm missing setting something explicitly. Would think since it works fine from my local cli it should work from the pipe with the same files. Thanks.

[danbrellis · Accepted solution]

I figured this out. In my bitbucket-pipelines.yml in the script that creates the application.zip artifact, I was running a zip command that did not include hidden files/directories.

I replaced 

zip -r application.zip *

with

git archive -o application.zip master 

and everything worked as expected.

Thanks for the help. 

[Halyna Berezovska]

@danbrellis hello! Thanks for using the pipe.

If this works locally for you, then we would like to see logs of pipeline's failing indeed , so that we know where is the issue root cause - aws, wrong config or our side.

For now I can only say that AWS has similar threads related to this error, but let's see the logs first.

Regards, Galyna

[danbrellis]

Hi Galyna, thanks for the reply!

I just ran the pipe and again got a success message from bitbucket:

+ umask 000

+ GIT_LFS_SKIP_SMUDGE=1 retry 6 git clone --branch="master" --depth 50 https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/$BITBUCKET_REPO_FULL_NAME.git $BUILD_DIR
Cloning into '/opt/atlassian/pipelines/agent/build'...

+ git reset --hard 2a5b5070b33915f3784413d71d909c5e2757fd80
HEAD is now at 2a5b507 Adds fade transition to videoPosterPlayer

+ git config user.name bitbucket-pipelines

+ git config user.email commits-noreply@bitbucket.org

+ git config push.default current

+ git config http.${BITBUCKET_GIT_HTTP_ORIGIN}.proxy http://localhost:29418/

+ git remote set-url origin http://bitbucket.org/$BITBUCKET_REPO_FULL_NAME

+ git reflog expire --expire=all --all

+ echo ".bitbucket/pipelines/generated" >> .git/info/exclude

+ chmod 777 $BUILD_DIR

Artifact "application.zip": Downloading
Artifact "application.zip": Downloaded 6.5 MiB in 0 seconds
Artifact "application.zip": Extracting
Artifact "application.zip": Extracted in 0 seconds

Images used:
build : docker.io/atlassian/default-image@sha256:d8ae266b47fce4078de5d193032220e9e1fb88106d89505a120dfe41cb592a7b
+ docker container run \
--volume=/opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build \
--volume=/opt/atlassian/pipelines/agent/ssh:/opt/atlassian/pipelines/agent/ssh:ro \
--volume=/usr/local/bin/docker:/usr/local/bin/docker:ro \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes \
--volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/aws-elasticbeanstalk-deploy:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/aws-elasticbeanstalk-deploy \
--workdir=$(pwd) \
--label=org.bitbucket.pipelines.system=true \
--env=BITBUCKET_STEP_TRIGGERER_UUID="$BITBUCKET_STEP_TRIGGERER_UUID" \
--env=BITBUCKET_REPO_FULL_NAME="$BITBUCKET_REPO_FULL_NAME" \
--env=BITBUCKET_GIT_HTTP_ORIGIN="$BITBUCKET_GIT_HTTP_ORIGIN" \
--env=BITBUCKET_PROJECT_UUID="$BITBUCKET_PROJECT_UUID" \
--env=BITBUCKET_REPO_IS_PRIVATE="$BITBUCKET_REPO_IS_PRIVATE" \
--env=BITBUCKET_WORKSPACE="$BITBUCKET_WORKSPACE" \
--env=BITBUCKET_DEPLOYMENT_ENVIRONMENT_UUID="$BITBUCKET_DEPLOYMENT_ENVIRONMENT_UUID" \
--env=BITBUCKET_REPO_OWNER_UUID="$BITBUCKET_REPO_OWNER_UUID" \
--env=BITBUCKET_BRANCH="$BITBUCKET_BRANCH" \
--env=BITBUCKET_REPO_UUID="$BITBUCKET_REPO_UUID" \
--env=BITBUCKET_PROJECT_KEY="$BITBUCKET_PROJECT_KEY" \
--env=BITBUCKET_DEPLOYMENT_ENVIRONMENT="$BITBUCKET_DEPLOYMENT_ENVIRONMENT" \
--env=BITBUCKET_REPO_SLUG="$BITBUCKET_REPO_SLUG" \
--env=CI="$CI" \
--env=BITBUCKET_REPO_OWNER="$BITBUCKET_REPO_OWNER" \
--env=BITBUCKET_STEP_RUN_NUMBER="$BITBUCKET_STEP_RUN_NUMBER" \
--env=BITBUCKET_BUILD_NUMBER="$BITBUCKET_BUILD_NUMBER" \
--env=BITBUCKET_GIT_SSH_ORIGIN="$BITBUCKET_GIT_SSH_ORIGIN" \
--env=BITBUCKET_PIPELINE_UUID="$BITBUCKET_PIPELINE_UUID" \
--env=BITBUCKET_COMMIT="$BITBUCKET_COMMIT" \
--env=BITBUCKET_CLONE_DIR="$BITBUCKET_CLONE_DIR" \
--env=PIPELINES_JWT_TOKEN="$PIPELINES_JWT_TOKEN" \
--env=BITBUCKET_STEP_UUID="$BITBUCKET_STEP_UUID" \
--env=BITBUCKET_DOCKER_HOST_INTERNAL="$BITBUCKET_DOCKER_HOST_INTERNAL" \
--env=DOCKER_HOST="tcp://host.docker.internal:2375" \
--env=BITBUCKET_PIPE_SHARED_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes" \
--env=BITBUCKET_PIPE_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/aws-elasticbeanstalk-deploy" \
--env=APPLICATION_NAME="$APPLICATION_NAME" \
--env=AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
--env=AWS_DEFAULT_REGION="$AWS_DEFAULT_REGION" \
--env=AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
--env=ENVIRONMENT_NAME="$ENVIRONMENT_NAME" \
--env=S3_BUCKET="$S3_BUCKET" \
--env=ZIP_FILE="application.zip" \
--add-host="host.docker.internal:$BITBUCKET_DOCKER_HOST_INTERNAL" \
bitbucketpipelines/aws-elasticbeanstalk-deploy:0.6.7@sha256:6e3c4e50faf97de2893d3e294e8cff2220e0514cfa7bbef5eb54e9621f345a8e
Unable to find image 'bitbucketpipelines/aws-elasticbeanstalk-deploy:0.6.7@sha256:6e3c4e50faf97de2893d3e294e8cff2220e0514cfa7bbef5eb54e9621f345a8e' locally
sha256:6e3c4e50faf97de2893d3e294e8cff2220e0514cfa7bbef5eb54e9621f345a8e: Pulling from bitbucketpipelines/aws-elasticbeanstalk-deploy
5d20c808ce19: Pulling fs layer
5600340360cf: Pulling fs layer
d1d4a9b4e3a6: Pulling fs layer
ae21185d0525: Pulling fs layer
68a6b5328b33: Pulling fs layer
ae21185d0525: Waiting
68a6b5328b33: Waiting
d1d4a9b4e3a6: Verifying Checksum
d1d4a9b4e3a6: Download complete
5d20c808ce19: Verifying Checksum
5d20c808ce19: Download complete
5d20c808ce19: Pull complete
ae21185d0525: Verifying Checksum
ae21185d0525: Download complete
68a6b5328b33: Download complete
5600340360cf: Verifying Checksum
5600340360cf: Download complete
5600340360cf: Pull complete
d1d4a9b4e3a6: Pull complete
ae21185d0525: Pull complete
68a6b5328b33: Pull complete
Digest: sha256:6e3c4e50faf97de2893d3e294e8cff2220e0514cfa7bbef5eb54e9621f345a8e
Status: Downloaded newer image for bitbucketpipelines/aws-elasticbeanstalk-deploy@sha256:6e3c4e50faf97de2893d3e294e8cff2220e0514cfa7bbef5eb54e9621f345a8e
 [36mINFO: The application source bundle doesn't have a known file extension (zip, jar or war). This might cause some issues. [0m
 [36mINFO: Uploading to s3 bucket: elasticbeanstalk-us-east-1-320236742817... [0m
Completed 256.0 KiB/6.5 MiB (2.7 MiB/s) with 1 file(s) remaining
Completed 512.0 KiB/6.5 MiB (5.3 MiB/s) with 1 file(s) remaining
Completed 768.0 KiB/6.5 MiB (7.8 MiB/s) with 1 file(s) remaining
Completed 1.0 MiB/6.5 MiB (10.3 MiB/s) with 1 file(s) remaining 
Completed 1.2 MiB/6.5 MiB (12.6 MiB/s) with 1 file(s) remaining 
Completed 1.5 MiB/6.5 MiB (15.0 MiB/s) with 1 file(s) remaining 
Completed 1.8 MiB/6.5 MiB (17.3 MiB/s) with 1 file(s) remaining 
Completed 2.0 MiB/6.5 MiB (19.5 MiB/s) with 1 file(s) remaining 
Completed 2.2 MiB/6.5 MiB (21.8 MiB/s) with 1 file(s) remaining 
Completed 2.5 MiB/6.5 MiB (23.9 MiB/s) with 1 file(s) remaining 
Completed 2.8 MiB/6.5 MiB (25.2 MiB/s) with 1 file(s) remaining 
Completed 3.0 MiB/6.5 MiB (27.0 MiB/s) with 1 file(s) remaining 
Completed 3.2 MiB/6.5 MiB (28.8 MiB/s) with 1 file(s) remaining 
Completed 3.5 MiB/6.5 MiB (30.6 MiB/s) with 1 file(s) remaining 
Completed 3.8 MiB/6.5 MiB (32.3 MiB/s) with 1 file(s) remaining 
Completed 4.0 MiB/6.5 MiB (34.1 MiB/s) with 1 file(s) remaining 
Completed 4.2 MiB/6.5 MiB (35.6 MiB/s) with 1 file(s) remaining 
Completed 4.5 MiB/6.5 MiB (37.2 MiB/s) with 1 file(s) remaining 
Completed 4.8 MiB/6.5 MiB (38.7 MiB/s) with 1 file(s) remaining 
Completed 5.0 MiB/6.5 MiB (40.2 MiB/s) with 1 file(s) remaining 
Completed 5.2 MiB/6.5 MiB (41.7 MiB/s) with 1 file(s) remaining 
Completed 5.5 MiB/6.5 MiB (43.1 MiB/s) with 1 file(s) remaining 
Completed 5.8 MiB/6.5 MiB (44.1 MiB/s) with 1 file(s) remaining 
Completed 6.0 MiB/6.5 MiB (45.1 MiB/s) with 1 file(s) remaining 
Completed 6.2 MiB/6.5 MiB (45.9 MiB/s) with 1 file(s) remaining 
Completed 6.5 MiB/6.5 MiB (32.1 MiB/s) with 1 file(s) remaining 
upload: ./application.zip to s3://elasticbeanstalk-us-east-1-320236742817/baybackpack/baybackpack-9-2a5b5070.zip
 [32m✔ Artifact uploaded successfully to s3://elasticbeanstalk-us-east-1-320236742817/baybackpack/baybackpack-9-2a5b5070.zip [0m
 [36mINFO: Creating application version in Elastic Beanstalk... [0m
{
"ApplicationVersion": {
"ApplicationName": "baybackpack", 
"Status": "UNPROCESSED", 
"VersionLabel": "baybackpack-9-2a5b5070", 
"Description": "Application version created from https://bitbucket.org/danbrellis/bay-backpack/addon/pipelines/home#!/results/9", 
"ApplicationVersionArn": "arn:aws:elasticbeanstalk:us-east-1:320236742817:applicationversion/baybackpack/baybackpack-9-2a5b5070", 
"DateCreated": "2020-12-11T16:01:25.603Z", 
"DateUpdated": "2020-12-11T16:01:25.603Z", 
"SourceBundle": {
"S3Bucket": "elasticbeanstalk-us-east-1-320236742817", 
"S3Key": "baybackpack/baybackpack-9-2a5b5070.zip"
}
}
}
 [32m✔ Application version baybackpack-9-2a5b5070 successfully created in Elastic Beanstalk. [0m
 [36mINFO: Updating environment in Elastic Beanstalk... [0m
{
"ApplicationName": "baybackpack", 
"EnvironmentName": "baybackpack-dev", 
"VersionLabel": "baybackpack-9-2a5b5070", 
"Status": "Updating", 
"Description": "Environment created from the EB CLI using \"eb create\"", 
"EnvironmentArn": "arn:aws:elasticbeanstalk:us-east-1:320236742817:environment/baybackpack/baybackpack-dev", 
"PlatformArn": "arn:aws:elasticbeanstalk:us-east-1::platform/PHP 7.4 running on 64bit Amazon Linux 2/3.1.3", 
"EndpointURL": "awseb-AWSEB-SPMAWE88IDPS-1080691972.us-east-1.elb.amazonaws.com", 
"SolutionStackName": "64bit Amazon Linux 2 v3.1.3 running PHP 7.4", 
"EnvironmentId": "e-c4dzsm4pr3", 
"CNAME": "baybackpack-dev.us-east-1.elasticbeanstalk.com", 
"AbortableOperationInProgress": true, 
"Tier": {
"Version": "1.0", 
"Type": "Standard", 
"Name": "WebServer"
}, 
"Health": "Grey", 
"DateUpdated": "2020-12-11T16:01:26.687Z", 
"DateCreated": "2020-12-01T18:39:34.072Z"
}
 [36mINFO: Deploying to environment "baybackpack-dev". Previous version: "app-cd5f-201206_144844" -> New version: "baybackpack-9-2a5b5070". [0m
 [32m✔ Deployment triggered successfully. URL: http://baybackpack-dev.us-east-1.elasticbeanstalk.com [0m
 [36mINFO: You can follow your deployment at https://console.aws.amazon.com/elasticbeanstalk/home?region=us-east-1#/environment/dashboard?applicationName=baybackpack&environmentId=e-c4dzsm4pr3 [0m
Searching for files matching artifact pattern .bitbucket/pipelines/generated/pipeline/pipes/**

Searching for test report files in directories named [test-results, failsafe-reports, test-reports, surefire-reports] down to a depth of 4
Finished scanning for test reports. Found 0 test report files.
Merged test suites, total number tests is 0, with 0 failures and 0 errors.

However, I once again got a failed AWS deployment. Can provide AWS logs, but not sure which one(s) would be useful. I'm not able to see any obvious errors in them other than the errors posted in the AWS Events (same as I posted above).

Thanks!

[Halyna Berezovska]

@danbrellis As in the pipe we cannot see the traceback, it would be useful to dive in to appropriate AWS cloudformation stack or if you don't have one, look into Environment logs. If you go to the appropriate environment dashboard , you can see the bunch of logs at the bottom of the page. If you find the error you think it can help to discover you or us the problem, share it with us, please.

In the meantime we will create a task to try reproducing the issue

Regards, Galyna

[danbrellis]

Galyna,

Thanks for your help, but I was able to find a solution which I posted above: https://community.atlassian.com/t5/Bitbucket-Pipelines-questions/Deploy-to-AWS-Beanstalk-fails-remove-environment-from-a-VPC/qaq-p/1550030#M3872

[Halyna Berezovska]

@danbrellis I am glad you figured this out, thanks!