Please update Boto3 to >= 1.18.4 for aws-lambda-deploy

[sergio_alba]

Hi!

The current version for Boto3 is below 1.18.4, where a ResourceConflictException error related to AWS Lambda Deploy has been fixed (src). This error is related to the changes explained here by the AWS team.

---

Pipeline output:

d0f52b114f67: Pull complete

bed928cd028b: Pull complete

Digest: sha256:04ecf0a4530e7cdafe22a638ddcd8f63c4e308f77b1e4ea1e1dd4aa106656b00

Status: Downloaded newer image for bitbucketpipelines/aws-lambda-deploy:1.1.0

INFO: Using default authentication with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

DEBUG: Starting new HTTPS connection (1): bitbucket.org:443

DEBUG: https://bitbucket.org:443 "GET /bitbucketpipelines/official-pipes/raw/master/pipes.prod.json HTTP/1.1" 200 None

INFO: Executing the aws-lambda-deploy pipe...

DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane

DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway

DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict

DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration

DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53

DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search

DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section

DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask

DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section

DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search

DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section

DEBUG: IMDS ENDPOINT: http://169.254.169.254/

DEBUG: Looking for credentials via: env

INFO: Found credentials in environment variables.

DEBUG: Loading JSON file: /usr/local/lib/python3.9/site-packages/botocore/data/endpoints.json

DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fa511c20f70>

DEBUG: Loading JSON file: /usr/local/lib/python3.9/site-packages/botocore/data/lambda/2015-03-31/service-2.json

DEBUG: Event creating-client-class.lambda: calling handler <function add_generate_presigned_url at 0x7fa511c4d790>

DEBUG: Setting lambda timeout as (60, 60)

DEBUG: Loading JSON file: /usr/local/lib/python3.9/site-packages/botocore/data/_retry.json

DEBUG: Registering retry handlers for service: lambda

INFO: Updating Lambda function configuration.

DEBUG: Event before-parameter-build.lambda.UpdateFunctionConfiguration: calling handler <function generate_idempotent_uuid at 0x7fa511bca670>

DEBUG: Event before-call.lambda.UpdateFunctionConfiguration: calling handler <function inject_api_version_header_if_needed at 0x7fa511bcbee0>

DEBUG: Making request for OperationModel(name=UpdateFunctionConfiguration) with params: {'url_path': '/2015-03-31/functions/xxxx/configuration', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.17.112 Python/3.9.6 Linux/5.4.92-flatcar Botocore/1.20.112'}, 'body': b'{"Role": "arn:aws:iam::205746061435:role/Nexplayer-License-Lambda", "Handler": "index.lambdaHandler", "Description": "DEV API v2 deployment", "Timeout": 30, "MemorySize": 256, "Environment": {"Variables": {"DEBUG": "true", "S3ApiVersion": "2006-03-01", "S3BucketName": "nexplayer-license-dev", "S3Region": "us-east-1"}}, "Layers": ["arn:aws:lambda:us-east-1:205746061435:layer:nexplayer-logger:6", "arn:aws:lambda:us-east-1:205746061435:layer:nexplayer-common:16", "arn:aws:lambda:us-east-1:205746061435:layer:psl:1", "arn:aws:lambda:us-east-1:205746061435:layer:axios:2"], "Runtime": "nodejs14.x"}', 'url': '', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fa5110f3100>, 'has_streaming_input': False, 'auth_type': None}}

DEBUG: Event request-created.lambda.UpdateFunctionConfiguration: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa5110f3070>>

DEBUG: Event choose-signer.lambda.UpdateFunctionConfiguration: calling handler <function set_operation_specific_signer at 0x7fa511bca550>

DEBUG: Calculating signature using v4 auth.

DEBUG: CanonicalRequest:

PUT

/2015-03-31/functions/xxxxx/configuration

host:lambda.us-east-1.amazonaws.com

x-amz-date:20211005T075411Z

host;x-amz-date

5a462927d44c5d995dd83c3c680c03fa0c800043545d962072da3c82fa65e0e4

DEBUG: StringToSign:

AWS4-HMAC-SHA256

20211005T075411Z

20211005/us-east-1/lambda/aws4_request

a188161dbb945e68d163fd50c27d0ef084134a3dcfea3d8cf3b05220d2d345c5

DEBUG: Signature:

df3e3ecc4ba0f1a04a8c0be2838aefa434fff06b16cd9fe60fd7add846528006

DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/xxxx/configuration, headers={'User-Agent': b'Boto3/1.17.112 Python/3.9.6 Linux/5.4.92-flatcar Botocore/1.20.112', 'X-Amz-Date': b'20211005T075411Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=$AWSID/20211005/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=df3e3ecc4ba0f1a04a8c0be2838aefa434fff06b16cd9fe60fd7add846528006', 'Content-Length': '598'}>

DEBUG: Certificate path: /usr/local/lib/python3.9/site-packages/certifi/cacert.pem

DEBUG: Starting new HTTPS connection (1): lambda.us-east-1.amazonaws.com:443

DEBUG: https://lambda.us-east-1.amazonaws.com:443 "PUT /2015-03-31/functions/xxxxx/configuration HTTP/1.1" 200 1825

DEBUG: Response headers: {'Date': 'Tue, 05 Oct 2021 07:54:11 GMT', 'Content-Type': 'application/json', 'Content-Length': '1825', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'f602613b-21ca-4e66-be6e-47b2f01fcbb1'}

DEBUG: Response body:

b'{"Description":"DEV API v2 deployment","TracingConfig":{"Mode":"PassThrough"},"VpcConfig":{"VpcId":"","SecurityGroupIds":[],"SubnetIds":[]},"SigningJobArn":null,"RevisionId":"3814806b-7ac5-4944-9c26-60cf087e72f8","LastModified":"2021-10-05T07:54:11.000+0000","FileSystemConfigs":null,"FunctionName":"xxx","Runtime":"nodejs14.x","Version":"$LATEST","PackageType":null,"LastUpdateStatus":"InProgress","Layers":","KMSKeyArn":null,"MemorySize":256,"ImageConfigResponse":null,"LastUpdateStatusReason":"The function is being created.","DeadLetterConfig":null,"Timeout":30,"Handler":"index.lambdaHandler","CodeSha256":"aoyZfk2bd+c28fwRU+sC5oUiZu5FHhyEeFwDmk6Z/tU=","Role":"arn:aws:iam::205746061435:role//***/","SigningProfileVersionArn":null,"MasterArn":null,"CodeSize":16107,"State":"Active","StateReason":null,"Environment":{"Variables":{"S3Region":"us-east-1","S3ApiVersion":"2006-03-01","S3BucketName":"***","DEBUG":"true"},"Error":null},"StateReasonCode":null,"LastUpdateStatusReasonCode":"Creating","Architectures":["x86_64"]}'

DEBUG: Event needs-retry.lambda.UpdateFunctionConfiguration: calling handler <botocore.retryhandler.RetryHandler object at 0x7fa5110f3a60>

DEBUG: No retry needed.

✔ Lambda configuration update succeeded.

INFO: Updating Lambda function.

DEBUG: Event before-parameter-build.lambda.UpdateFunctionCode: calling handler <function generate_idempotent_uuid at 0x7fa511bca670>

DEBUG: Event before-call.lambda.UpdateFunctionCode: calling handler <function inject_api_version_header_if_needed at 0x7fa511bcbee0>

DEBUG: Making request for OperationModel(name=UpdateFunctionCode) with params: {'url_path': '/2015-03-31/functions/***/code', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.17.112 Python/3.9.6 Linux/5.4.92-flatcar Botocore/1.20.112'}, 'body': b'{"Publish": true, "ZipFile": , 'url': '', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fa5110f3100>, 'has_streaming_input': False, 'auth_type': None}}

DEBUG: Event request-created.lambda.UpdateFunctionCode: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa5110f3070>>

DEBUG: Event choose-signer.lambda.UpdateFunctionCode: calling handler <function set_operation_specific_signer at 0x7fa511bca550>

DEBUG: Calculating signature using v4 auth.

DEBUG: CanonicalRequest:

PUT

/2015-03-31/functions/****/code

host:lambda.us-east-1.amazonaws.com

x-amz-date:20211005T075411Z

host;x-amz-date

43ec3d19fdba9634a61fb714d9d2b0c1a5b14a3f4f96fe1dffcf2e52b97b5706

DEBUG: StringToSign:

AWS4-HMAC-SHA256

20211005T075411Z

20211005/us-east-1/lambda/aws4_request

ac68570bf41afea71521dac5989439e6b38c726adb91f1ff2fb26a62b1115814

DEBUG: Signature:

49ebcda9365ecf57d7ae62b1b9bff742c6a81212eea2261f1397419830860b10

DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/****/code, headers={'User-Agent': b'Boto3/1.17.112 Python/3.9.6 Linux/5.4.92-flatcar Botocore/1.20.112', 'X-Amz-Date': b'20211005T075411Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=$AWSID/20211005/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=49ebcda9365ecf57d7ae62b1b9bff742c6a81212eea2261f1397419830860b10', 'Content-Length': '23060'}>

DEBUG: Certificate path: /usr/local/lib/python3.9/site-packages/certifi/cacert.pem

DEBUG: https://lambda.us-east-1.amazonaws.com:443 "PUT /2015-03-31/functions/xxxx/code HTTP/1.1" 409 180

DEBUG: Response headers: {'Date': 'Tue, 05 Oct 2021 07:54:11 GMT', 'Content-Type': 'application/json', 'Content-Length': '180', 'Connection': 'keep-alive', 'x-amzn-RequestId': '07322f5c-ebe9-43f8-9f4d-da4154f44c4f', 'x-amzn-ErrorType': 'ResourceConflictException'}

DEBUG: Response body:

b'{"Type":"User","message":"The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:****

DEBUG: Response headers: {'Date': 'Tue, 05 Oct 2021 07:54:11 GMT', 'Content-Type': 'application/json', 'Content-Length': '180', 'Connection': 'keep-alive', 'x-amzn-RequestId': '07322f5c-ebe9-43f8-9f4d-da4154f44c4f', 'x-amzn-ErrorType': 'ResourceConflictException'}

DEBUG: Response body:

b'{"Type":"User","message":"The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:****

DEBUG: Event needs-retry.lambda.UpdateFunctionCode: calling handler <botocore.retryhandler.RetryHandler object at 0x7fa5110f3a60>

DEBUG: No retry needed.

✖ Failed to update Lambda function code. Error: An error occurred (ResourceConflictException) when calling the UpdateFunctionCode operation: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:xxxx

[Oleksandr Kyrdan]

Hi @sergio_alba

Thank you for your question!

We'll investigate the issue and notify you.

Best regards,
Oleksandr Kyrdan

[sergio_alba]

Hi Oleksandr,

Many thanks for your response! Any updates on this will be greatly appreciated since we heavily depend on the pipe for our AWS Lambda Deployments.

Best regards,

Sergio.

[Oleksandr Kyrdan]

Hi @sergio_alba ,

Good news we have released updated version of the aws-lambda-deploy pipe:

• boto3 bumped to the latest 1.18.* version

script:
  - pipe: atlassian/aws-lambda-deploy:1.2.0
    variables:
      AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
      AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
      AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
      FUNCTION_NAME: 'my-lambda-function'
      COMMAND: 'update'
      ZIP_FILE: 'my-function-code.zip'

It would be nice if you could provide us with your feedback about updated pipe.

Cheers,
Oleksandr Kyrdan

[sergio_alba]

Hi @Oleksandr Kyrdan,

Thanks a lot for the update! I have tested it on a test function and the pipe works.

Best regards and thanks for all the good work,

Sergio.

EDIT:

Hi again @Oleksandr Kyrdan ,

I tried running the pipeline on our dev ver. stack with no luck, although the pipe version is correct I keep getting the error message:

 Status: Downloaded newer image for bitbucketpipelines/aws-lambda-deploy:1.2.0
[...]
INFO: Updating Lambda function configuration.
✔ Lambda configuration update succeeded.
INFO: Updating Lambda function.
✖ Failed to update Lambda function code. Error: An error occurred (ResourceConflictException) when calling the UpdateFunctionCode operation: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:xxxxx:function:xxxx

Any ideas on why we keep getting this error?

Best regards,

Sergio.

[Alejandro Alfonso Fernández]

Hi!

Same problem in our recipe: the function updates right, but the logs returns the same error @sergio_alba reports

Best regards

[sergio_alba]

Hi!

Any updates on this issue?

Thanks,

[Igor Stoyanov]

@sergio_alba  Hi. Please provide configuration of your pipe. Seems like you use it with FUNCTION_CONFIGURATION and despite of the fact of this log message:

✔ Lambda configuration update succeeded.

the state of the lambda is not equal:

{
    "State": "Active",
    "LastUpdateStatus": "Successful"
}

 so UpdateFunctionCode operation fails.

Can you confirm that you use in your pipe configuration FUNCTION_CONFIGURATION parameter and check if all is ok with only update function code without updating function configuration ?

From our side we will try to reproduce and fix this error.

Regards, Igor. 

[sergio_alba]

@Igor Stoyanov Hi.

#1 Function logs (Ok):

[...]
Status: Downloaded newer image for bitbucketpipelines/aws-lambda-deploy:1.2.0

INFO: Using default authentication with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

INFO: Executing the aws-lambda-deploy pipe...

INFO: Found credentials in environment variables.

INFO: Updating Lambda function.

#2 Function logs (Failed):

[...]
Digest: sha256:5b74f29da9b7d58ea23a6cadea54d7a3959556e0e0dd0653b31b6dae6e4828df

Status: Downloaded newer image for bitbucketpipelines/aws-lambda-deploy:1.2.0

INFO: Using default authentication with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

INFO: Executing the aws-lambda-deploy pipe...

INFO: Found credentials in environment variables.

INFO: Updating Lambda function configuration.

✔ Lambda configuration update succeeded.

INFO: Updating Lambda function.

✖ Failed to update Lambda function code. Error: An error occurred (ResourceConflictException) when calling the UpdateFunctionCode operation: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:205746061435:function:check-user-license

I have removed the FUNCTION_CONFIGURATION parameter on the first function deployment and it indeed worked. I left this parameter on the second function of the pipeline, which did indeed fail.

Configuration is as follows (redacted zipping of function files and function names):

- step:

name: Deploy [function-name]

script:

- pipe: atlassian/aws-lambda-deploy:1.2.0

variables:

AWS_ACCESS_KEY_ID: ${AWSID}

AWS_SECRET_ACCESS_KEY: ${AWSKEY}

AWS_DEFAULT_REGION: 'us-east-1'

FUNCTION_NAME: 'function-name'

COMMAND: 'update'

ZIP_FILE: 'function-name/code.zip'

# FUNCTION_CONFIGURATION: 'function-name/dev_config.json'

# Delete dir, otherwise it fails

- rm /opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/aws-lambda-deploy-env

- step:

name: Deploy [function-name]

script:

- pipe: atlassian/aws-lambda-deploy:1.2.0

variables:

AWS_ACCESS_KEY_ID: ${AWSID}

AWS_SECRET_ACCESS_KEY: ${AWSKEY}

AWS_DEFAULT_REGION: 'us-east-1'

FUNCTION_NAME: 'check-user-license'

COMMAND: 'update'

ZIP_FILE: 'function-name/code.zip'

FUNCTION_CONFIGURATION: 'function-name/dev_config.json'

- rm /opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/aws-lambda-deploy-env

[Igor Stoyanov]

@sergio_alba Hi. 

Thanks for your response. As i see the problem is with lambda state between configuration and code update. We will provide a fix in near future.

Regards, Igor.

[Igor Stoyanov]

@sergio_alba hi. We released new version of aws-lambda-deploy pipe with implemented waiter. Now you could do lambda update configuration and update code with the custom WAIT_INTERVAL:

script:
  - pipe: atlassian/aws-lambda-deploy:1.3.0
    variables:
      AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
      AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
      AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
      FUNCTION_NAME: 'my-lambda-function'
      COMMAND: 'update'
      ZIP_FILE: 'my-function-code.zip'
      FUNCTION_CONFIGURATION: 'new_function_configuration.json'
      WAIT_INTERVAL: '15'

or with default WAIT_INTERVAL=5 sec if you don't want to provide your custom value.

Let us know if the new version of pipe fixes your problem.

Regards, Igor.