Container 'Build' exceeded memory limit. Node js

[Murali]

Facing issues exceeding memory limit even set size is 2x.

image: nikolaik/python-nodejs:python3.10-nodejs14
  pipelines:
    branches:
      master:
        - step:
           size: 2x # Double resources available for this step.
           name: Deploy to production
           deployment: production
           caches:
              - node
           script: # Modify the commands below to build your repository.
              - pip install awscli --upgrade --ignore-installed six
              - export NODE_OPTIONS=--max-old-space-size=8192
              - yarn install
              - CI=false yarn run build
              # - yarn run test 

Note: Its work fine when I deleted existing build logs

[Theodora Boudale · Accepted solution]

Hi @Murali,

Steps with size: 2x have 8192 MB memory in total. However, the build container is given 1024 MB of the total memory, which covers your build process and some Pipelines overheads (agent container, logging, etc). This means that the remaining memory for 2x steps is 7128 MB.

I see in your yml file the command

- export NODE_OPTIONS=--max-old-space-size=8192

I'm not sure if that command is allocating memory, but if it does, please keep in mind that the remaining memory for the step is 7128 MB and not 8192 MB, so I would suggest decreasing that value.

Kind regards,
Theodora

[Murali]

Hi @Theodora Boudale ,

I tried 4096  but still facing the same issue.

 - export NODE_OPTIONS=--max-old-space-size=4096

Maybe our app needs more than 2x. Can you check this?

[Theodora Boudale]

Hi @Murali,

We don't have visibility on memory consumption in Pipelines builds.

If you want to debug this in Pipelines, you can use the following commands at the beginning of the script (in the step that is failing):

- while true; do date && ps -aux && sleep 5 && echo ""; done &
- while true; do date && echo "Memory usage in megabytes:" && echo $((`cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes | awk '{print $1}'`/1048576)) && echo "" && sleep 5; done &

When you run a build with these commands, you can then see memory usage in the Pipelines build log.

If you won't want to consume Pipelines minutes in order to debug this, you can debug the build locally with Docker as per the steps we mention here:

• https://support.atlassian.com/bitbucket-cloud/docs/debug-pipelines-locally-with-docker/

In this case, you would need to use docker stats to evaluate the memory usage in the container. Once you have the setup ready and the build is running, run (locally) the Docker Stats command to print the output to a file in a loop. You would have to run this in a separate window in parallel.

while true; do date >> mem.txt && docker stats -a --no-stream >> mem.txt && sleep 2; done &

As I mentioned, steps with size: 2x have 8192 MB. 1024 MB of that goes to the build container, and the remaining memory is 7128 MB. If you cannot configure your build to use less memory than that, then it will not be possible to run this build in Pipelines, as 8192 MB is the maximum memory that can be configured for a step.

We have a feature request for the ability to increase memory more than that, I would suggest adding your vote and comment there to express your interest: https://jira.atlassian.com/browse/BCLOUD-17260

In the meantime, one option for steps that require more than 8192 MB of memory would be to use a runner that you can configure in one of your own servers. With runners, you can configure up to 32GB (8x) of memory:

• https://support.atlassian.com/bitbucket-cloud/docs/runners/

Kind regards,
Theodora

[Murali]

@Theodora Boudale  Thank you for sharing the debug memory code. Please check the below code

 Wed Nov 16 08:18:02 UTC 2022
 Memory usage in megabytes: 8192

I think we need to configure the runners or will need to reduce our app build timings(Package update or move to Vitejs for build)

Can you share your suggestion?

[Theodora Boudale]

Hi @Murali,

You can try debugging this locally as well and see if you can change some configurations so that the build uses less memory.

If this is not possible, then you can look into using runners for this specific step.

Kind regards,
Theodora