ci: simplify integration test by maxday · Pull Request #216 · awslabs/aws-lambda-cpp

maxday · 2026-05-29T13:16:28Z

Summary

Replace the legacy CodeBuild-based CI with a GitHub Actions workflow that runs unit tests in containers
and integration tests against real Lambda functions deployed as OCI images.

Remove all ci/codebuild/ infrastructure, vendored gtest, and old integration test harness (~27k lines
deleted)
Add a new integration-tests.yml workflow with a matrix of 5 OS targets (al2023, al2023-arm, ubuntu,
alpine, arch)
Unit tests run in native containers; integration tests build Docker images, push to ECR, deploy as
Lambda functions, invoke them, and assert against snapshot files

Changes

.github/workflows/tests.yml new workflow: unit-test job (container matrix) +
integration-test job (build → ECR push → Lambda deploy → invoke → assert → cleanup)
ci/integ/ new helper scripts: install-deps.sh, unit-test.sh, invoke.sh, assert.sh, and 4
Dockerfiles (al2023, alpine, arch, ubuntu)
tests/ replace vendored gtest + integration tests with a snapshot-based approach using
lambda_function.cpp multi-handler binary
ci/codebuild/, ci/docker/, ci/update-images.sh, ci/README.md deleted (superseded by GHA)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

bmoffatt · 2026-05-29T17:01:30Z


    // Read the handler from the environment variable
-    const char* handler_name = std::getenv("_HANDLER");
+    const char* handler_name = std::getenv("HANDLER");


~~_HANDLER is correct though https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html#configuration-envvars-runtime~~

oh I see it's related to the .zip -> oci swap and movement of resource management from the tests code to the github actions code

bmoffatt · 2026-05-29T17:19:07Z

-        Model::FunctionCode funcode;
-        funcode.SetZipFile(std::move(zip_file_bytes));
-        create_function_request.SetCode(std::move(funcode));
-        create_function_request.SetRuntime(Aws::Lambda::Model::Runtime::provided_al2);


overall feedback: this integration test coverage isn't equivalent to what was being done in CodeBuild. Now the functions are uploaded as images, whereas before they were uploaded as .zip

In otherwords, this test workflow refactor loses coverage of the https://github.com/awslabs/aws-lambda-cpp/blob/master/packaging/packager script (see add_custom_target(aws-lambda-package-lambda-test-fun in test/resources/CMakeLists.txt) which was one of the things we spent a bunch of time figuring out for this project.

This is important coverage to keep, as this project has been usable from multiple operating systems from the start without requiring Docker

I see. Just to make sure we want to:

still build the RIC from different OS (in container)

package the resulting binary in zip

test on zip-deployed AWS Lambda with provided al2023 runtime
is that correct?

In any case, I would love to keep those OCI tests, as it increases the coverage and ensure that the RIC is compatible with different base OS when deployed as OCI image. WDYT?

oh yeah for sure keep the OCI versions too.

zip versions I think would be a similar matrix to what you have already for the unit-test, with an added step to ninja install aws-lambda-package-lambda-test-fun before doing similar function create + invoke steps

bmoffatt · 2026-05-29T17:34:25Z

 endif()


-find_package(AWSSDK COMPONENTS lambda iam QUIET)


torn on losing this.

Like it's nice that there's not an AWS SDK CPP install required any more, which took ages and needed to be hidden by caching in the build images. But now there's not a way to run the integration tests without triggering CI. https://github.com/awslabs/aws-lambda-cpp/blob/master/README.md#running-unit-tests-locally

I think it's worth keeping (but also changing in whatever way is needed to not explode CI runtimes if the caching is getting lost)

M-Elsaeed · 2026-07-01T10:32:38Z

Why are we using the aws in bash and not boto3 and python?

I would prefer not to ask people to install Python just to contribute to this cpp project + personal preference, to be honest: those files are for tests only, so we're not going to test the tests. If something fails, the test itself will fail. I thought it was a good trade-off. Happy to change if you have more concerns

I think it easier to read python than bash but both work, approved.

darklight3it · 2026-07-01T11:14:05Z

+            -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG \
+            .
+          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
+          echo "image_uri=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> "$GITHUB_ENV"


This is clear but I have a question. Would moving on the SAM cli help us defining this kind of infrastructure better?

This feels like a larger scope change. My inclination is to defer this to a follow-up PR if we feel it's a blocker.

darklight3it

LGTM 🚀

benrkia · 2026-07-01T14:47:40Z

+
+  integration-test:
+    runs-on: ${{ matrix.build.runner }}
+    container: ${{ matrix.deploy_type == 'zip' && matrix.build.container || '' }}


Clever condition but not that readable, i'd prefer having something more readable

I've split it into multiple jobs, no more conditions

benrkia · 2026-07-01T14:48:40Z

+    strategy:
+      fail-fast: false
+      matrix:
+        deploy_type: [oci, zip]


We have plenty of conditions based on the deploy_type this is a hint that splitting the jobs would be more readable and easy to maintain

I've split it into multiple jobs, no more conditions

benrkia · 2026-07-01T14:52:01Z

+      - name: Deploy Lambda function (zip)
+        if: matrix.deploy_type == 'zip'
+        env:
+          FUNCTION_NAME: ${{ env.FUNCTION_PREFIX }}-${{ matrix.deploy_type }}-${{ matrix.build.os }}-${{ matrix.test.name }}-${{ github.run_id }}


This re-defined in other places, we could have it defined in a single place and re-use it

unfortunately, GitHub Actions doesn't allow matrix.* in job-level env, I could hack into creating a step before and store it in GITHUB_ENV but I would prefer to keep it as is if that's fine with you

benrkia · 2026-07-01T14:54:15Z

+  $PAYLOAD_ARG \
+  /tmp/response_payload > /tmp/invoke_result.json
+
+echo "Status code: $(jq '.StatusCode' /tmp/invoke_result.json)"


I don't like that we're using a hard-coded path. 1. this might cause conflicting jobs to mutate the same file, 2. We tightly couple assert.sh with the path we have in here

I'd change this to use something specific to a single run and have a mechanism to get the generated path and then pass it to the assertion script, or easier just have assert and invoke in the same file

now using mktemp to prevent conflicts (which I don't think were possible since they're running on different hosts, but better to be safe, good catch)

benrkia · 2026-07-01T14:55:36Z

+  AWS_REGION: ${{ secrets.AWS_REGION }}
+  FUNCTION_PREFIX: lambda-cpp-integ
+
+jobs:


Let's define an explicit timeout to avoid waiting for 6h in case of an issue

benrkia · 2026-07-01T15:21:50Z

+  PAYLOAD_ARG=""
+fi
+
+aws lambda invoke \


What if this fails?

The script already has set -euo pipefail, so a failed aws lambda invoke will exit non-zero and fail the step

benrkia · 2026-07-01T15:25:10Z

+To run the full zip integration test (build, deploy, invoke, assert, cleanup) in one shot:
+
+```bash
+./ci/integ/run-zip-integ.sh [OS] [HANDLER] [PAYLOAD] [ASSERTION] [ARCH]


Have you tried running this on a mac just to catch any potential issues?

since mac is not supported on Lambda, I think it's safe to assume that development needs to happen on a unix based system

benrkia · 2026-07-01T15:26:07Z

+  --zip-file fileb://build/tests/resources/lambda-test-fun.zip \
+  --environment "Variables={HANDLER=$HANDLER}"
+
+aws lambda wait function-active-v2 --function-name "$FUNCTION_NAME"


Do we need an explicit timeout here?

benrkia · 2026-07-01T15:32:16Z

+WORKDIR /src
+COPY . .
+
+RUN mkdir build && cd build && \


Why we don't use ENABLE_SANITIZERS here?

benrkia · 2026-07-01T15:38:52Z

-      - echo Build started on `date`
-      - ./ci/codebuild/build.sh -DTEST_RESOURCE_PREFIX=lambda-cpp-al2_$(arch)
-      - ./ci/codebuild/run-tests.sh aws-lambda-package-lambda-test-fun
-      - ./ci/codebuild/run-tests.sh aws-lambda-package-lambda-test-fun-no-glibc


Is this intentionally dropped? i see no backfill in the current approach

they're now back, good catch

bmoffatt · 2026-07-01T18:34:57Z

+| `AWS_REGION` | AWS region (e.g. `us-east-1`) |
+| `LAMBDA_EXECUTION_ROLE_ARN` | ARN of the Lambda execution role |
+
+## Running Zip Integration Tests Locally


bmoffatt · 2026-07-01T18:35:07Z

+```bash
+export AWS_REGION="us-east-1"
+export LAMBDA_EXECUTION_ROLE_ARN="arn:aws:iam::123456789012:role/your-lambda-role"
+


uber nit, it'd be nice if the test.yml used this entry was well, to help ensure it stays working for local runs.

not necessarily need to DRY it, an extra workflow that validated this script worked would suffice too

bmoffatt · 2026-07-01T18:37:37Z

+fi
+
+rm -rf build && mkdir build && cd build
+cmake .. -GNinja \


maybe call out in the integ README that local test script needs ninja, aws cli installed

bmoffatt · 2026-07-01T18:42:07Z

+
+      - name: Build and run unit tests
+        shell: bash
+        run: ./ci/integ/unit-test.sh ${{ matrix.os }}


~/aws-lambda-cpp $ ./ci/integ/unit-test.sh al2023 -- Unit tests skipped: Not in GitHub Actions environment -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /home/moffattb/aws-lambda-cpp/build [ 62%] Built target aws-lambda-runtime [100%] Built target lambda-test-fun Test project /home/moffattb/aws-lambda-cpp/build No tests were found!!!

can this be resolved in this PR too?

On AL2023, ldd reports the loader as /lib64/ld-linux-x86-64.so.2 but rpm lists it under /usr/lib64/. The string mismatch causes hasElement to miss it, leaving PKG_LD unset. In no-glibc mode (-d), the fallback loop that rescues PKG_LD is skipped entirely since INCLUDE_LIBC=false. Add fallback loader detection in the main ldd loop for libraries that aren't matched in libc_libs but whose basename starts with ld-*. Also only require PKG_LD when INCLUDE_LIBC=true, since -d mode uses the binary directly as the bootstrap and never references the loader.

The worst-case function name (zip/no-glibc/al2023-arm/crash_backtrace) was 68 characters, exceeding Lambda's 64-character limit. Shorten the prefix from "lambda-cpp-integ" to "cpp-integ" (worst case now 61).

…r limit" This reverts commit 594724e.

The worst-case zip function name (no-glibc/al2023-arm/crash_backtrace) was 68 characters, exceeding Lambda's 64-character limit. Drop the redundant "zip-" segment and use a short packaging id ("noglibc" instead of "no-glibc") to bring the worst case down to 63 characters.

maxday added 16 commits May 28, 2026 14:10

feat: simplify intest test

aac1bd1

fix: region

899b40f

fix: multi layer docker

6d0b71c

fix: create ECR / clean ECR

1ee87b1

fix: handler

1b079fd

fix: ECR permissions

f249bec

fix: payload

9ab46c2

fix: payload

48cd4e3

fix: snapshot

7dd8edd

fix: test

6e9779c

fix: timestamp

a87e974

fix: bash -> sh

825921c

fix: deps

c1b1c34

feat: cleanup tests

b4bf26a

fix: cleanup

b344b14

fix: rename

4605370

bmoffatt reviewed May 29, 2026

View reviewed changes

maxday added 7 commits July 1, 2026 09:56

fix: pr comments

4008aa2

fix: add aws cli

582f03b

fix: unzip + curl

5584916

fix: unzip + curl

cbe6a64

fix: jq

0dd0210

fix: pin GitHub actions

c1bb6fb

fix: remove intake

edaa344

maxday requested a review from bmoffatt July 1, 2026 10:30

M-Elsaeed reviewed Jul 1, 2026

View reviewed changes

M-Elsaeed approved these changes Jul 1, 2026

View reviewed changes

darklight3it reviewed Jul 1, 2026

View reviewed changes

darklight3it approved these changes Jul 1, 2026

View reviewed changes

benrkia reviewed Jul 1, 2026

View reviewed changes

bmoffatt approved these changes Jul 1, 2026

View reviewed changes

maxday added 5 commits July 2, 2026 08:52

fix: pr review

4178bea

fix: shorten Lambda function name prefix to stay under 64-char limit

594724e

The worst-case function name (zip/no-glibc/al2023-arm/crash_backtrace) was 68 characters, exceeding Lambda's 64-character limit. Shorten the prefix from "lambda-cpp-integ" to "cpp-integ" (worst case now 61).

Revert "fix: shorten Lambda function name prefix to stay under 64-cha…

2c26ce8

…r limit" This reverts commit 594724e.

benrkia approved these changes Jul 2, 2026

View reviewed changes

maxday merged commit 1460d99 into master Jul 2, 2026
66 checks passed

Uh oh!

Conversation

maxday commented May 29, 2026

Summary

Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darklight3it left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment