Skip to content

[v2] Add blackbox tests - #10591

Open
aemous wants to merge 5 commits into
v2from
blackbox-tests-foundation
Open

[v2] Add blackbox tests#10591
aemous wants to merge 5 commits into
v2from
blackbox-tests-foundation

Conversation

@aemous

@aemous aemous commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Notes:

  • Merging this PR is currently a no-op. They can be run ad-hoc by developers but are not invoked in any automated CI.

Description of changes:

  • Added scaffolding to easily add blackbox tests. The main helper function is assert_operation which lets callers verify if an HTTP request matches an operation. This lets test-writers easily verify if the expected operations were called in the expected order.
  • Added S3 blackbox tests. These are ports of our existing S3 functional tests (except cp, mv, or sync commands). The only tests not ported are those that cannot be feasibly ported to a blackbox case (e.g. relies on a Python monkey-patch of datetime).
  • Added new blackbox requirements test file, and updated regenerate-lock-files and GitHub workflows to account for this.

Description of tests:

  • Ran and passed all new blackbox tests via PyTest.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@aemous aemous added the s3 label Aug 20, 2026
@aemous
aemous requested a review from a team as a code owner August 20, 2026 17:41
@aemous aemous added v2 customization Issues related to CLI customizations (located in /awscli/customizations) labels Aug 20, 2026
@aemous
aemous force-pushed the blackbox-tests-foundation branch 2 times, most recently from 844b3ac to 7f16724 Compare August 21, 2026 13:43
@aemous
aemous requested a review from hssyoo August 21, 2026 13:43
Comment thread tests/blackbox/utils.py Outdated
Comment thread tests/blackbox/utils.py
Comment thread tests/blackbox/utils.py Outdated
@aemous
aemous force-pushed the blackbox-tests-foundation branch 3 times, most recently from e02a9a2 to 9d8b865 Compare August 24, 2026 13:23

@hssyoo hssyoo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, mainly had some structural feedback.

Comment thread tests/blackbox/conftest.py
Comment thread tests/blackbox/test_mb_command.py
Comment thread tests/blackbox/s3_assertions.py
@aemous
aemous force-pushed the blackbox-tests-foundation branch 2 times, most recently from 3dad06f to 9a9d3de Compare September 2, 2026 23:18

@hssyoo hssyoo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, can we take a test dependency on localstub if we're going to be merging to v2

@aemous
aemous requested a review from hssyoo September 3, 2026 18:45
@aemous
aemous force-pushed the blackbox-tests-foundation branch 2 times, most recently from 1e04c22 to ad67a96 Compare September 4, 2026 19:35
@aemous
aemous force-pushed the blackbox-tests-foundation branch from ad67a96 to d99c989 Compare September 9, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

customization Issues related to CLI customizations (located in /awscli/customizations) s3 v2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants