feat(cc): introduce py_extension rule for C/C++ Python extension modules - #3973
Merged
Conversation
...and sycophantic toasters are bad collaborators
…into py-extension
This builds on the existing `py-extension` branch. It adds support for platform and abi tags in the resulting library filename. The `:py_extension_test`, `:py_extension_analysis_tests`, and `:py_limited_api_tests` test targets in `//tests/cc/py_extension` now build and pass. Relates to #3283
…tension (#3875) Per feedback from #3851, this PR re-works the platform detection logic to rely on platform constraints instead of the Python runtime/toolchain. ### Description This PR contains updates to the experimental `py_extension` implementation. It re-works the platform tag detection to rely on modern platform constraints, and refactors the compilation and linking mechanism to delegate to Bazel's native `cc_shared_library` and `cc_library` rules. This PR targets the main repository's `py-extension` branch (not `main`). ### Key Changes #### 1. Platform Tag & ABI Tag Derivation * **Introduced `abi_tag` to `PyCcToolchainInfo`**: Added the `abi_tag` field to the `PyCcToolchainInfo` provider, populated by the `py_cc_toolchain` rule. It defaults to deriving the tag from `python_version` (e.g. `cpython-311`) for backward compatibility. * **Constraint-Based Detection**: Replaced legacy parsing of `cc_toolchain` CPU names with direct lookup in rules_python's central `PLATFORMS` registry using platform constraints. * **Limited API Configuration**: Updated how limited API (`.abi3.so`) suffixes are appended based on the toolchain configuration. #### 2. Compilation & Linking Delegation (Refactor to `cc_shared_library`) * **Macro Fusion**: Replaced the custom linking logic in the `py_extension` rule with a macro of the same name. It now accepts C/C++ source/header files directly (`srcs`, `hdrs`, `copts`, `defines`), implicitly wrapping them in a private `cc_library` under the hood. * **Consolidated Attributes**: Consolidated static linkage under standard `deps` (removing the redundant `static_deps` attribute) and aligned linker arguments with `cc_shared_library`'s `user_link_flags`. * **Wrapper Rule (`_py_extension_wrapper`)**: Added a lightweight, private rule that wraps the `cc_shared_library` output to: * Query the Python toolchain for platform/ABI tags to resolve PEP 3149 compliant filenames (e.g. `.cpython-311-x86_64-linux-gnu.so` or `.abi3.so`). * Create a cheap `symlink` from the CSL output to the PEP 3149 name. * Propagate `PyInfo` (for python rules) and `CcSharedLibraryInfo` (for dynamic C++ dependencies). * **ODR Validation**: Leveraging native `cc_shared_library` ensures strict analysis-time validation against One Definition Rule (ODR) violations (e.g., duplicate static linkage in dynamic chains). #### 3. Runfiles and Data Support * **Runfiles Propagation**: Re-implemented runfiles collection using the repository's standard `builders.RunfilesBuilder` to support runtime assets (`data` attribute) and dynamic library dependency propagation. ### Verification & Testing * Added parity test cases in `dependency_graph_tests.bzl` to verify dynamic dependency chains and static sharing behavior. * Added runfiles validation in `py_extension_tests.bzl` to verify data asset propagation. * Updated limited API tests to match CSL output structure. * All 14 tests in `//tests/cc/py_extension/...` are compiling and passing. ────── TAG=agy CONV=23f8a5e8-2d99-401a-9903-1256b7d42b0e --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Why: Merge the latest upstream changes into py-extension to keep the branch up to date and compatible with Starlarkified rules_cc in Bazel 9. How: - Resolve conflict in dev/pyproject.toml and remove stale docs/requirements.txt. - Explicitly load CcSharedLibraryInfo from rules_cc. - Use cc_common.link without main_output to avoid private API restrictions, naming the intermediate library with a leading underscore and .pyextimpl suffix. - Add missing bzl_library targets and fix header includes in tests.
Why: Integrate upstream/py-extension (#3875) platform detection logic and cc_shared_library macro wrapper architecture into our local branch. How: - Resolve conflict in py_extension_rule.bzl in favor of upstream wrapper implementation. - Preserve runfiles dynamic lookup test in py_extension_test.py. - Fix function docstring header formatting in py_cc_toolchain_rule.bzl.
…e type annotations
Why:
Mark py_extension as a public but volatile API in documentation and improve docstring parameter type readability.
How:
- Add volatile API inclusion block to py_extension macro docstring.
- Cross-reference py_library.imports using {attr}.
- Add {type} annotations for all parameters and remove redundant Optional wording.
…3971) Per feedback from #3875 , this PR makes a few changes: * Removed an unnecessary hasattr check on py_cc_toolchain * Renamed the arbitrary "os" and "cpu" identifiers to terms standardized in PEP 508. Also marks `py_extension` as experimental. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Richard Levasseur <richardlev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Why: The py_cc_toolchain_rule.bzl loads flags.bzl, but :flags was missing from the deps attribute of //python/private:py_cc_toolchain_rule, breaking starlark doc extraction in docs build test. How: Add :flags to deps of //python/private:py_cc_toolchain_rule in python/private/BUILD.bazel.
…brary Why: py_cc_toolchain_macro.bzl loads //python/private/pypi:pep508_env.bzl, but the dependency was missing from the py_cc_toolchain_macro target in //python/private:BUILD.bazel. How: Add //python/private/pypi:pep508_env to deps of py_cc_toolchain_macro in python/private/BUILD.bazel.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
rickeylev
marked this pull request as draft
July 29, 2026 03:48
…D file Why: CI failed because tests/cc/py_extension/BUILD.bazel failed Buildifier checks and python/private/cc/BUILD.bazel referenced invalid bzl_library target names from @rules_cc. How: - Run buildifier to format tests/cc/py_extension/BUILD.bazel. - Update py_extension_macro and py_extension_rule bzl_library deps in python/private/cc/BUILD.bazel to use correct target names.
Why: WORKSPACE mode builds failed on Buildkite because bzlmod-only target names were used for rules_cc bzl_library dependencies. How: Use @rules_cc//cc:core_rules and @rules_cc//cc/common in python/private/cc/BUILD.bazel, which are supported in both WORKSPACE mode and bzlmod mode.
Why: Linking C extension shared libraries (.dylib) on macOS failed with undefined Python C symbols because ld requires -undefined dynamic_lookup when symbols are resolved at runtime. How: Add default user_link_flags with -undefined dynamic_lookup for @platforms//os:osx and @platforms//os:macos in py_extension_macro.bzl.
Why: GitHub actions CI failed on the ruff format check due to an extra blank line in py_extension_pkg_test.py. How: Reformat tests/cc/py_extension/py_extension_pkg_test.py using ruff format.
Collaborator
Author
|
cc @rsartor-cmd : something that you might run into integrating on the protobuf side: 65886db tldr, some flags necessary for building on mac are missing (that commit adds them) |
Why: py_extension_bzl is superseded by the py_extension bzl_library target in //python/cc:BUILD.bazel. How: Remove the py_extension_bzl bzl_library target definition from python/cc/BUILD.bazel.
Why: pyelftools is ELF-specific and fails when parsing Mach-O (.dylib on macOS) or PE (.dll on Windows) dynamic libraries. How: Decorate test_inspect_elf with @unittest.skipIf(sys.platform != 'linux', ...) in tests/cc/py_extension/py_extension_test.py.
…ames Why: Analysis tests in py_extension_tests.bzl and py_cc_toolchain_tests.bzl failed on macOS because extension file basenames and toolchain target platform tags vary across operating systems. How: Set explicit platform_machine and sys_platform on fake_py_cc_toolchain_impl in tests/support/cc_toolchains/BUILD.bazel, and use file_path_matches pattern matching for extension filenames in tests/cc/py_extension/py_extension_tests.bzl.
Document LNK1104, Bzlmod exports_filter, PyMODINIT_FUNC dllexport, LNK1107, and current_py_cc_libs propagation solutions.
…ion_macro Why: - Avoid duplicate label errors in cc_shared_library deps when srcs/hdrs are omitted. - Ensure exports_filter uses canonical Bzlmod Label strings so current_py_cc_libs is matched and retained by link.exe on Windows. How: - Wrap alias target labels with str(Label(...)) for canonical string expansion. - Only append py_cc_libs_win_alias to csl_deps_with_win when srcs or hdrs are provided.
…dows
Why:
- Bazel dereferences alias rules during cc_shared_library dependency processing.
- exports_filter matching failed because it contained the alias target label rather than the dereferenced actual target label (//python/cc:current_py_cc_libs), causing current_py_cc_libs to be pruned and link.exe to fail with LNK1104.
How:
- Add str(Label("//python/cc:current_py_cc_libs")) to win_exports_filter in py_extension_macro.bzl.
… in exports_filter
Define _PY_CC_HEADERS_ALIAS, _PY_CC_LIBS_ALIAS, and _PY_CC_LIBS_TARGET as uppercase module constants.
Evaluating str(Label(...)) at top-level module load time bakes in @@//... instead of caller's canonical Bzlmod repository context (e.g. @@rules_python+...). Evaluating Label inside def py_extension(...) ensures exports_filter matches cc_shared_library canonical target labels on Windows.
Why:
- Matching exact target labels in exports_filter across Bzlmod repository canonical names can fail if the dereferenced target label format differs from the macro stringification.
How:
- Use str(Label("//python/cc:__subpackages__")) and str(Label("//python/private/cc:__subpackages__")) in win_exports_filter.
Why: - Adding restrictive package label strings to exports_filter prevented matching external toolchain targets (e.g. @@+python+python_3_11_15...//:python_lib). How: - Pass csl_deps_with_win directly to exports_filter when exports_filter is None.
…ilter Why: - Passing str(Label(...)) pre-evaluated the label string as @@//... instead of letting rules_cc resolve string literals inside the target caller's Bzlmod context (@@rules_python+...). How: - Use plain string literals "//python/cc:current_py_cc_libs" and "//python/private/cc:current_py_cc_libs_private_alias" in exports_filter.
Why:
- CPython import libraries (python3xx.lib) are imported with system_provided = True in cc_import, which causes Bazel's CcInfo provider to suppress propagating the .lib path to link.exe.
How:
- Pass in user_link_flags and Label("//python/cc:current_py_cc_libs") in additional_linker_inputs on Windows.
Why: - Clean up macro attribute normalization, use canonical platform labels from common_labels.bzl, and organize Windows-specific linking requirements in a contiguous block. How: - Add PLATFORMS_OS_MACOS to common_labels.bzl. - Define top-level target constants (_BASE_TARGET and _CANONICAL_LABEL). - Normalize list attributes at start of py_extension. - Group Windows platform select statements before cc_shared_library.
Why: - Adding PLATFORMS_OS_MACOS to common_labels.bzl updated transitive module extension hashes loaded by uv.bzl in bzlmod_lockfile test workspace. How: - Ran bazel mod deps --lockfile_mode=update in tests/integration/bzlmod_lockfile.
…o.bzl Why: - final_csl_deps can be a select() expression, which cannot be iterated over via list comprehension at macro expansion time. - Using target strings in deps ensures final_csl_deps elements are strings, allowing direct assignment to exports_filter. How: - Use _PY_CC_HEADERS_ALIAS_BASE_TARGET and _PY_CC_LIBS_ALIAS_BASE_TARGET strings in deps. - Set csl_kwargs["exports_filter"] = final_csl_deps directly.
…cro.bzl Why: - Ensure dependencies use rules-python resolved canonical target strings while maintaining string type compatibility for exports_filter in cc_shared_library. How: - Define _CANONICAL_TARGET constants using str(Label(...)) at top-level. - Append _CANONICAL_TARGET constants to deps.
Why:
- Simplify csl_deps fallbacks by reassigning the list directly and removing final_csl_deps.
- Remove redundant exports_filter default assignment since cc_shared_library exports deps by default.
- Inline _EMPTY_BASE_TARGET into _EMPTY_CANONICAL_TARGET.
How:
- Define _EMPTY_CANONICAL_TARGET = str(Label("//python/private/cc:empty")).
- Fallback csl_deps to deps and then _EMPTY_CANONICAL_TARGET if empty.
- Only set exports_filter when non-None.
Why: - Document the addition of the experimental py_extension macro for release changelogs. How: - Create news/3973.added.md noting the addition of experimental py_extension.
…utes Why: - Expose new public //python/cc targets in python/features.bzl for feature detection. - Document new py_cc_toolchain attributes in news entry. How: - Add //python/cc:current_py_cc_headers, //python/cc:current_py_cc_libs, and //python/cc:py_extension to _TARGETS in python/features.bzl. - Update news/3973.added.md with py_cc_toolchain attribute documentation.
Why:
- Ensure news entry symbols use Sphinx MyST {obj} cross-references.
- Clean up features.bzl target entries.
How:
- Add {obj} xrefs to py_extension, py_cc_toolchain, and PyCcToolchainInfo in news/3973.added.md.
- Remove current_py_cc_headers and current_py_cc_libs from python/features.bzl.
Why: - Remove temporary plan file from tracked repository sources. How: - Git rm .agents/plans/windows_build_issues.md.
rickeylev
marked this pull request as ready for review
August 1, 2026 09:01
rickeylev
enabled auto-merge
August 1, 2026 09:13
Collaborator
Author
|
OK! Finally ready. |
aignas
approved these changes
Aug 1, 2026
| # - https://github.com/python/cpython/issues/67169 | ||
| # - https://github.com/python/cpython/commit/03a144bb6ac3d7631a3bdb895e2a1f2d021fb08b | ||
| version_parts = ctx.attr.python_version.split(".") | ||
| prefix = "cp" if ctx.attr.sys_platform == "win32" else "cpython-" |
Collaborator
There was a problem hiding this comment.
Should we have the implementation as well? cp and cpython seem to be well known names.
|
|
||
| libc = ctx.attr.libc or LibcFlag.get_value(ctx) | ||
|
|
||
| platform_tag = _get_platform_tag( |
Collaborator
There was a problem hiding this comment.
Just a reminder that we have the EnvMarkerInfo provider, that might be useful.
| @@ -0,0 +1,140 @@ | |||
| # Copyright 2025 The Bazel Authors. All rights reserved. | |||
Collaborator
There was a problem hiding this comment.
I would vote to remove these headers everywhere, because they are unmaintainable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Building C/C++ Python extension modules currently requires manually
configuring complex
cc_binaryorcc_shared_librarytargets andsetting up platform, ABI3, and extension tags. The new
py_extensionrule provides a high-level Starlark API for compilingnative extensions that integrate seamlessly with
rules_pythontoolchains and
py_librarydependencies.At a high level, the implementation is fairly simple:
Users can provide raw C/C++
srcs, which are put into acc_library.That
cc_libraryis then given tocc_shared_library, which performsthe linking to create e.g.
libfoo.so. That artifact is then givento a custom rule that handles rename it according to Python needs,
e.g.
foo.abi3.so.cc_shared_libraryis used because it provides various facilitiesto control creation of a shared library, namely, control over what
is and isn't linked into the shared library.
Credit to @rsartor-cmd for finishing the prototype, expanding test
coverage, and verifying behavior.
Closes #3283.