GH-51329: [C++][CI] Test static linking with S3 and fix Azure/GCS dependencies

[tadeja]

Rationale for this change

Fix #51329
Additional downstream static-linking test, after #50195 moved S3 into the separate ArrowS3 library.

What changes are included in this PR?

Test downstream static linking of ArrowS3::arrow_s3_static via minimal_build example and cpp_test.sh,
Fix dependencies for static Azure and GCS consumers: append libcurl for bundled GCS, append libcurl and LibXml2 for bundled Azure(NOT WIN32),
append links for system Azure SDK
and pkg-config --static arrow to include libcurl and libxml-2.0 for bundled GCS/Azure

Are these changes tested?

Verified by CI

Are there any user-facing changes?

Downstream consumers now have required static dependencies for Azure/GCS

• GitHub Issue: [C++] arrow_static's exported CMake interface omits LibXml2, so static links fail on the bundled Azure SDK #51329

[raulcd]

@tadeja do you mean adding this as part of #50195
or is this just for testing and follow up on a new issue PR to increase coverage?
I am ok testing it here and potentially adding improvements for testing the different packages (as also suggested Azure).

[raulcd]

@github-actions crossbow submit test-conda-cpp

[github-actions]

Revision: b6b76f8

Submitted crossbow builds: ursacomputing/crossbow @ actions-c1f06c30ce

Task	Status
test-conda-cpp

[tadeja]

@tadeja do you mean adding this as part of #50195 or is this just for testing and follow up on a new issue PR to increase coverage?

I don't mind, whichever is neater and more convenient to implement?
Also, I missed adding the change to cpp_test.sh 🤦 Adding it now.

[tadeja]

@github-actions crossbow submit test-conda-cpp

[github-actions]

Revision: e28fd84

Submitted crossbow builds: ursacomputing/crossbow @ actions-4282328564

Task	Status
test-conda-cpp

[tadeja]

yay, new static linking test shows further failures for bundled Azure and google-cloud-cpp:
job ARM64 macos 14 C++

[ 50%] Built target arrow-example
[ 75%] Building CXX object CMakeFiles/arrow-example-s3-static.dir/example.cc.o
[100%] Linking CXX executable arrow-example-s3-static
Undefined symbols for architecture arm64:
  "_curl_easy_cleanup", referenced from:
      Azure::Core::_internal::UniqueHandleDeleter<void, curl_easy_cleanup>::operator()(void*) in libarrow_bundled_dependencies.a[2](curl.cpp.o)
      google::cloud::rest_internal::v3_2::CurlPtrCleanup::operator()(void*) const in libarrow_bundled_dependencies.a[214](curl_handle.cc.o)

job ARM64 Ubuntu 22.04 C++

[ 75%] Building CXX object CMakeFiles/arrow-example-s3-static.dir/example.cc.o
[100%] Linking CXX executable arrow-example-s3-static
/usr/bin/ld: /usr/local/lib/libarrow_bundled_dependencies.a(xml_wrapper.cpp.o): in function `Azure::Storage::_internal::XmlReader::XmlReader(char const*, unsigned long)':
/build/cpp/_deps/azure_sdk-src/sdk/storage/azure-storage-common/src/xml_wrapper.cpp:416: undefined reference to `xmlReaderForMemory'
...

[raulcd]

In my opinion it is better if we tackle this separately on its own issue.

[tadeja]

@github-actions crossbow submit test-conda-cpp

[raulcd]

this won't do anything here, right? Doesn't this has to be executed on top of my branch?

[tadeja]

Yes, this is to be able to potentially merge it either before or after #50195. Or within #50195 :)

[github-actions]

Revision: 4134d2d

Submitted crossbow builds: ursacomputing/crossbow @ actions-e76cd70dbd

Task	Status
test-conda-cpp

[tadeja]

I've just temporarily added here workaround of #50949 (cherry-pick 76eea10) to get test-conda-cpp succeed (still without the fix of #50195).

[tadeja]

Now with temporarily added workaround of #50949 (cherry-pick 76eea10) to get rid of
137 - arrow-filesystem-test (Failed)
143 - arrow-s3fs-module-test (Failed)
we get job AMD64 Conda C++ AVX2 to succeed
and also Docker Test conda-cpp to succeed
and both show the new static linking test is OK for now (before testing with current changes of #50195)

[100%] Linking CXX executable arrow-example-s3-static
[100%] Built target arrow-example-s3-static
/arrow/cpp/examples/minimal_build /build/cpp /
+ pushd /arrow/cpp/examples/minimal_build
+ PATH=/opt/conda/envs/arrow/bin:...
+ /build/cpp/examples/minimal_build/arrow-example
...
* Verify S3 initializes and finalizes
...

[timosachsenberg]

In my opinion it is better if we tackle this separately on its own issue.

Filed as #51329 — it reproduces the libxml2 half against a released artifact (libarrow-dev 24.0.0-1 from the APT repo, where libarrow.so records NEEDED libxml2.so.2 but Arrow::arrow_static declares nothing), and it ends in a dlopen-time undefined symbol: xmlBufferFree for a downstream consumer rather than a link error, which is why it went unnoticed. Happy for it to be closed as a duplicate if you'd rather fold this into #50195.

Two observations from writing it up, in case they're useful for wherever the fix lands:

• The pkg-config side needs its own line. The diff here adds LibXml2 to ARROW_SYSTEM_DEPENDENCIES and LibXml2::LibXml2 to ARROW_STATIC_INSTALL_INTERFACE_LIBS, but ARROW_PC_REQUIRES_PRIVATE is only ever populated via resolve_dependency(... PC_PACKAGE_NAMES ...), so a bare find_package(LibXml2 REQUIRED) won't reach arrow.pc. The bundled-AWS branch already does this explicitly at ThirdpartyToolchain.cmake:4262 — string(APPEND ARROW_PC_REQUIRES_PRIVATE " libcurl") — and the analogous line for Azure would be libxml-2.0 (plus libcurl).

• There's existing precedent for the whole shape of this. cpp/src/arrow/CMakeLists.txt populates ARROW_STATIC_INSTALL_INTERFACE_LIBS from fifteen SYSTEM branches and exactly one BUNDLED branch — AWSSDK_SOURCE STREQUAL "BUNDLED" → CURL::libcurl. So "a vendored dependency's own external requirements still have to be declared" is already an established pattern; Azure just never got it, which is also why the macOS _curl_easy_cleanup failure above is the same bug rather than a separate one.

Worth noting this is a different axis from #50753: that one is about SYSTEM declarations being easy to forget because they sit apart from resolve_dependency, whereas libxml2 isn't a dependency Arrow resolves at all, so co-locating wouldn't have surfaced it.

[tadeja]

@github-actions crossbow submit test-ubuntu-22.04-cpp-no-threading

[Copilot]

Copilot review overview

🔵 Needs a closer look

Affected Debian/Ubuntu builder images must install the newly declared libxml2 build dependency.

Review effort: Lite
Findings: 1

Open (1)

• Add libxml2-dev to Debian/Ubuntu package-builder images

[github-actions]

Revision: 9af899f

Submitted crossbow builds: ursacomputing/crossbow @ actions-bda051f4d8

Task	Status
test-ubuntu-22.04-cpp-no-threading

[tadeja]

@github-actions crossbow submit test-ubuntu-22.04-cpp-no-threading

[github-actions]

Revision: bc38a7d

Submitted crossbow builds: ursacomputing/crossbow @ actions-bea3766b73

Task	Status
test-ubuntu-22.04-cpp-no-threading

[Copilot]

Copilot review overview

🔵 Needs a closer look

Debian builder images do not install the newly required libxml2-dev package.

Review effort: Lite
Findings: 1

Open (1)

• Add libxml2-dev to Debian/Ubuntu package-builder images

[Copilot]

Copilot review overview

🟢 Approval recommended

No unresolved issues block approval.

Review effort: Lite
Findings: None

Resolved since last review (1)

• Add libxml2-dev to Debian/Ubuntu package-builder images

[Copilot]

Copilot review overview

🟢 Approval recommended

No unresolved issues blocking approval were identified.

Review effort: Lite
Findings: 1

Open (1)

• Add libxml2-dev to Debian/Ubuntu package-builder images

[Copilot]

Copilot review overview

🟢 Approval recommended

No unresolved blocking issues were identified, and all assessments support approval.

Review effort: Lite
Findings: 1

Open (1)

• Add libxml2-dev to Debian/Ubuntu package-builder images