Resolve object-store URLs in MultiFileDataSource via the cloud registry

[balicat]

Summary

MultiFileDataSource::with_glob documents S3/GCS support through an
explicit FileSystemRef, but a bare URL with fs: None is treated as a
local path — cwd-joined and mangled — so object-store URLs only work for
callers that carry their own store wiring (the Python and Java bindings
each do). Anything on the C FFI — C#, C, Swift bindings — hits a
local-filesystem error on URL paths.

Reworked per review (originally this lived in the FFI): URLs are now
resolved in core, through the existing vortex-cloud Registry, so every
consumer gets the same behavior and the FFI change reduces to enabling a
feature.

Changes

• with_glob leaves object-store URLs untouched (the cwd-join for
  relative paths would corrupt them — s3://… counts as a relative path
  on Windows).
• build() routes each (url, None) source through a shared
  vortex_cloud::Registry — the same env-based resolution as the Python
  binding's resolve_store — wrapping the store in Compat (for the
  HTTP client's tokio requirement) and an ObjectStoreFileSystem.
• Parse failures, file:// URLs, and single-character schemes (Windows
  drive paths like C:/data.vortex) keep resolving locally, exactly as
  before.
• Gated behind a new vortex-file feature object_store_registry
  (bringing in vortex-cloud/registry), threaded through the existing
  facade feature of the same name, and enabled by vortex-ffi — so the
  FFI's own diff is one feature flag plus a test.
• With the feature off, behavior is byte-for-byte unchanged.

Tests at both levels (vortex-file and vortex-ffi) assert that an
unconfigured az:// URL fails inside the registry's store builder rather
than as an unmatched local glob — a deterministic, network-free probe
that the registry path was taken.

Validated end-to-end from a .NET binding over the C FFI against
Cloudflare R2 on the public internet: a single-series predicate scan of
an 18 MB / 4.5 M-row file completes in ~2 s cold via both https:// and
signed s3:// — one HEAD, one 64 KB footer read, and two parallel range
reads, byte-identical to the ranges the same plan issues against a local
file; the local-file path is regression-identical to a stock build.

[joseph-isaacs]

Should we do this at a lower level (can object store not do this?)

[balicat]

Agreed — that's the better home for it. Reworking to resolve URLs in core instead, in the fs: None path of MultiFileDataSource next to where the local filesystem is auto-created, so every consumer gets it and the FFI change all but disappears. Will update this PR.

[balicat]

Reworked as discussed — resolution now happens in core: with_glob keeps URLs intact and build() routes (url, None) sources through the existing vortex-cloud::Registry, gated behind a vortex-file/object_store_registry feature threaded through the facade. The FFI diff is down to a feature flag plus a test. Title and description updated to match.

[codspeed-hq]

Merging this PR will degrade performance by 5.56%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 4 regressed benchmarks
✅ 2190 untouched benchmarks
⏩ 218 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	arrow_checked_add_u32_neon[16384]	12.3 µs	20.3 µs	-39.64%
❌	WallTime	dict_canonicalize_gt_u8_avx2[16000000]	8.9 ms	10.6 ms	-15.94%
❌	WallTime	words_gather_scalar_avx2[65536]	8.2 µs	9.4 µs	-12.16%
❌	WallTime	mul_u32_nonnull_avx512	5.6 µs	6.2 µs	-10.02%
⚡	Simulation	decompress[u64, (4000, 1024)]	87.2 µs	71.8 µs	+21.52%
⚡	WallTime	arrow_checked_add_u32_avx2[16384]	21.4 µs	17.7 µs	+21.17%
⚡	Simulation	allocate_drop_arrow[0]	456.9 ns	402.7 ns	+13.45%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing balicat:ffi-object-store-urls (94cf0c5) with develop (8e2aa05)}

Footnotes

1. 218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[balicat]

CI fixes pushed:

• taplo fmt on the two manifests it flagged.
• Removed an intra-doc link to a function-locally imported type that broke the docs build.
• The publish dry-run failure was real: this PR's vortex-file → vortex-cloud dependency closes a cycle with vortex-cloud's dev-dependency on vortex-file, and publish ordering broke. Fixed by making that dev-dependency path-only (no version) so it drops out of the publish graph — same pattern as vortex-alp in the fastlanes encoding.

On CodSpeed: the one regressed benchmark (compress_fsst[(500, 64, 8)], −10.7%) is a code path this PR doesn't touch, and the same benchmark family moved −11.6% on #9614 and +10.9% on #9616 — looks like runner noise. Happy for someone with dashboard access to acknowledge it.

[github-actions]

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

[balicat]

Rebased onto current develop (c93f5a9) and re-ran the checks locally: vortex-file (with object_store_registry) and vortex-ffi test suites green, clippy with -D warnings on both crates, cargo fmt --check on the nightly the CI pins, taplo on the manifests, and the lockfile under --locked.

Could a maintainer approve the CI run and take a look when convenient? The rework follows @joseph-isaacs's suggestion to resolve URLs in core rather than in the FFI, so the FFI diff is now just the feature flag plus a test.

[myrrc]

Hi @balicat. We've taken the route for the integrations to use their own globbing. For example, vortex-duckdb and vortex-datafusion both use query engines' implementation for that and operate on single files only.

MultiFileDataSource is used only in FFI, JNI, and one benchmark integration, and we plan to phase it out eventually.

[balicat]

Thanks, that makes sense. I've moved the resolution out of core.

The whole change now lives in vortex-ffi: it parses each incoming path and, for an
object-store URL, resolves it through the shared vortex-cloud registry (the same
resolve the Python binding's resolve_store uses) and hands MultiFileDataSource
an explicit FileSystemRef. Local paths, file://, and Windows drive paths stay
ordinary local globs. So the FFI now owns its globbing the way vortex-duckdb and
vortex-datafusion own theirs, and if MultiFileDataSource is later phased out the
resolution moves with the FFI rather than being stranded in core.

vortex-file, vortex-cloud, and vortex are back to their develop state; the
core diff and the publish-graph feature threading are gone. The change is ~30 lines
plus a test in vortex-ffi.

[myrrc]

LGTM. Let's shorten the comments, make the tests pass, update the PR description and name, and I'll merge this.
Also, please disclose LLM usage, if any, in the PR description.

[myrrc]

Let's move the "why" out of comments.