Skip to content

Materialize Any and dict targets like the untyped parse; write Any containers statically - #486

Merged
quinnj merged 6 commits into
masterfrom
jq/geojson-followups
Sep 19, 2026
Merged

quinnj merged 6 commits into
masterfrom
jq/geojson-followups

Conversation

@quinnj

@quinnj quinnj commented Sep 17, 2026 •

Copy link
Copy Markdown
Member

Follow-ups from JuliaGeo/GeoJSON.jl#128. Typed Any slots and common dictionary/array targets reuse concrete callback materialization; large Object{String,Any} targets append in linear time for distinct keys. Any-container writing uses concrete branches plus the JSON.applyany extension point. This pairs with JuliaServices/StructUtils.jl#73 and also passes CI with released StructUtils 2.9.1.

Customization contracts are preserved:

  • Every key and index passes through lowerkey, including Vector{Any} and undefined slots. JSON accepts a string or real number from this hook. Its default numeric method returns the number unchanged; only object writing and sorting convert it to a string. Numeric object keys remain quoted and sort lexicographically. Existing AbstractString keys are not copied.
  • Every recognized write value still goes through lower, including nested containers and missing/null values. Concrete sorted dictionaries retain specialized writes instead of boxing each value.
  • Fast typed reads are limited to the default inner style. Custom styles retain make, lift, liftkey, and initialization; mismatched shapes retain the generic fallback.
  • A dedicated Dict-parent method breaks the recursive inference cycle that prevented nested Object/array/Dict writes from compiling under safe trimming. Both method bodies share one source.
  • Tiny objects defer the temporary key set until the fifth distinct key. Larger objects retain hashed duplicate detection and direct append.
  • Output-buffer growth at least doubles, and nested JSON Lines writes retain the caller's style and buffer size.

Validation at fc4a655eab9e8c95a27a074d3d052cb108f854ff, paired locally with StructUtils 4f4e220e91b474bcd3d2cdc6bb74f9cd13aa40c1:

  • All 12 CI checks pass: coverage, docs, minimum/current/prerelease Julia, 32-bit, and Parsers 2.8.8/3.
  • Full local suite: 2,341 checks pass on Julia 1.10.11, 1.12.6 and 1.13.0.
  • All 986 published GeoJSON core checks pass. Alternating array/object round-trips pass through 2,048 levels.
  • Safe-trim tests include nested Object/Dict/array writes, a closed style, small-buffer IO, and numeric-key sorting.
  • benchmarks/geojson_regressions.jl provides 111 runtime workloads plus cold, load, and precompile modes. The numeric-key follow-up adds no allocations across the 333 runtime comparisons on Julia 1.10/1.12/1.13 after matching cache state.

The numeric-key fix retains the large-array gain while restoring all index hooks. On local macOS ARM64 / Julia 1.12, writing one million Float64 values takes 36.9 ms and 31 allocations, versus 36.8 ms and 31 allocations before this follow-up; the package base took about 102 ms and 2,000,031 allocations. Three alternating before/after samples give about 51.7 / 51.3 ms for combined cached loading and 5.84 / 5.84 s for forced package rebuilds with dependencies cached.

Tradeoffs remain explicit:

  • In this follow-up, one nested Any first-write sample increased by about 1 ms. A focused four-key numeric dictionary sample on Julia 1.12 was about 5% slower with unchanged allocations. Initial nested-IO timing outliers did not repeat.
  • Julia 1.13 numeric-parser allocation counts depend on compiled cache state: both revisions use 130,240 bytes for 1,000 small structs without package caches and 178,240 after explicitly rebuilding both caches. This behavior is not specific to the numeric-key fix.
  • Versus the package bases, the broader review found higher bytes for a GeoJSON-loaded tiny-object probe, small load/precompile costs, and slower small/wide-union first writes and some warm wide-struct writes. The numeric-key follow-up does not erase those earlier tradeoffs.
  • The published GeoJSON revision already uses optimized traversal wrappers. Its plotting extensions and separate trim workload were not included in the downstream core check.

Timings are local measurements, not guarantees across machines. Safe trimming requires a closed fallback and statically resolvable custom lowering.

Co-authored by Codex

…ntainers statically

Reading:
- `StructUtils.make(style, Any, x)` and the `Vector{Any}`, `Object{String,Any}` and
  `AbstractDict{String,Any}` targets now go through a styled `applyvalue(f, x, style)`,
  the typed counterpart of the untyped materializer: each value reaches its sink with a
  concrete type instead of a boxed `(value::Any, pos)` pair. A 137-key object into
  `Object{String,Any}` drops from 105us/1433 allocations to 9us/428 (untyped: 425), and a
  3000-key nested object from 17ms to 0.17ms (the generic dict path filled `Object`
  through `setindex!`, a linear scan per key).

Writing:
- `Any`-valued dicts, vectors and pair vectors hand each of the types the untyped parse
  produces to the writer statically; anything else goes through the new
  `JSON.applyany(style, f, key, value)` hook, whose default lowers dynamically. A style
  that overrides it to throw has no dynamic call on its write path under
  `juliac --trim=safe`.
- The output buffer grows by at least doubling, so a large output no longer needs a
  `sizeguess` method to avoid repeated copies.
- Nested writes under `jsonlines=true` keep the caller's `style` and `bufsize`; a custom
  style was silently replaced by the default for every nested value.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.40000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.75%. Comparing base (9e50068) to head (1d1822b).

Files with missing lines Patch % Lines
src/parse.jl 98.75% 1 Missing ⚠️
src/write.jl 97.77% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #486      +/-   ##
==========================================
+ Coverage   91.30%   91.75%   +0.44%     
==========================================
  Files           7        7              
  Lines        1680     1794     +114     
==========================================
+ Hits         1534     1646     +112     
- Misses        146      148       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@quinnj
quinnj merged commit af6aa93 into master Sep 19, 2026
21 of 22 checks passed
@quinnj
quinnj deleted the jq/geojson-followups branch September 19, 2026 04:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant