Skip to content

fix: std.base64 rejects string codepoints outside byte range - #1108

Closed
He-Pin wants to merge 1 commit into
databricks:masterfrom
He-Pin:fix-base64-string-codepoint-range
Closed

fix: std.base64 rejects string codepoints outside byte range#1108
He-Pin wants to merge 1 commit into
databricks:masterfrom
He-Pin:fix-base64-string-codepoint-range

Conversation

@He-Pin

@He-Pin He-Pin commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Motivation

The Jsonnet standard library defines std.base64 input as a string or an array whose codepoints/numbers are in the 0–255 range. sjsonnet already rejected out-of-range array values, but the string path passed every non-ASCII string straight to PlatformBase64, which silently UTF-8-encoded codepoints above 255:

std.base64(std.char(256))   // sjsonnet (before): "xIA="   go-jsonnet: RUNTIME ERROR
std.base64([256])           // both: error

The same logical input (std.char(256) vs [256]) behaved differently, and the string result does not round-trip through std.base64Decode on other implementations. This supersedes #1008, which went stale.

Modification

  • EncodingModule.scala: validate non-ASCII-safe string inputs codepoint-by-codepoint before encoding; fail on the first codepoint above 255 with a message mirroring the array path (base64 encountered invalid codepoint value in the string (must be 0 <= X <= 255), got N). The ASCII fast path and existing UTF-8 behavior for in-range (≤255) non-ASCII codepoints are unchanged.
  • Un-skip go_test_suite/builtinBase64_string_high_codepoint.jsonnet (JVM/Native and JS harnesses) and update its golden to the error output.
  • Add new_test_suite/error.base64_string_codepoint_out_of_range.jsonnet and error.base64_string_mid_codepoint_out_of_range.jsonnet covering the boundary (256) and a mid-string CJK codepoint.
  • Update Base64Tests.scala and base64_comprehensive.jsonnet so they no longer assert encoding of >255 codepoints.

Result

Verified against go-jsonnet v0.22.0 — behavior now matches:

Expression sjsonnet (after) go-jsonnet
std.base64(std.char(256)) error, got 256 error, got 256
std.base64("hello" + std.char(19990)) error, got 19990 error, got 19990
std.base64(std.char(255)) "w78=" "w78="
std.base64(std.char(200)) "w4g=" "w4g="
std.base64("café") "Y2Fmw6k=" "Y2Fmw6k="

Test plan

  • ./mill 'sjsonnet.jvm[2.13.18]'.test — all pass (including the un-skipped go_test_suite case and the two new error tests)
  • ./mill __.checkFormat — clean

Motivation:
The Jsonnet standard library defines std.base64 input as a string or an array whose codepoints or numbers are in the 0 to 255 range. sjsonnet already rejected out-of-range array values, but the string path passed every non-ASCII string to PlatformBase64 and UTF-8 encoded invalid codepoints instead of reporting an error.

Modification:
Validate non-AsciiSafeStr string inputs codepoint-by-codepoint before encoding and fail on the first codepoint above 255. Keep the existing ASCII fast path and existing UTF-8 behavior for in-range non-ASCII codepoints. Update Scala tests and file-test fixtures, including the official go_test_suite high-codepoint case, to cover the rejection behavior across JVM, JS, and Native file-test harnesses.

Result:
std.base64 now rejects string inputs outside the byte-string domain while preserving existing behavior for valid inputs and fast ASCII strings.
@He-Pin He-Pin closed this Aug 5, 2026
@He-Pin

He-Pin commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Closing: this change is wrong for this project.

Re-checking the behavior across implementations:

Expression jrsonnet 0.5.0 go-jsonnet v0.22.0 sjsonnet (master)
std.base64(std.char(256)) "xIA=" error "xIA="
std.base64("你好") "5L2g5aW9" error "5L2g5aW9"
std.base64Decode(std.base64("你好")) "你好" error "你好"
std.base64([256]) error error error

sjsonnet's string-path behavior is a deliberate design decision (the skipped go test builtinBase64_string_high_codepoint.jsonnet is annotated "We support base64 of unicode strings") and matches jrsonnet: strings are UTF-8 text (any codepoint allowed, round-trips through std.base64Decode), while arrays are the byte-string domain (0–255 enforced). The official docs do constrain codepoints to 0–255, so this is a conscious superset extension, not an oversight — go-jsonnet is the docs-strict implementation here.

This PR would have regressed that extension and broken parity with jrsonnet. Same conclusion applies to #1008.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant