feat: shen.x.sha256 host (OpenSSL) - #48
Open
pyrex41 wants to merge 7 commits into
Open
Conversation
… live lambdas
On the S41.2 kernel every (define ...) in a loaded file runs
shen.update-lambdatable, which (set)s shen.*lambdatable* to an assoc
list whose entries hold live curried lambda functions. The fasl
recorder captured that as an ordinary "g" (set) record, kdata_ser hit
the closure ("unserializable KDATA value: function"), and the whole
file became fasl-uncacheable — in practice ~every stdlib and user file,
so the ~1s stdlib load re-compiled on every boot and every (load) paid
full reader+macro+typecheck cost on every run.
Record a new "lt" record instead: diff the new table against the
current global (the recorder runs before the original set, so the old
table is still live) and store only the NAMES whose entry was
added/replaced. On replay, rebuild each entry from the live defun the
same way shen.update-lambdatable does — shen.lambda-entry (which reads
the arity property, already replayed by the preceding "p" record in
stream order) + shen.assoc->. Mirrors the existing "lf" special case
for the older kernel's shen.lambda-form put path.
Bump FASL_FORMAT to SHENFASL5 so stale caches miss cleanly.
Measured (arm64 macOS, LuaJIT 2.1 rolling): trivial-script startup
1.5s -> 0.34s warm; all 21 StLib files now fasl-hit (previously 19/21
uncacheable). make test 500/0, run-kernel-tests 134/0, warm/cold suite
stdout identical modulo the by-design 'run time' banner drop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… user output -q sets *hush*, which on the 41.2 kernel gates pr itself, so a suite run under -q prints NOTHING including the tests' own (output ...) lines (issue #46 item 3) — unusable for golden-output batch runners. Meanwhile default mode echoes every loaded form's value/type (~100 lines on urdr suites) plus nondeterministic 'run time: N secs' banners. --hush-load (bin/shen flag, or env SHEN_HUSH_LOAD=1) drops ONLY what load itself prints to standard output: the per-form value/type echo (shen.eval-and-print / shen.work-through) and the run-time/typechecked banners. User (output ...)/pr from loaded forms is untouched, as are file streams and stderr. Mechanism reuses the distinction the fasl 'e' capture already relies on: load's chatter is pr'd OUTSIDE any eval-kl chunk, at the chunk nesting depth load was entered at, while user output runs INSIDE a chunk. boot.lua wraps load to publish that depth (P.LOADPR_DEPTH, saved/restored so nested loads compose); prims' native pr drops stdout writes at exactly that depth; prims.compile_and_load maintains the depth counter (only when recording or the flag is on — the default path stays a bare tail call). Fasl integration: the load wrapper sits inside install_fasl's wrappers, so a miss recorded under --hush-load still captures the full 'e' echo bytes (replayable to a later non-hushed run); a replay hit is gated at the 'e' re-emit site. Verified: default-mode stdout byte-identical (warm hit) / identical mod timing lines (cold) on urdr prng+world suites; hush warm==cold==env form; every hush-mode line present in shen-cl's output for the same suite; cache recorded under hush replays full echo non-hushed; -q still fully silent; make test 500/0; run-kernel-tests 134/0 ok. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…egen jit.p profiling of urdr's software-SHA-256 suite (shen/tests/prng, LuaJIT 2.1 rolling, arm64) showed 27% of time in the interpreter with cons/append/list.take at the top, and jit.v showed why: 282 of the ~350 aborted trace attempts were 'blacklisted at prims.lua:239' — the tiny F["cons"] wrapper proto. cons is hot enough that traces root in it, those aborts blacklist the proto, and from then on EVERY caller trace that reaches a cons call aborts, demoting the whole workload to the interpreter. Emit the construction inline instead — ENV.CMT is the runtime Cons metatable — so there is no proto to blacklist and hot traces can sink the allocation. Same late-binding tradeoff as the existing ARITH2 fast paths (a runtime redefinition of cons is not seen by compiled exact-arity call sites). urdr prng suite wall (warm fasl, best of 5): 3.20s -> 1.43s (-55%). world suite: 4.4s -> 3.08s. After: 0 blacklisted traces, interpreter share 27% -> 8%, compiled share 37% -> 57%. Verified: suite stdout byte-identical (warm and cold); world golden exact; make test 500/0; run-kernel-tests 134/0 ok. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After the cons-inline fix, jit.p on urdr's prng suite still shows ~27% of wall time in the GC — the SHA-256/bigint workload is cons churn the port cannot avoid (bits are Shen lists). Interleaved A/B on CPU time: pause=400 -15% prng, -20% world, peak RSS 30MB -> 60MB pause=400 mul=100 -16% (stepmul adds ~nothing) pause=800 mul=100 -25-28% peak RSS 108MB pause=200 mul=100 no change (stepmul alone is not the lever) Ship pause=400 as the boot default: solid suite-level win for a bounded 2x peak-RSS cost. SHEN_GC=off preserves the host's defaults (embedders managing their own GC); SHEN_GC="pause[,stepmul]" sets explicit values (800,100 documented for batch runs). Also rejected: jit.opt loopunroll=60/callunroll=10 — helps only warm in-process re-runs, cold-process suite wall got 30% WORSE (trace compile churn); left at current defaults. Verified: prng warm stdout byte-identical; world golden exact; make test 500/0; run-kernel-tests 134/0 ok. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…unction The 41.2 shen->kl translator compiles every call to a function whose arity is unknown at translation time (chiefly forward references within a file) as ((fn name) args...). Kernel fn (reader.kl) then pays an arity property get PLUS an assoc over the whole shen.*lambdatable* per call, and returns the table's curried lambda, which APP applies one MKFUN closure at a time. jit.p F3 stacks on urdr's prng suite put ~35% of samples under fn (is_cons/equal < assoc < fn < urdr.prng.word.add-all/add2/...). Native fn_fast (prims.install_native_stdlib): for a symbol naming a live F entry with known positive arity, verify the lambdatable entry ONCE per (name, lambdatable identity) through the original fn — a miss still raises the kernel's "fn: X is undefined" — then return the RAW F[name] function. APP dispatches a raw function with recorded arity identically to the curried chain (exact, partial, and over-application) without the per-argument closure allocations. Every (set shen.*lambdatable* ...) builds a fresh spine, so table identity invalidates the verification cache; arity-0 (kernel fn CALLS those) and unknown-arity names always take the original path. Interleaved A/B (user+sys CPU, 6 runs each, sorted): prng: 3.14 3.34 3.43 3.61 3.68 3.88 -> 2.32 2.47 2.47 2.51 2.54 3.38 (-27%) world: 4.59 4.70 4.82 5.01 5.09 5.39 -> 3.92 4.16 4.35 4.54 4.61 5.35 (-12%) Verified: prng warm stdout byte-identical; world golden exact; make test 500/0; run-kernel-tests 134/0 ok. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jit.p Fl on urdr's prng suite put ~10% of leaf samples in the kernel
append (sys.kl): non-tail-recursive, one F["hd"]/F["tl"]/F["append"]
lookup per element of the first list. word.rotr / word.bits concatenate
32-bit bit-lists through append on every SHA round.
Native install_native_stdlib override: walk A once building a fresh
spine iteratively (exactly |A| new cons cells — same allocation as the
recursive form), splice B onto the last cell. Fresh cells are unshared
until return, so in-place tail assignment is unobservable. Improper /
non-list inputs delegate to the original for a byte-identical
simple-error.
Serial warm A/B (user+sys CPU, dual SHEN_FASL_DIR caches, N=8–10):
prng: med 0.93 -> 0.91 (-2%), mean 1.00 -> 0.92 (-8%);
final recheck med 0.90 -> 0.87 (-3%), mean 0.95 -> 0.91 (-4%)
world: med 2.28 -> 1.85 (-19%)
Also tried (not shipped): pure_tail_self bare f-error relaxation —
correctly emits LOOP for multi-clause defines (list.drop/nth) but mean
prng regressed ~+14% on short-list SHA walks; left documented in
doc/PERF-URDR-RESULTS.md.
Verified: prng/world warm stdout golden-identical; make test 500/0;
run-kernel-tests 134/0 ok.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Binds shen.x.sha256-octets-host from install_native_stdlib when libcrypto loads. Disable with SHEN_X_SHA256=pure. See pyrex41/shen-extensions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SHA256via LuaJIT FFI for shen-extensionsSHEN_X_SHA256=puredisablesTest plan