fs: add native fast paths for recursive readdir and buffer I/O by anonrig · Pull Request #64396 · nodejs/node

anonrig · 2026-07-09T18:27:22Z

Summary

Adds C++ fast paths for common fs sync operations, following the same pattern as the existing readFileUtf8 / writeFileUtf8 bindings:

fs.readdirSync(path, { recursive: true }) — full directory walk in C++ (readdirRecursiveSync). Uses scandir dirent types so most entries skip the extra internalModuleStat the JS walk does per entry. Symlinks / UV_DIRENT_UNKNOWN still stat/lstat as needed for correct descent and Dirent types.
fs.readFileSync(path) / encoding: null — single native readFileBuffer (open / fstat / read / close). Throws ERR_FS_FILE_TOO_LARGE when size exceeds 2 GiB.
fs.writeFileSync(path, buffer) — single native writeFileBuffer for ArrayBufferView data when flush is not set.

Public API and results (sorted) stay the same; sequential/parallel readdir recursive and read/write file tests pass.

Benchmarks (arm64 macOS, Release)

Tree: ~2110 entries. Short runs (n=25 for readdir).

Path	ops/s	vs JS walk
C++ `readdirRecursiveSync`	354	1.77×
JS `readdir` + `internalModuleStat`	199	1×
`fs.readdirSync({ recursive: true })`	435	~2.2×
same + `withFileTypes`	364	~1.8×

Buffer I/O (same binary, short n):

Case	C++ vs multi-call JS
`readFileSync` small	~1.04×
`readFileSync` 64 KiB	~1.08×
`writeFileSync` 1 KiB buffer	~1.04×

Larger trees show recursive readdir closer to ~3× (stats dominate).

Test plan

test-fs-readdir-recursive (parallel + sequential)
test-fs-readdir-types / test-fs-readdir-types-symlinks
test-fs-readfile (+ empty, fd, flags, utf8 fast path)
test-fs-write-file* (sync, buffer, typedarrays, flush)
CI on this PR

mcollina · 2026-07-09T18:32:41Z

format cpp ;)

codecov · 2026-07-09T20:42:34Z

Codecov Report

❌ Patch coverage is 69.32907% with 96 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.08%. Comparing base (ad7a5b8) to head (f1bd58e).
⚠️ Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
src/node_file.cc	63.07%	35 Missing and 61 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #64396      +/-   ##
==========================================
- Coverage   90.11%   90.08%   -0.04%     
==========================================
  Files         752      752              
  Lines      251569   251845     +276     
  Branches    47268    47349      +81     
==========================================
+ Hits       226701   226863     +162     
- Misses      16233    16265      +32     
- Partials     8635     8717      +82

Files with missing lines	Coverage Δ
lib/fs.js	`98.29% <100.00%> (-0.07%)`	⬇️
src/node_errors.h	`86.95% <ø> (ø)`
src/node_file.cc	`73.28% <63.07%> (-0.95%)`	⬇️

... and 29 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

TheOneTheOnlyJJ · 2026-07-10T10:07:45Z

-    return binding.readFileUtf8(path, stringToFlags(options.flag));
+    const flags = stringToFlags(options.flag);
+    // C++ fast paths: single native call instead of open/stat/read/close.
+    if (options.encoding === 'utf8' || options.encoding === 'utf-8') {


Could these not also be uppercase? #64341 adds a new isUtf8Encoding to avoid missing valid encoding strings. Maybe a new lint rule would help avoid missing these.

Exactly! My PR #64341 introduces isUtf8Encoding() specifically to catch all valid case-insensitive UTF-8 variations (like UTF-8, utf-8, etc.) without needing repetitive if/else checks. Once #64341 lands, we can easily reuse that helper right here!

Done. Added isUtf8Encoding() that uses normalizeEncoding(), so the C++ fast paths also accept UTF8, UTF-8, and mixed-case aliases (not only lowercase utf8/utf-8).

jasnell · 2026-07-22T03:31:16Z

+    if (offset < static_cast<size_t>(size)) {
+      // Truncate logical length without reallocating when the file shrank.
+      args.GetReturnValue().Set(
+          Buffer::Copy(isolate, data, offset).ToLocalChecked());


Let's not use ToLocalChecked().

Done — ReadFileBuffer now uses ToLocal() and returns on allocation failure instead of ToLocalChecked().

Move readdirSync({ recursive: true }) into a C++ walk that uses scandir dirent types so most entries avoid an extra stat(). Add readFileBuffer and writeFileBuffer bindings so Buffer readFileSync and writeFileSync avoid open/stat/read/write/close round-trips in JS, matching the existing utf8 fast paths. Throw ERR_FS_FILE_TOO_LARGE from C++ when the file exceeds 2 GiB. On a ~2k-entry tree, recursive readdir is about 2x faster than the JS walk; buffer read/write see smaller single-digit gains. Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>

Always call the native readdirRecursiveSync binding and remove the unused JS fallback. Document readFileBuffer, writeFileBuffer, and readdirRecursiveSync on the fs internal binding typings. Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>

Replace the extra S_IS* macros with a switch on S_IFMT, avoid a temporary string when encoding relative paths, and tighten the directory-descent type checks. Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>

Use primordials Array instead of the global, return undefined explicitly for the JSDoc check, and match clang-format on the directory descent condition. Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>

The C++ readdirRecursiveSync fast path handles Buffer paths, so test-fs-readdir-sync-recursive-with-buffer now passes. Known-issues tests are expected to fail; a passing negative test fails CI. Also avoid ToLocalChecked() in ReadFileBuffer and drop the always-true st_size >= 0 comparison. Refs: nodejs#58892

The node-core eslint rules require test files to load common first.

Use normalizeEncoding so readFileSync/writeFileSync take the C++ fast path for UTF8, UTF-8, and mixed-case variants, not only lowercase utf8/utf-8. Refs: nodejs#64341

mcollina

lgtm

nodejs-github-bot · 2026-08-16T18:25:27Z

CI: https://ci.nodejs.org/job/node-test-pull-request/75892/

jasnell · 2026-08-17T04:08:07Z

Large number of directly relevant failures on Windows.

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run. labels Jul 9, 2026

anonrig requested review from mcollina and ronag July 9, 2026 18:30

anonrig force-pushed the fs-huge-perf-win branch from 0aef62a to e76ef36 Compare July 9, 2026 18:32

anonrig force-pushed the fs-huge-perf-win branch from 3073a3b to f0e1dc9 Compare July 9, 2026 18:51

TheOneTheOnlyJJ reviewed Jul 10, 2026

View reviewed changes

jasnell reviewed Jul 22, 2026

View reviewed changes

anonrig added 5 commits August 16, 2026 17:17

src: simplify fs recursive readdir helpers

1bedeac

Replace the extra S_IS* macros with a switch on S_IFMT, avoid a temporary string when encoding relative paths, and tighten the directory-descent type checks. Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>

fs: fix lint for recursive readdir fast path

aef3f10

Use primordials Array instead of the global, return undefined explicitly for the JSDoc check, and match clang-format on the directory descent condition. Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>

anonrig force-pushed the fs-huge-perf-win branch from 5a4bf75 to bdaef34 Compare August 16, 2026 17:19

anonrig added 2 commits August 16, 2026 17:57

test: require common in recursive readdir Buffer path test

a0a9640

The node-core eslint rules require test files to load common first.

fs: accept all utf8 encoding aliases on buffer I/O fast paths

f1bd58e

Use normalizeEncoding so readFileSync/writeFileSync take the C++ fast path for UTF8, UTF-8, and mixed-case variants, not only lowercase utf8/utf-8. Refs: nodejs#64341

mcollina approved these changes Aug 16, 2026

View reviewed changes

mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 16, 2026

github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 16, 2026

Navigation Menu

Uh oh!

Uh oh!

fs: add native fast paths for recursive readdir and buffer I/O - #64396

fs: add native fast paths for recursive readdir and buffer I/O#64396
anonrig wants to merge 7 commits into
nodejs:mainfrom
anonrig:fs-huge-perf-win

anonrig commented Jul 9, 2026

Uh oh!

mcollina commented Jul 9, 2026

Uh oh!

codecov Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

TheOneTheOnlyJJ Jul 10, 2026

Uh oh!

hamidrezaghavami Jul 14, 2026

Uh oh!

anonrig Aug 16, 2026

Uh oh!

jasnell Jul 22, 2026

Uh oh!

anonrig Aug 16, 2026

Uh oh!

mcollina left a comment

Uh oh!

nodejs-github-bot commented Aug 16, 2026

Uh oh!

jasnell commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Uh oh!

Conversation

anonrig commented Jul 9, 2026

Summary

Benchmarks (arm64 macOS, Release)

Test plan

Uh oh!

mcollina commented Jul 9, 2026

Uh oh!

codecov Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

TheOneTheOnlyJJ Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

hamidrezaghavami Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

anonrig Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

jasnell Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

anonrig Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Aug 16, 2026

Uh oh!

jasnell commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov Bot commented Jul 9, 2026 •

edited

Loading