Skip to content

Parallelize generated source file writes - #963

Merged
iuliiasobolevska merged 1 commit into
masterfrom
feature/parallel-file-writing
Sep 23, 2026
Merged

iuliiasobolevska merged 1 commit into
masterfrom
feature/parallel-file-writing

Conversation

@iuliiasobolevska

Copy link
Copy Markdown
Collaborator

Summary

  • write independent Java and Kotlin source files through a dedicated bounded executor after generation completes
  • default file-write parallelism to min(availableProcessors, 8), configurable through CodeGenConfig, the Gradle task, and --file-write-parallelism; setting it to 1 preserves serial writing
  • reject duplicate output paths, including case-only collisions, before writing
  • propagate write failures with the generated path, cancel queued work, and wait for in-flight writers to terminate before returning
  • keep example data fetchers and documentation on their existing serial paths because they use different output directories

Performance

The results vary with how much of a workload is output rendering and disk emission.

Consumer schema

The 3.7 MB consumer schema generated 9,000 files. Median CLI generation time improved from 5.06 s to 3.94 s, a 22% reduction, with byte-identical output across all files.

Adversarial synthetic benchmark

The review benchmark used 300 object types with 15 fields each, client API and interfaces enabled, and produced 1,502 files. It ran three warmed iterations for each setting:

Measurement 1 writer 8 writers Median change
File writing only 257 / 249 / 235 ms 76 / 78 / 75 ms 249 to 76 ms, 3.3x faster
Full CodeGen.generate() 7090 / 7151 / 7119 ms 6854 / 6938 / 7050 ms 7119 to 6938 ms, 2.5% reduction

The synthetic result shows that this change only accelerates emission; parsing and model construction remain serial and dominate schemas where writing is a small fraction of total time.

Behavior changes

  • Client projection write failures now fail generation instead of being printed and swallowed.
  • Multiple generated sources targeting the same path now fail before emission instead of silently overwriting one another.

Verification

  • Java and Kotlin serial/parallel runs produce the same complete path and content maps.
  • Failure tests verify that the generated path is reported and no writer thread remains alive after write() throws.
  • Duplicate and case-only duplicate destinations are rejected before output is written.
  • :graphql-dgs-codegen-core:test passes.
  • :graphql-dgs-codegen-gradle:test passes.

@iuliiasobolevska iuliiasobolevska added the enhancement New feature or request label Sep 21, 2026
@iuliiasobolevska iuliiasobolevska added the java Pull requests that update Java code label Sep 21, 2026
@iuliiasobolevska
iuliiasobolevska merged commit 634fb25 into master Sep 23, 2026
2 of 3 checks passed
@iuliiasobolevska
iuliiasobolevska deleted the feature/parallel-file-writing branch September 23, 2026 15:00

This branch is waiting to be deployed

1 waiting deployment
Publish d8a193ac Waiting Sep 21, 2026 by iuliiasobolevska via build #328
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request java Pull requests that update Java code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants