perf: reuse projection schema in OptimizeProjections instead of recomputing it

[zhuqi-lucas]

Which issue does this close?

Closes #24264. Related to prior OptimizeProjections perf work (#21726).

Rationale for this change

rewrite_projection_given_requirements (the core of the OptimizeProjections rule) prunes a projection's expressions to the subset actually required, then rebuilds the projection with Projection::try_new:

Projection::try_new(exprs_used, Arc::new(input))

try_new recomputes the output schema from scratch via projection_schema, which:

• calls Expr::to_field for every retained expression (type/nullability/qualifier/metadata inference), and
• for column expressions resolves the field through DFSchema::field_from_column → index_of_column_by_name, which is a linear scan over the input schema (there is no name→index map).

So recomputing one projection's schema is O(exprs * schema_width), and it runs for every projection on every optimizer pass. For wide, SELECT *-style projections over wide schemas (tens of columns) this becomes effectively quadratic and shows up prominently in planning profiles.

But the retained expressions are a subset of the projection's original expressions, and pruning unreferenced sibling columns cannot change the retained columns' output fields. The answer is already sitting in proj.schema — no need to re-derive it.

What changes are included in this PR?

• rewrite_projection_given_requirements now derives the pruned output schema by selecting the already-computed fields from the existing projection schema (project_schema_by_indices) and builds the projection with Projection::try_new_with_schema, instead of recomputing via try_new.
• When nothing is pruned (the identity / SELECT * case), the existing schema Arc is reused as-is.
• Functional dependencies are projected through the kept indices (FunctionalDependencies::project_functional_dependencies).
• This mirrors the schema reuse already performed in merge_consecutive_projections (which reuses schema unchanged when the expression list is unchanged).

Complexity for a pruned projection goes from O(exprs * schema_width) (schema recompute) to O(k) (field slice), and to O(1) when nothing is pruned.

Correctness

The sliced schema is identical to the one projection_schema would recompute: field i of the projection schema corresponds to expression i, and RequiredIndices yields a sorted, deduplicated index subset, so slicing proj.schema at those indices produces exactly the fields of the retained expressions, with qualifiers and metadata preserved.

New test project_schema_by_indices_matches_recompute asserts, for a mixed expression list (plain column, computed binary expr, alias, nullable literal, qualified column) and every representative index subset, that project_schema_by_indices(schema, indices) produces the same fields and qualifiers as projection_schema(input, exprs_used), and that the identity subset reuses the same Arc.

The full datafusion-optimizer test suite (760 unit + 26 integration, including the EXPLAIN plan snapshots) passes unchanged, i.e. no optimized plan output changes.

Are there any user-facing changes?

No. This is an internal optimizer performance improvement; planned/optimized plans are unchanged.

[Copilot]

Pull request overview

Improves OptimizeProjections performance by avoiding repeated recomputation of projection output schemas when pruning projection expressions, instead reusing/slicing the already-computed Projection.schema.

Changes:

• Update rewrite_projection_given_requirements to build pruned projections via Projection::try_new_with_schema using a sliced schema from the existing projection schema.
• Add project_schema_by_indices helper to project fields + functional dependencies while reusing schema metadata.
• Add a unit test validating that sliced schemas match projection_schema recomputation across representative index subsets.

Suppressed comments (1)

datafusion/optimizer/src/optimize_projections/mod.rs:1285

• project_schema_by_indices also projects functional dependencies and preserves schema-level metadata, but the test currently only compares fields and qualifiers. Adding assertions for functional dependencies and schema metadata will better protect the behavior this PR relies on.

            // Output fields (name, data type, nullability, field metadata) must
            // match the from-scratch computation exactly.
            assert_eq!(
                reused.fields(),
                recomputed.fields(),
                "fields differ for indices {indices:?}"
            );

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.