Skip to content

Iceberg serde: residual reflection failures can silently drop residual filtering #5992

Description

@unikdahal

Describe the bug

CometIcebergNativeScan can turn residual-expression reflection or conversion failures into None.

At serde time, native execution has already been selected, so None means the task is serialized without that residual predicate.

This is fine when a residual is intentionally not pushed and an exact filter above the scan still enforces correctness. It is not fine when reflection unexpectedly fails, because both cases currently look the same.

Steps to reproduce

No response

Expected behavior

Residual conversion should distinguish between:

  • intentionally unsupported / not pushed; and
  • reflection or conversion failure.

Unexpected failures during native serde should fail the query when correctness may depend on the residual, instead of silently dropping it.

Additional context

Found during review of #5515.

Related to the same planning-vs-serde failure boundary tracked in #5257, but the residual-expression path is not covered there.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:scanParquet scan / data readingbugSomething isn't workingpriority:criticalData corruption, silent wrong results, security issues

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions