Skip to content

feat: support reading dictionary-encoded columns over Arrow Flight - #433

Open
owencorrigan wants to merge 2 commits into
apache:mainfrom
owencorrigan:dictionary-flight-support
Open

feat: support reading dictionary-encoded columns over Arrow Flight#433
owencorrigan wants to merge 2 commits into
apache:mainfrom
owencorrigan:dictionary-flight-support

Conversation

@owencorrigan

Copy link
Copy Markdown

What's Changed

The C# Arrow Flight reader (RecordBatchReaderImplementation) threw
NotImplementedException on any dictionary batch, so a Flight stream containing a
dictionary-encoded column could not be read.

  • ReadNextRecordBatchAsync now loops over the incoming Flight messages, feeding
    each through the base CreateArrowObjectFromMessage (which reads dictionary
    batches into the DictionaryMemo and returns null for them) until it yields a
    record batch, instead of throwing on any non-record-batch message. Handles both
    single and replacement dictionaries.
  • The schema is decoded using the reader's DictionaryMemo via a new base helper
    ReadSchemaFromMessage, so dictionary-encoded fields are registered
    (FlightMessageSerializer previously discarded them).

The change routes dictionary batches to the existing ArrowReaderImplementation
decoder, so it is not specific to any particular index or value type. Validated
against a pyarrow RecordBatchStream serving a dictionary<int16, string> column
(single + replacement dictionaries); the current reader throws on the same stream.

Refs #180 (receive side).

Generated-By: Claude Opus 4.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant