Flink: Support variant for Avro readers and writers

[Guosmilesmile]

Iceberg v3 defines the Avro encoding for variant: a record with metadata and value binary fields (shredding is not supported in Avro). The core avro layer (ValueWriters.variants() / ValueReaders.variants()) and the Spark bridges (SparkAvroWriter / SparkPlannedAvroReader) already implement this encoding, and the Flink Parquet bridges also support variant.

However, the Flink Avro bridges do not: FlinkAvroWriter and FlinkPlannedAvroReader do not override variant(), so any read or write of an Avro data file containing a variant column fails with:

java.lang.UnsupportedOperationException: Visitor does not support variant

This pr aim to add support variant for Avro readers and writers.

TestFlinkAvroReaderWriter passes with the variant round-trip cases enabled (write records / read RowData, and write RowData via FlinkAvroWriter / read back via FlinkPlannedAvroReader).

Remove the Miss Feature variant from Avro, and use the file format TCK to test this feature as well.

[pvary]

CC: @nssalian

[uros-b]

LGTM, please resolve conflicts @Guosmilesmile

[nssalian]

The writer's bytes are only checked by reading them back with the Flink reader. Writing with FlinkAvroWriter and reading back with the generic PlannedDataReader would confirm them without depending on the Flink reader. Not a blocker, the current coverage matches the Spark tests.
Up to you to do it here or a follow up. Lgtm otherwise. Thanks @Guosmilesmile

[Guosmilesmile]

@nssalian Makes sense. I'd like to keep this PR aligned with the Spark test structure and will open a follow-up adding the generic-reader cross-check for the writer path.

[Guosmilesmile]

@nssalian I double-checked this, and the generic-reader cross-check is already covered by the format model TCK.

BaseFormatModelTests.testDataWriterEngineWriteGenericRead writes data using the engine writer registered in FlinkFormatModels (FlinkAvroWriter) and reads it back as generic Records. The test is parameterized over FORMAT_AND_GENERATOR, which includes (AVRO, Variant), so the bytes produced by the writer are already validated against an independent implementation.

The same cross-check is also covered for Spark via TestSparkFormatModel. In fact, this PR fixes the previously failing case.

[nssalian]

Sounds great. Thanks for checking @Guosmilesmile. @pvary any comments?

[pvary]

Merged to main.
Thanks @Guosmilesmile for the PR and @uros-b and @nssalian for the reviews!