HIVE-29841: Iceberg: Limit compaction with a filter to the latest par…

[difin]

…tition spec only

What changes were proposed in this pull request?

For Iceberg compaction with a filter in the compaction command on a table with partition evolution, limit the compaction to partitions under the latest partition spec only, and skip the old-spec multi-partition compaction when a filter is present.

Why are the changes needed?

Currently the filter can match only a subset of the partitions with old partition spec, but all of those partitions are compacted by Iceberg compaction with a filter.

Does this PR introduce any user-facing change?

Yes, for Iceberg compaction with a filter in the compaction command on a table with partition evolution, only partitions with the current spec will be compacted.

How was this patch tested?

Executed Iceberg Compaction q-tests and updated the golden files according to the new logic.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.6% Duplication on New Code

See analysis details on SonarQube Cloud

[Aggarwal-Raghav]

@difin , this is my understanding of the patch. Can you please correct me if I missed anything?
Possible Limitation:
After this PR, user can no longer manually target old partition specs using a WHERE filter. Users will now have to rely on the background auto-compactor (below ROW 18) or run a full-table compaction without a WHERE clause correct?

Testing:
I checked the behaviour explained in description locally and can confirm it works exactly as described.

Before this patch, running ALTER TABLE tbl1 COMPACT 'MAJOR' AND WAIT WHERE company_id = 100 on an evolved Iceberg table would correctly enqueue the new partitions, but also blindly enqueue a --- compaction (ROW 19), confirming a rewrite of all old-spec data.

    | compactionid | tabname | partname                 | type  | state     | initiatorid |
    | 16           | tbl1    | company_id=100/dept_id=2 | MAJOR | refused   | manual      |
    | 17           | tbl1    | company_id=100/dept_id=1 | MAJOR | succeeded | manual      |
    | 18           | tbl1    | ---                      | MAJOR | succeeded | 884 (auto)  |
    | 19           | tbl1    | ---                      | MAJOR | refused   | manual      |

After this PR, The --- row with manual entry is entirely GONE!. I think this is expected fix!

    | 23 | test | tbl1 | company_id=100/dept_id=1 | MAJOR | succeeded | manual |
    | 22 | test | tbl1 | company_id=100/dept_id=2 | MAJOR | refused   | manual |

[Aggarwal-Raghav]

Raised #6737 for a bug I found during testing this PR.