Iceberg Table Maintenance: Acceleration Opportunities

[Shekharrajak]

What is the problem the feature request solves?

We should explore and benchmark Iceberg table maintenance operations, while checking for any missing DataFusion implementations that could accelerate Spark Iceberg table maintenance.

Describe the potential solution

No response

Additional context

Iceberg table maintenance using Spark: https://iceberg.apache.org/docs/latest/maintenance/#delete-orphan-files

Medium blog compaction with datafusion + rust : https://medium.com/@amitgil87/building-a-distributed-compaction-engine-for-apache-iceberg-with-rust-datafusion-09faa63c1438

Some old discussion to refer : apache/iceberg-rust#624

[mbutrovich]

It's worth pursuing at some point. We'd need to start by characterizing the writer features in iceberg-rust. We started with the reader first since those gaps seemed more important to close first.

[Shekharrajak]

We'd need to start by characterizing the writer features in iceberg-rust

Can you please share reference links - discussion thread, issue tracker ? I will also look into.

[mbutrovich]

We'd need to start by characterizing the writer features in iceberg-rust

Can you please share reference links - discussion thread, issue tracker ? I will also look into.

I don’t have any. I’d be starting from the same iceberg-rust discussion issue you linked and reading the code.