Accelerating Apache Spark Queries (and Iceberg Rust Development) with Apache DataFusion Comet
Apache Iceberg's ecosystem spans multiple query engines and language implementations that work together to give users a consistent experience across the data lakehouse. This post explores one integration within that ecosystem, Iceberg Rust and Apache DataFusion Comet, and the two benefits their relationship brings. Comet accelerates Apache Spark's reads over Iceberg tables by running them natively through Iceberg Rust. That same integration turns Iceberg Java's nearly 10,000 Spark tests into a differential-testing harness whose benefits run both ways: Iceberg Rust gets exercised against a broad corpus of real-world scenarios, and the comparison has even caught bugs in Iceberg Java. The resulting fixes land upstream and benefit every project built on these libraries, not just Comet, as the Iceberg and DataFusion communities build on each other's strengths.