Skip to content

blog🔗

  • in blog
  • 7 min read

Semi-Structured Data in Apache Iceberg: Meet the Variant Type

Semi-structured data, such as JSON-like documents whose fields differ from row to row, has always been a poor fit for table formats built around fixed schemas. Iceberg v3 adds the Variant type for exactly this data: a single column can hold values of arbitrary, evolving shape, stored as a compact binary value that engines read and write consistently.

This is the first post in a series on Variant in Apache Iceberg: what Variant is, why it exists, and how it fits into an Iceberg table. Parquet already defines the Variant type and its binary encoding, so this post focuses on what Iceberg adds on top: how Variant fits the table's schema, files, snapshots, and statistics.

Variant is stored in Parquet, Avro, and ORC. Shredding, an optimization that stores commonly queried Variant fields as their own typed columns, is available only in Parquet; a later post covers how it works.

  • in blog
  • 9 min read

Accelerating Apache Spark Queries (and Iceberg Rust Development) with Apache DataFusion Comet

Apache Iceberg's ecosystem spans multiple query engines and language implementations that work together to give users a consistent experience across the data lakehouse. This post explores one integration within that ecosystem, Iceberg Rust and Apache DataFusion Comet, and the two benefits their relationship brings. Comet accelerates Apache Spark's reads over Iceberg tables by running them natively through Iceberg Rust. That same integration turns Iceberg Java's nearly 10,000 Spark tests into a differential-testing harness whose benefits run both ways: Iceberg Rust gets exercised against a broad corpus of real-world scenarios, and the comparison has even caught bugs in Iceberg Java. The resulting fixes land upstream and benefit every project built on these libraries, not just Comet, as the Iceberg and DataFusion communities build on each other's strengths.