Package org.apache.iceberg.spark

Class RollbackStagedTable

  • All Implemented Interfaces:
    org.apache.spark.sql.connector.catalog.StagedTable, org.apache.spark.sql.connector.catalog.SupportsDelete, org.apache.spark.sql.connector.catalog.SupportsDeleteV2, org.apache.spark.sql.connector.catalog.SupportsRead, org.apache.spark.sql.connector.catalog.SupportsWrite, org.apache.spark.sql.connector.catalog.Table, org.apache.spark.sql.connector.catalog.TruncatableTable
    public class RollbackStagedTable
extends java.lang.Object
implements org.apache.spark.sql.connector.catalog.StagedTable, org.apache.spark.sql.connector.catalog.SupportsRead, org.apache.spark.sql.connector.catalog.SupportsWrite, org.apache.spark.sql.connector.catalog.SupportsDelete
    An implementation of StagedTable that mimics the behavior of Spark's non-atomic CTAS and RTAS.

    A Spark catalog can implement StagingTableCatalog to support atomic operations by producing StagedTable. But if a catalog implements StagingTableCatalog, Spark expects the catalog to be able to produce a StagedTable for any table loaded by the catalog. This assumption doesn't always work, as in the case of SparkSessionCatalog, which supports atomic operations can produce a StagedTable for Iceberg tables, but wraps the session catalog and cannot necessarily produce a working StagedTable implementation for tables that it loads.

    The work-around is this class, which implements the StagedTable interface but does not have atomic behavior. Instead, the StagedTable interface is used to implement the behavior of the non-atomic SQL plans that will create a table, write, and will drop the table to roll back.

    This StagedTable implements SupportsRead, SupportsWrite, and SupportsDelete by passing the calls to the real table. Implementing those interfaces is safe because Spark will not use them unless the table supports them and returns the corresponding capabilities from capabilities().

    • Constructor Summary

      Constructors 
      Constructor Description
      RollbackStagedTable​(org.apache.spark.sql.connector.catalog.TableCatalog catalog, org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.connector.catalog.Table table)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void abortStagedChanges()  
      java.util.Set<org.apache.spark.sql.connector.catalog.TableCapability> capabilities()  
      void commitStagedChanges()  
      void deleteWhere​(org.apache.spark.sql.sources.Filter[] filters)  
      java.lang.String name()  
      org.apache.spark.sql.connector.read.ScanBuilder newScanBuilder​(org.apache.spark.sql.util.CaseInsensitiveStringMap options)  
      org.apache.spark.sql.connector.write.WriteBuilder newWriteBuilder​(org.apache.spark.sql.connector.write.LogicalWriteInfo info)  
      org.apache.spark.sql.connector.expressions.Transform[] partitioning()  
      java.util.Map<java.lang.String,​java.lang.String> properties()  
      org.apache.spark.sql.types.StructType schema()  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface org.apache.spark.sql.connector.catalog.SupportsDelete

        canDeleteWhere, canDeleteWhere, deleteWhere, truncateTable

      • Methods inherited from interface org.apache.spark.sql.connector.catalog.Table

        columns

    • Constructor Detail

      • RollbackStagedTable

        public RollbackStagedTable​(org.apache.spark.sql.connector.catalog.TableCatalog catalog,
                           org.apache.spark.sql.connector.catalog.Identifier ident,
                           org.apache.spark.sql.connector.catalog.Table table)

    • Method Detail

      • commitStagedChanges

        public void commitStagedChanges()
        Specified by:
        commitStagedChanges in interface org.apache.spark.sql.connector.catalog.StagedTable

      • abortStagedChanges

        public void abortStagedChanges()
        Specified by:
        abortStagedChanges in interface org.apache.spark.sql.connector.catalog.StagedTable

      • name

        public java.lang.String name()
        Specified by:
        name in interface org.apache.spark.sql.connector.catalog.Table

      • schema

        public org.apache.spark.sql.types.StructType schema()
        Specified by:
        schema in interface org.apache.spark.sql.connector.catalog.Table

      • partitioning

        public org.apache.spark.sql.connector.expressions.Transform[] partitioning()
        Specified by:
        partitioning in interface org.apache.spark.sql.connector.catalog.Table

      • properties

        public java.util.Map<java.lang.String,​java.lang.String> properties()
        Specified by:
        properties in interface org.apache.spark.sql.connector.catalog.Table

      • capabilities

        public java.util.Set<org.apache.spark.sql.connector.catalog.TableCapability> capabilities()
        Specified by:
        capabilities in interface org.apache.spark.sql.connector.catalog.Table

      • deleteWhere

        public void deleteWhere​(org.apache.spark.sql.sources.Filter[] filters)
        Specified by:
        deleteWhere in interface org.apache.spark.sql.connector.catalog.SupportsDelete

      • newScanBuilder

        public org.apache.spark.sql.connector.read.ScanBuilder newScanBuilder​(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
        Specified by:
        newScanBuilder in interface org.apache.spark.sql.connector.catalog.SupportsRead

      • newWriteBuilder

        public org.apache.spark.sql.connector.write.WriteBuilder newWriteBuilder​(org.apache.spark.sql.connector.write.LogicalWriteInfo info)
        Specified by:
        newWriteBuilder in interface org.apache.spark.sql.connector.catalog.SupportsWrite