Package org.apache.iceberg.spark.actions

Class BaseSnapshotTableSparkAction

  • All Implemented Interfaces:
    Action<SnapshotTable,​SnapshotTable.Result>, SnapshotTable
    public class BaseSnapshotTableSparkAction
extends java.lang.Object
implements SnapshotTable
    Creates a new Iceberg table based on a source Spark table. The new Iceberg table will have a different data and metadata directory allowing it to exist independently of the source table.

    • Constructor Summary

      Constructors 
      Constructor Description
      BaseSnapshotTableSparkAction​(org.apache.spark.sql.SparkSession spark, org.apache.spark.sql.connector.catalog.CatalogPlugin sourceCatalog, org.apache.spark.sql.connector.catalog.Identifier sourceTableIdent, org.apache.spark.sql.connector.catalog.CatalogPlugin destCatalog, org.apache.spark.sql.connector.catalog.Identifier destTableIdent)  

    • Field Detail

      • ICEBERG_METADATA_FOLDER

        protected static final java.lang.String ICEBERG_METADATA_FOLDER
        See Also:
        Constant Field Values

      • EXCLUDED_PROPERTIES

        protected static final java.util.List<java.lang.String> EXCLUDED_PROPERTIES

    • Constructor Detail

      • BaseSnapshotTableSparkAction

        public BaseSnapshotTableSparkAction​(org.apache.spark.sql.SparkSession spark,
                                    org.apache.spark.sql.connector.catalog.CatalogPlugin sourceCatalog,
                                    org.apache.spark.sql.connector.catalog.Identifier sourceTableIdent,
                                    org.apache.spark.sql.connector.catalog.CatalogPlugin destCatalog,
                                    org.apache.spark.sql.connector.catalog.Identifier destTableIdent)

    • Method Detail

      • destCatalog

        protected org.apache.spark.sql.connector.catalog.StagingTableCatalog destCatalog()

      • destTableIdent

        protected org.apache.spark.sql.connector.catalog.Identifier destTableIdent()

      • as

        public SnapshotTable as​(java.lang.String ident)
        Description copied from interface: SnapshotTable
        Sets the table identifier for the newly created Iceberg table.
        Specified by:
        as in interface SnapshotTable
        Parameters:
        ident - the destination table identifier
        Returns:
        this for method chaining

      • tableProperties

        public SnapshotTable tableProperties​(java.util.Map<java.lang.String,​java.lang.String> properties)
        Description copied from interface: SnapshotTable
        Sets table properties in the newly created Iceberg table. Any properties with the same key name will be overwritten.
        Specified by:
        tableProperties in interface SnapshotTable
        Parameters:
        properties - a map of properties to be included
        Returns:
        this for method chaining

      • tableProperty

        public SnapshotTable tableProperty​(java.lang.String property,
                                   java.lang.String value)
        Description copied from interface: SnapshotTable
        Sets a table property in the newly created Iceberg table. Any properties with the same key name will be overwritten.
        Specified by:
        tableProperty in interface SnapshotTable
        Parameters:
        property - the key of the property to add
        value - the value of the property to add
        Returns:
        this for method chaining

      • destTableProps

        protected java.util.Map<java.lang.String,​java.lang.String> destTableProps()

      • checkSourceCatalog

        protected org.apache.spark.sql.connector.catalog.TableCatalog checkSourceCatalog​(org.apache.spark.sql.connector.catalog.CatalogPlugin catalog)

      • tableLocation

        public SnapshotTable tableLocation​(java.lang.String location)
        Description copied from interface: SnapshotTable
        Sets the table location for the newly created Iceberg table.
        Specified by:
        tableLocation in interface SnapshotTable
        Parameters:
        location - a table location
        Returns:
        this for method chaining

      • sourceTableLocation

        protected java.lang.String sourceTableLocation()

      • v1SourceTable

        protected org.apache.spark.sql.catalyst.catalog.CatalogTable v1SourceTable()

      • sourceCatalog

        protected org.apache.spark.sql.connector.catalog.TableCatalog sourceCatalog()

      • sourceTableIdent

        protected org.apache.spark.sql.connector.catalog.Identifier sourceTableIdent()

      • setProperties

        protected void setProperties​(java.util.Map<java.lang.String,​java.lang.String> properties)

      • setProperty

        protected void setProperty​(java.lang.String key,
                           java.lang.String value)

      • additionalProperties

        protected java.util.Map<java.lang.String,​java.lang.String> additionalProperties()

      • checkDestinationCatalog

        protected org.apache.spark.sql.connector.catalog.StagingTableCatalog checkDestinationCatalog​(org.apache.spark.sql.connector.catalog.CatalogPlugin catalog)

      • ensureNameMappingPresent

        protected void ensureNameMappingPresent​(Table table)

      • getMetadataLocation

        protected java.lang.String getMetadataLocation​(Table table)

      • spark

        protected org.apache.spark.sql.SparkSession spark()

      • sparkContext

        protected org.apache.spark.api.java.JavaSparkContext sparkContext()

      • option

        public ThisT option​(java.lang.String name,
                    java.lang.String value)
        Description copied from interface: Action
        Configures this action with an extra option.

        Certain actions allow users to control internal details of their execution via options.

        Specified by:
        option in interface Action<ThisT,​R>
        Parameters:
        name - an option name
        value - an option value
        Returns:
        this for method chaining

      • options

        public ThisT options​(java.util.Map<java.lang.String,​java.lang.String> newOptions)
        Description copied from interface: Action
        Configures this action with extra options.

        Certain actions allow users to control internal details of their execution via options.

        Specified by:
        options in interface Action<ThisT,​R>
        Parameters:
        newOptions - a map of extra options
        Returns:
        this for method chaining

      • options

        protected java.util.Map<java.lang.String,​java.lang.String> options()

      • withJobGroupInfo

        protected <T> T withJobGroupInfo​(JobGroupInfo info,
                                 java.util.function.Supplier<T> supplier)

      • newJobGroupInfo

        protected JobGroupInfo newJobGroupInfo​(java.lang.String groupId,
                                       java.lang.String desc)

      • buildValidDataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidDataFileDF​(Table table)

      • buildManifestFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestFileDF​(Table table)

      • buildManifestListDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestListDF​(Table table)

      • buildOtherMetadataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildOtherMetadataFileDF​(Table table)

      • buildValidMetadataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidMetadataFileDF​(Table table)

      • loadMetadataTable

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> loadMetadataTable​(Table table,
                                                                                   MetadataTableType type)