Class BaseRewriteDataFilesSpark3Action

    • Constructor Detail

      • BaseRewriteDataFilesSpark3Action

        protected BaseRewriteDataFilesSpark3Action​(org.apache.spark.sql.SparkSession spark,
                                                   Table table)
    • Method Detail

      • table

        protected Table table()
      • sort

        public RewriteDataFiles sort​(SortOrder sortOrder)
        Description copied from interface: RewriteDataFiles
        Choose SORT as a strategy for this rewrite operation and manually specify the sortOrder to use
        Specified by:
        sort in interface RewriteDataFiles
        Parameters:
        sortOrder - user defined sortOrder
        Returns:
        this for method chaining
      • filter

        public RewriteDataFiles filter​(Expression expression)
        Description copied from interface: RewriteDataFiles
        A user provided filter for determining which files will be considered by the rewrite strategy. This will be used in addition to whatever rules the rewrite strategy generates. For example this would be used for providing a restriction to only run rewrite on a specific partition.
        Specified by:
        filter in interface RewriteDataFiles
        Parameters:
        expression - An iceberg expression used to determine which files will be considered for rewriting
        Returns:
        this for chaining
      • snapshotProperty

        public ThisT snapshotProperty​(java.lang.String property,
                                      java.lang.String value)
        Description copied from interface: SnapshotUpdate
        Sets a summary property in the snapshot produced by this action.
        Specified by:
        snapshotProperty in interface SnapshotUpdate<ThisT,​R>
        Parameters:
        property - a snapshot property name
        value - a snapshot property value
        Returns:
        this for method chaining
      • spark

        protected org.apache.spark.sql.SparkSession spark()
      • sparkContext

        protected org.apache.spark.api.java.JavaSparkContext sparkContext()
      • option

        public ThisT option​(java.lang.String name,
                            java.lang.String value)
        Description copied from interface: Action
        Configures this action with an extra option.

        Certain actions allow users to control internal details of their execution via options.

        Specified by:
        option in interface Action<ThisT,​R>
        Parameters:
        name - an option name
        value - an option value
        Returns:
        this for method chaining
      • options

        public ThisT options​(java.util.Map<java.lang.String,​java.lang.String> newOptions)
        Description copied from interface: Action
        Configures this action with extra options.

        Certain actions allow users to control internal details of their execution via options.

        Specified by:
        options in interface Action<ThisT,​R>
        Parameters:
        newOptions - a map of extra options
        Returns:
        this for method chaining
      • options

        protected java.util.Map<java.lang.String,​java.lang.String> options()
      • withJobGroupInfo

        protected <T> T withJobGroupInfo​(JobGroupInfo info,
                                         java.util.function.Supplier<T> supplier)
      • newJobGroupInfo

        protected JobGroupInfo newJobGroupInfo​(java.lang.String groupId,
                                               java.lang.String desc)
      • buildValidDataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidDataFileDF​(Table table)
      • buildManifestFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestFileDF​(Table table)
      • buildManifestListDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestListDF​(Table table)
      • buildOtherMetadataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildOtherMetadataFileDF​(Table table)
      • buildValidMetadataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidMetadataFileDF​(Table table)
      • loadMetadataTable

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> loadMetadataTable​(Table table,
                                                                                           MetadataTableType type)