Class RewriteManifestsSparkAction

    • Method Detail

      • specId

        public RewriteManifestsSparkAction specId​(int specId)
        Description copied from interface: RewriteManifests
        Rewrites manifests for a given spec id.

        If not set, defaults to the table's default spec ID.

        Specified by:
        specId in interface RewriteManifests
        Parameters:
        specId - a spec id
        Returns:
        this for method chaining
      • rewriteIf

        public RewriteManifestsSparkAction rewriteIf​(java.util.function.Predicate<ManifestFile> newPredicate)
        Description copied from interface: RewriteManifests
        Rewrites only manifests that match the given predicate.

        If not set, all manifests will be rewritten.

        Specified by:
        rewriteIf in interface RewriteManifests
        Parameters:
        newPredicate - a predicate
        Returns:
        this for method chaining
      • stagingLocation

        public RewriteManifestsSparkAction stagingLocation​(java.lang.String newStagingLocation)
        Description copied from interface: RewriteManifests
        Passes a location where the staged manifests should be written.

        If not set, defaults to the table's metadata location.

        Specified by:
        stagingLocation in interface RewriteManifests
        Parameters:
        newStagingLocation - a staging location
        Returns:
        this for method chaining
      • snapshotProperty

        public ThisT snapshotProperty​(java.lang.String property,
                                      java.lang.String value)
      • spark

        protected org.apache.spark.sql.SparkSession spark()
      • sparkContext

        protected org.apache.spark.api.java.JavaSparkContext sparkContext()
      • option

        public ThisT option​(java.lang.String name,
                            java.lang.String value)
      • options

        public ThisT options​(java.util.Map<java.lang.String,​java.lang.String> newOptions)
      • options

        protected java.util.Map<java.lang.String,​java.lang.String> options()
      • withJobGroupInfo

        protected <T> T withJobGroupInfo​(JobGroupInfo info,
                                         java.util.function.Supplier<T> supplier)
      • newJobGroupInfo

        protected JobGroupInfo newJobGroupInfo​(java.lang.String groupId,
                                               java.lang.String desc)
      • buildValidContentFileWithTypeDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidContentFileWithTypeDF​(Table table)
      • buildValidContentFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidContentFileDF​(Table table)
      • buildManifestFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestFileDF​(Table table)
      • buildManifestListDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestListDF​(Table table)
      • buildOtherMetadataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildOtherMetadataFileDF​(Table table)
      • buildAllReachableOtherMetadataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildAllReachableOtherMetadataFileDF​(Table table)
      • buildValidMetadataFileDF

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidMetadataFileDF​(Table table)
      • withFileType

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> withFileType​(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> ds,
                                                                                      java.lang.String type)
      • loadMetadataTable

        protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> loadMetadataTable​(Table table,
                                                                                           MetadataTableType type)