org.apache.iceberg.spark.actions

Class BaseDeleteReachableFilesSparkAction

java.lang.Object

org.apache.iceberg.spark.actions.BaseDeleteReachableFilesSparkAction

All Implemented Interfaces:

Action<DeleteReachableFiles,DeleteReachableFiles.Result>, DeleteReachableFiles

public class BaseDeleteReachableFilesSparkAction
extends java.lang.Object
implements DeleteReachableFiles

An implementation of DeleteReachableFiles that uses metadata tables in Spark to determine which files should be deleted.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.iceberg.actions.DeleteReachableFiles

DeleteReachableFiles.Result

Constructor Summary

Constructors

Constructor and Description
BaseDeleteReachableFilesSparkAction(org.apache.spark.sql.SparkSession spark, java.lang.String metadataLocation)

Method Summary

Modifier and Type	Method and Description
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildManifestFileDF(Table table)
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildManifestListDF(Table table)
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildOtherMetadataFileDF(Table table)
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildValidDataFileDF(Table table)
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildValidMetadataFileDF(Table table)
DeleteReachableFiles	deleteWith(java.util.function.Consumer<java.lang.String> deleteFunc) Passes an alternative delete implementation that will be used for files.
DeleteReachableFiles.Result	execute() Executes this action.
DeleteReachableFiles	executeDeleteWith(java.util.concurrent.ExecutorService executorService) Passes an alternative executor service that will be used for files removal.
DeleteReachableFiles	io(FileIO fileIO) Set the FileIO to be used for files removal
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	loadMetadataTable(Table table, MetadataTableType type)
protected JobGroupInfo	newJobGroupInfo(java.lang.String groupId, java.lang.String desc)
protected Table	newStaticTable(TableMetadata metadata, FileIO io)
ThisT	option(java.lang.String name, java.lang.String value) Configures this action with an extra option.
protected java.util.Map<java.lang.String,java.lang.String>	options()
ThisT	options(java.util.Map<java.lang.String,java.lang.String> newOptions) Configures this action with extra options.
protected DeleteReachableFiles	self()
protected org.apache.spark.sql.SparkSession	spark()
protected org.apache.spark.api.java.JavaSparkContext	sparkContext()
protected <T> T	withJobGroupInfo(JobGroupInfo info, java.util.function.Supplier<T> supplier)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.actions.Action

option, options

Constructor Detail

BaseDeleteReachableFilesSparkAction

public BaseDeleteReachableFilesSparkAction(org.apache.spark.sql.SparkSession spark,
                                           java.lang.String metadataLocation)

Method Detail

self

protected DeleteReachableFiles self()

io

public DeleteReachableFiles io(FileIO fileIO)

Description copied from interface: DeleteReachableFiles

Set the FileIO to be used for files removal

Specified by:

io in interface DeleteReachableFiles

Parameters:

fileIO - FileIO to use for files removal

Returns:

this for method chaining

deleteWith

public DeleteReachableFiles deleteWith(java.util.function.Consumer<java.lang.String> deleteFunc)

Description copied from interface: DeleteReachableFiles

Passes an alternative delete implementation that will be used for files.

Specified by:

deleteWith in interface DeleteReachableFiles

Parameters:

deleteFunc - a function that will be called to delete files. The function accepts path to file as an argument.

Returns:

this for method chaining

executeDeleteWith

public DeleteReachableFiles executeDeleteWith(java.util.concurrent.ExecutorService executorService)

Description copied from interface: DeleteReachableFiles

Passes an alternative executor service that will be used for files removal.

If this method is not called, files will be deleted in the current thread.

Specified by:

executeDeleteWith in interface DeleteReachableFiles

Parameters:

executorService - the service to use

Returns:

this for method chaining

execute

public DeleteReachableFiles.Result execute()

Description copied from interface: Action

Executes this action.

Specified by:

execute in interface Action<DeleteReachableFiles,DeleteReachableFiles.Result>

Returns:

the result of this action

buildOtherMetadataFileDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildOtherMetadataFileDF(Table table)

spark

protected org.apache.spark.sql.SparkSession spark()

sparkContext

protected org.apache.spark.api.java.JavaSparkContext sparkContext()

option

public ThisT option(java.lang.String name,
                    java.lang.String value)

Description copied from interface: Action

Configures this action with an extra option.

Certain actions allow users to control internal details of their execution via options.

Specified by:

option in interface Action<ThisT,R>

Parameters:

name - an option name

value - an option value

Returns:

this for method chaining

options

public ThisT options(java.util.Map<java.lang.String,java.lang.String> newOptions)

Description copied from interface: Action

Configures this action with extra options.

Certain actions allow users to control internal details of their execution via options.

Specified by:

options in interface Action<ThisT,R>

Parameters:

newOptions - a map of extra options

Returns:

this for method chaining

options

protected java.util.Map<java.lang.String,java.lang.String> options()

withJobGroupInfo

protected <T> T withJobGroupInfo(JobGroupInfo info,
                                 java.util.function.Supplier<T> supplier)

newJobGroupInfo

protected JobGroupInfo newJobGroupInfo(java.lang.String groupId,
                                       java.lang.String desc)

newStaticTable

protected Table newStaticTable(TableMetadata metadata,
                               FileIO io)

buildValidDataFileDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidDataFileDF(Table table)

buildManifestFileDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestFileDF(Table table)

buildManifestListDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestListDF(Table table)

buildValidMetadataFileDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidMetadataFileDF(Table table)

loadMetadataTable

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> loadMetadataTable(Table table,
                                                                                   MetadataTableType type)