Package org.apache.iceberg.spark.actions

Class DeleteReachableFilesSparkAction

java.lang.Object

org.apache.iceberg.spark.actions.DeleteReachableFilesSparkAction

All Implemented Interfaces:

Action<DeleteReachableFiles,DeleteReachableFiles.Result>, DeleteReachableFiles

public class DeleteReachableFilesSparkAction
extends Object
implements DeleteReachableFiles

An implementation of DeleteReachableFiles that uses metadata tables in Spark to determine which files should be deleted.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.iceberg.actions.DeleteReachableFiles

DeleteReachableFiles.Result

Field Summary

Fields

Modifier and Type	Field	Description
protected static final org.apache.iceberg.relocated.com.google.common.base.Joiner	COMMA_JOINER
protected static final org.apache.iceberg.relocated.com.google.common.base.Splitter	COMMA_SPLITTER
protected static final String	FILE_PATH
protected static final String	LAST_MODIFIED
protected static final String	MANIFEST
protected static final String	MANIFEST_LIST
protected static final String	OTHERS
protected static final String	STATISTICS_FILES
static final String	STREAM_RESULTS
static final boolean	STREAM_RESULTS_DEFAULT

Method Summary

Modifier and Type	Method	Description
protected org.apache.spark.sql.Dataset<FileInfo>	allReachableOtherMetadataFileDS(Table table)
protected org.apache.spark.sql.Dataset<FileInfo>	contentFileDS(Table table)
protected org.apache.spark.sql.Dataset<FileInfo>	contentFileDS(Table table, Set<Long> snapshotIds)
protected org.apache.iceberg.spark.actions.BaseSparkAction.DeleteSummary	deleteFiles(ExecutorService executorService, Consumer<String> deleteFunc, Iterator<FileInfo> files)	Deletes files and keeps track of how many files were removed for each file type.
protected org.apache.iceberg.spark.actions.BaseSparkAction.DeleteSummary	deleteFiles(SupportsBulkOperations io, Iterator<FileInfo> files)
DeleteReachableFilesSparkAction	deleteWith(Consumer<String> newDeleteFunc)	Passes an alternative delete implementation that will be used for files.
DeleteReachableFiles.Result	execute()	Executes this action.
DeleteReachableFilesSparkAction	executeDeleteWith(ExecutorService executorService)	Passes an alternative executor service that will be used for files removal.
DeleteReachableFilesSparkAction	io(FileIO fileIO)	Set the FileIO to be used for files removal
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	loadMetadataTable(Table table, MetadataTableType type)
protected org.apache.spark.sql.Dataset<FileInfo>	manifestDS(Table table)
protected org.apache.spark.sql.Dataset<FileInfo>	manifestDS(Table table, Set<Long> snapshotIds)
protected org.apache.spark.sql.Dataset<FileInfo>	manifestListDS(Table table)
protected org.apache.spark.sql.Dataset<FileInfo>	manifestListDS(Table table, Set<Long> snapshotIds)
protected JobGroupInfo	newJobGroupInfo(String groupId, String desc)
protected Table	newStaticTable(TableMetadata metadata, FileIO io)
DeleteReachableFilesSparkAction	option(String name, String value)
protected Map<String,String>	options()
DeleteReachableFilesSparkAction	options(Map<String,String> newOptions)
protected org.apache.spark.sql.Dataset<FileInfo>	otherMetadataFileDS(Table table)
protected DeleteReachableFilesSparkAction	self()
protected org.apache.spark.sql.SparkSession	spark()
protected org.apache.spark.api.java.JavaSparkContext	sparkContext()
protected org.apache.spark.sql.Dataset<FileInfo>	statisticsFileDS(Table table, Set<Long> snapshotIds)
protected <T> T	withJobGroupInfo(JobGroupInfo info, Supplier<T> supplier)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.actions.Action

option, options

Field Details

STREAM_RESULTS

public static final String STREAM_RESULTS

See Also:

• Constant Field Values

STREAM_RESULTS_DEFAULT

public static final boolean STREAM_RESULTS_DEFAULT

See Also:

• Constant Field Values

MANIFEST

protected static final String MANIFEST

See Also:

• Constant Field Values

MANIFEST_LIST

protected static final String MANIFEST_LIST

See Also:

• Constant Field Values

STATISTICS_FILES

protected static final String STATISTICS_FILES

See Also:

• Constant Field Values

OTHERS

protected static final String OTHERS

See Also:

• Constant Field Values

FILE_PATH

protected static final String FILE_PATH

See Also:

• Constant Field Values

LAST_MODIFIED

protected static final String LAST_MODIFIED

See Also:

• Constant Field Values

COMMA_SPLITTER

protected static final org.apache.iceberg.relocated.com.google.common.base.Splitter COMMA_SPLITTER

COMMA_JOINER

protected static final org.apache.iceberg.relocated.com.google.common.base.Joiner COMMA_JOINER

Method Details

self

protected DeleteReachableFilesSparkAction self()

io

public DeleteReachableFilesSparkAction io(FileIO fileIO)

Description copied from interface: DeleteReachableFiles

Set the FileIO to be used for files removal

Specified by:

io in interface DeleteReachableFiles

Parameters:

fileIO - FileIO to use for files removal

Returns:

this for method chaining

deleteWith

public DeleteReachableFilesSparkAction deleteWith(Consumer<String> newDeleteFunc)

Description copied from interface: DeleteReachableFiles

Passes an alternative delete implementation that will be used for files.

Specified by:

deleteWith in interface DeleteReachableFiles

Parameters:

newDeleteFunc - a function that will be called to delete files. The function accepts path to file as an argument.

Returns:

this for method chaining

executeDeleteWith

public DeleteReachableFilesSparkAction executeDeleteWith(ExecutorService executorService)

Description copied from interface: DeleteReachableFiles

Passes an alternative executor service that will be used for files removal. This service will only be used if a custom delete function is provided by DeleteReachableFiles.deleteWith(Consumer) or if the FileIO does not support bulk deletes. Otherwise, parallelism should be controlled by the IO specific deleteFiles method.

Specified by:

executeDeleteWith in interface DeleteReachableFiles

Parameters:

executorService - the service to use

Returns:

this for method chaining

execute

public DeleteReachableFiles.Result execute()

Description copied from interface: Action

Executes this action.

Specified by:

execute in interface Action<DeleteReachableFiles,DeleteReachableFiles.Result>

Returns:

the result of this action

spark

protected org.apache.spark.sql.SparkSession spark()

sparkContext

protected org.apache.spark.api.java.JavaSparkContext sparkContext()

option

public DeleteReachableFilesSparkAction option(String name,
 String value)

options

public DeleteReachableFilesSparkAction options(Map<String,String> newOptions)

options

protected Map<String,String> options()

withJobGroupInfo

protected <T> T withJobGroupInfo(JobGroupInfo info,
 Supplier<T> supplier)

newJobGroupInfo

protected JobGroupInfo newJobGroupInfo(String groupId,
 String desc)

newStaticTable

protected Table newStaticTable(TableMetadata metadata,
 FileIO io)

contentFileDS

protected org.apache.spark.sql.Dataset<FileInfo> contentFileDS(Table table)

contentFileDS

protected org.apache.spark.sql.Dataset<FileInfo> contentFileDS(Table table,
 Set<Long> snapshotIds)

manifestDS

protected org.apache.spark.sql.Dataset<FileInfo> manifestDS(Table table)

manifestDS

protected org.apache.spark.sql.Dataset<FileInfo> manifestDS(Table table,
 Set<Long> snapshotIds)

manifestListDS

protected org.apache.spark.sql.Dataset<FileInfo> manifestListDS(Table table)

manifestListDS

protected org.apache.spark.sql.Dataset<FileInfo> manifestListDS(Table table,
 Set<Long> snapshotIds)

statisticsFileDS

protected org.apache.spark.sql.Dataset<FileInfo> statisticsFileDS(Table table,
 Set<Long> snapshotIds)

otherMetadataFileDS

protected org.apache.spark.sql.Dataset<FileInfo> otherMetadataFileDS(Table table)

allReachableOtherMetadataFileDS

protected org.apache.spark.sql.Dataset<FileInfo> allReachableOtherMetadataFileDS(Table table)

loadMetadataTable

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> loadMetadataTable(Table table,
 MetadataTableType type)

deleteFiles

protected org.apache.iceberg.spark.actions.BaseSparkAction.DeleteSummary deleteFiles(ExecutorService executorService,
 Consumer<String> deleteFunc,
 Iterator<FileInfo> files)

Deletes files and keeps track of how many files were removed for each file type.

Parameters:

executorService - an executor service to use for parallel deletes

deleteFunc - a delete func

files - an iterator of Spark rows of the structure (path: String, type: String)

Returns:

stats on which files were deleted

deleteFiles

protected org.apache.iceberg.spark.actions.BaseSparkAction.DeleteSummary deleteFiles(SupportsBulkOperations io,
 Iterator<FileInfo> files)