org.apache.iceberg.actions

Class ExpireSnapshotsAction

java.lang.Object

org.apache.iceberg.actions.ExpireSnapshotsAction

All Implemented Interfaces:

Action<ExpireSnapshotsActionResult>

public class ExpireSnapshotsAction
extends java.lang.Object

An action which performs the same operation as ExpireSnapshots but uses Spark to determine the delta in files between the pre and post-expiration table metadata. All of the same restrictions of Remove Snapshots also apply to this action.

This implementation uses the metadata tables for the table being expired to list all Manifest and DataFiles. This is made into a Dataframe which are anti-joined with the same list read after the expiration. This operation will require a shuffle so parallelism can be controlled through spark.sql.shuffle.partitions. The expiration is done locally using a direct call to RemoveSnapshots. The snapshot expiration will be fully committed before any deletes are issued. Deletes are still performed locally after retrieving the results from the Spark executors.

Method Summary

Modifier and Type	Method and Description
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildManifestFileDF(org.apache.spark.sql.SparkSession spark, java.lang.String tableName)
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildManifestListDF(org.apache.spark.sql.SparkSession spark, java.lang.String metadataFileLocation)
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildManifestListDF(org.apache.spark.sql.SparkSession spark, Table table)
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildOtherMetadataFileDF(org.apache.spark.sql.SparkSession spark, TableOperations ops)
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildValidDataFileDF(org.apache.spark.sql.SparkSession spark)
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildValidDataFileDF(org.apache.spark.sql.SparkSession spark, java.lang.String tableName)
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	buildValidMetadataFileDF(org.apache.spark.sql.SparkSession spark, Table table, TableOperations ops)
ExpireSnapshotsAction	deleteWith(java.util.function.Consumer<java.lang.String> newDeleteFunc) The Consumer used on files which have been determined to be expired.
ExpireSnapshotsActionResult	execute() Executes this action.
ExpireSnapshotsAction	executeDeleteWith(java.util.concurrent.ExecutorService executorService) An executor service used when deleting files.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	expire() Expires snapshots and commits the changes to the table, returning a Dataset of files to delete.
ExpireSnapshotsAction	expireOlderThan(long timestampMillis) Expire all snapshots older than a given timestamp.
ExpireSnapshotsAction	expireSnapshotId(long expireSnapshotId) A specific snapshot to expire.
protected java.util.List<java.lang.String>	getManifestListPaths(java.lang.Iterable<Snapshot> snapshots) Returns all the path locations of all Manifest Lists for a given list of snapshots
protected java.util.List<java.lang.String>	getOtherMetadataFilePaths(TableOperations ops) Returns all Metadata file paths which may not be in the current metadata.
protected java.lang.String	metadataTableName(MetadataTableType type)
protected java.lang.String	metadataTableName(java.lang.String tableName, MetadataTableType type)
ExpireSnapshotsAction	retainLast(int numSnapshots) Retain at least x snapshots when expiring Identical to ExpireSnapshots.retainLast(int)
ExpireSnapshotsAction	streamDeleteResults(boolean stream) By default, all files to delete are brought to the driver at once which may be an issue with very long file lists.
protected Table	table()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

table

protected Table table()

streamDeleteResults

public ExpireSnapshotsAction streamDeleteResults(boolean stream)

By default, all files to delete are brought to the driver at once which may be an issue with very long file lists. Set this to true to use toLocalIterator if you are running into memory issues when collecting the list of files to be deleted.

Parameters:

stream - whether to use toLocalIterator to stream results instead of collect.

Returns:

this for method chaining

executeDeleteWith

public ExpireSnapshotsAction executeDeleteWith(java.util.concurrent.ExecutorService executorService)

An executor service used when deleting files. Only used during the local delete phase of this Spark action. Similar to ExpireSnapshots.executeDeleteWith(ExecutorService)

Parameters:

executorService - the service to use

Returns:

this for method chaining

expireSnapshotId

public ExpireSnapshotsAction expireSnapshotId(long expireSnapshotId)

A specific snapshot to expire. Identical to ExpireSnapshots.expireSnapshotId(long)

Parameters:

expireSnapshotId - Id of the snapshot to expire

Returns:

this for method chaining

expireOlderThan

public ExpireSnapshotsAction expireOlderThan(long timestampMillis)

Expire all snapshots older than a given timestamp. Identical to ExpireSnapshots.expireOlderThan(long)

Parameters:

timestampMillis - all snapshots before this time will be expired

Returns:

this for method chaining

retainLast

public ExpireSnapshotsAction retainLast(int numSnapshots)

Retain at least x snapshots when expiring Identical to ExpireSnapshots.retainLast(int)

Parameters:

numSnapshots - number of snapshots to leave

Returns:

this for method chaining

deleteWith

public ExpireSnapshotsAction deleteWith(java.util.function.Consumer<java.lang.String> newDeleteFunc)

The Consumer used on files which have been determined to be expired. By default uses a filesystem delete. Identical to ExpireSnapshots.deleteWith(Consumer)

Parameters:

newDeleteFunc - Consumer which takes a path and deletes it

Returns:

this for method chaining

expire

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> expire()

Expires snapshots and commits the changes to the table, returning a Dataset of files to delete.

This does not delete data files. To delete data files, run execute().

This may be called before or after execute() is called to return the expired file list.

Returns:

a Dataset of files that are no longer referenced by the table

execute

public ExpireSnapshotsActionResult execute()

Description copied from interface: Action

Executes this action.

Returns:

the result of this action

buildValidDataFileDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidDataFileDF(org.apache.spark.sql.SparkSession spark)

buildValidDataFileDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidDataFileDF(org.apache.spark.sql.SparkSession spark,
                                                                                      java.lang.String tableName)

buildManifestFileDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestFileDF(org.apache.spark.sql.SparkSession spark,
                                                                                     java.lang.String tableName)

buildManifestListDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestListDF(org.apache.spark.sql.SparkSession spark,
                                                                                     Table table)

buildManifestListDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildManifestListDF(org.apache.spark.sql.SparkSession spark,
                                                                                     java.lang.String metadataFileLocation)

buildOtherMetadataFileDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildOtherMetadataFileDF(org.apache.spark.sql.SparkSession spark,
                                                                                          TableOperations ops)

buildValidMetadataFileDF

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> buildValidMetadataFileDF(org.apache.spark.sql.SparkSession spark,
                                                                                          Table table,
                                                                                          TableOperations ops)

metadataTableName

protected java.lang.String metadataTableName(MetadataTableType type)

metadataTableName

protected java.lang.String metadataTableName(java.lang.String tableName,
                                             MetadataTableType type)

getManifestListPaths

protected java.util.List<java.lang.String> getManifestListPaths(java.lang.Iterable<Snapshot> snapshots)

Returns all the path locations of all Manifest Lists for a given list of snapshots

Parameters:

snapshots - snapshots

Returns:

the paths of the Manifest Lists

getOtherMetadataFilePaths

protected java.util.List<java.lang.String> getOtherMetadataFilePaths(TableOperations ops)

Returns all Metadata file paths which may not be in the current metadata. Specifically this includes "version-hint" files as well as entries in metadata.previousFiles.

Parameters:

ops - TableOperations for the table we will be getting paths from

Returns:

a list of paths to metadata files