Package org.apache.iceberg

Class SnapshotManager

java.lang.Object

org.apache.iceberg.SnapshotManager

All Implemented Interfaces:

ManageSnapshots, PendingUpdate<Snapshot>, SnapshotUpdate<ManageSnapshots>

public class SnapshotManager
extends java.lang.Object
implements ManageSnapshots

Method Summary

Modifier and Type	Method	Description
protected void	add(DataFile file)	Add a data file to the new snapshot.
protected void	add(DeleteFile file)	Add a delete file to the new snapshot.
protected void	add(ManifestFile manifest)	Add all files in a manifest to the new snapshot.
protected java.util.List<DataFile>	addedFiles()
Snapshot	apply()	Apply the pending changes and return the uncommitted changes for validation.
java.util.List<ManifestFile>	apply(TableMetadata base)	Apply the update's changes to the base table metadata and return the new manifest list.
ThisT	caseSensitive(boolean isCaseSensitive)
ManageSnapshots	cherrypick(long snapshotId)	Apply supported changes in given snapshot and create a new snapshot which will be set as the current snapshot on commit.
protected void	cleanAll()
protected void	cleanUncommitted(java.util.Set<ManifestFile> committed)	Clean up any uncommitted manifests that were created.
void	commit()	Apply the pending changes and commit.
protected TableMetadata	current()
protected PartitionSpec	dataSpec()
protected void	delete(java.lang.CharSequence path)	Add a specific data path to be deleted in the new snapshot.
protected void	delete(DataFile file)	Add a specific data file to be deleted in the new snapshot.
protected void	delete(DeleteFile file)	Add a specific delete file to be deleted in the new snapshot.
protected void	deleteByRowFilter(Expression expr)	Add a filter to match files to delete.
protected void	deleteFile(java.lang.String path)
ThisT	deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback)	Set a callback to delete files instead of the table's default.
protected void	dropPartition(int specId, StructLike partition)	Add a partition tuple to drop from the table during the delete phase.
protected void	failAnyDelete()
protected void	failMissingDeletePaths()
protected boolean	isCaseSensitive()
protected OutputFile	manifestListPath()
protected ManifestReader<DeleteFile>	newDeleteManifestReader(ManifestFile manifest)
protected ManifestWriter<DeleteFile>	newDeleteManifestWriter(PartitionSpec spec)
protected OutputFile	newManifestOutput()
protected ManifestReader<DataFile>	newManifestReader(ManifestFile manifest)
protected ManifestWriter<DataFile>	newManifestWriter(PartitionSpec spec)
protected java.lang.String	operation()	A string that describes the action that produced the new snapshot.
protected TableMetadata	refresh()
ManageSnapshots	rollbackTo(long snapshotId)	Rollback table's state to a specific Snapshot identified by id.
ManageSnapshots	rollbackToTime(long timestampMillis)	Roll this table's data back to the last Snapshot before the given timestamp.
protected Expression	rowFilter()
protected ManageSnapshots	self()
ThisT	set(java.lang.String property, java.lang.String value)	Set a summary property in the snapshot produced by this update.
ManageSnapshots	setCurrentSnapshot(long snapshotId)	Roll this table's data back to a specific Snapshot identified by id.
protected void	setNewFilesSequenceNumber(long sequenceNumber)
protected long	snapshotId()
ThisT	stageOnly()	Called to stage a snapshot in table metadata, but not update the current snapshot id.
protected java.util.Map<java.lang.String,java.lang.String>	summary()
java.lang.Object	updateEvent()	Generates update event to notify about metadata changes
protected void	validate(TableMetadata base)	Validate the current metadata.
protected void	validateAddedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression conflictDetectionFilter)	Validates that no files matching a filter have been added to the table since a starting snapshot.
protected void	validateDataFilesExist(TableMetadata base, java.lang.Long startingSnapshotId, CharSequenceSet requiredDataFiles, boolean skipDeletes, Expression conflictDetectionFilter)
protected void	validateNoNewDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter)	Validates that no delete files matching a filter have been added to the table since a starting snapshot.
protected void	validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, java.lang.Iterable<DataFile> dataFiles)	Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.
protected void	validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, java.lang.Iterable<DataFile> dataFiles)	Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.PendingUpdate

commit

Method Detail

self

protected ManageSnapshots self()

operation

protected java.lang.String operation()

A string that describes the action that produced the new snapshot.

Returns:

a string operation

cherrypick

public ManageSnapshots cherrypick(long snapshotId)

Description copied from interface: ManageSnapshots

Apply supported changes in given snapshot and create a new snapshot which will be set as the current snapshot on commit.

Specified by:

cherrypick in interface ManageSnapshots

Parameters:

snapshotId - a snapshotId whose changes to apply

Returns:

this for method chaining

setCurrentSnapshot

public ManageSnapshots setCurrentSnapshot(long snapshotId)

Description copied from interface: ManageSnapshots

Roll this table's data back to a specific Snapshot identified by id.

Specified by:

setCurrentSnapshot in interface ManageSnapshots

Parameters:

snapshotId - long id of the snapshot to roll back table data to

Returns:

this for method chaining

rollbackToTime

public ManageSnapshots rollbackToTime(long timestampMillis)

Description copied from interface: ManageSnapshots

Roll this table's data back to the last Snapshot before the given timestamp.

Specified by:

rollbackToTime in interface ManageSnapshots

Parameters:

timestampMillis - a long timestamp, as returned by System.currentTimeMillis()

Returns:

this for method chaining

rollbackTo

public ManageSnapshots rollbackTo(long snapshotId)

Description copied from interface: ManageSnapshots

Rollback table's state to a specific Snapshot identified by id.

Specified by:

rollbackTo in interface ManageSnapshots

Parameters:

snapshotId - long id of snapshot id to roll back table to. Must be an ancestor of the current snapshot

updateEvent

public java.lang.Object updateEvent()

Description copied from interface: PendingUpdate

Generates update event to notify about metadata changes

Specified by:

updateEvent in interface PendingUpdate<Snapshot>

Returns:

the generated event

validate

protected void validate(TableMetadata base)

Validate the current metadata.

Child operations can override this to add custom validation.

Parameters:

base - current table metadata to validate

apply

public Snapshot apply()

Description copied from interface: PendingUpdate

Apply the pending changes and return the uncommitted changes for validation.

This does not result in a permanent update.

Specified by:

apply in interface PendingUpdate<Snapshot>

Returns:

the uncommitted changes that would be committed by calling PendingUpdate.commit()

set

public ThisT set(java.lang.String property,
                 java.lang.String value)

Description copied from interface: SnapshotUpdate

Set a summary property in the snapshot produced by this update.

Parameters:

property - a String property name

value - a String property value

Returns:

this for method chaining

caseSensitive

public ThisT caseSensitive(boolean isCaseSensitive)

isCaseSensitive

protected boolean isCaseSensitive()

dataSpec

protected PartitionSpec dataSpec()

rowFilter

protected Expression rowFilter()

addedFiles

protected java.util.List<DataFile> addedFiles()

failAnyDelete

protected void failAnyDelete()

failMissingDeletePaths

protected void failMissingDeletePaths()

deleteByRowFilter

protected void deleteByRowFilter(Expression expr)

Add a filter to match files to delete. A file will be deleted if all of the rows it contains match this or any other filter passed to this method.

Parameters:

expr - an expression to match rows.

dropPartition

protected void dropPartition(int specId,
                             StructLike partition)

Add a partition tuple to drop from the table during the delete phase.

delete

protected void delete(DataFile file)

Add a specific data file to be deleted in the new snapshot.

delete

protected void delete(DeleteFile file)

Add a specific delete file to be deleted in the new snapshot.

delete

protected void delete(java.lang.CharSequence path)

Add a specific data path to be deleted in the new snapshot.

add

protected void add(DataFile file)

Add a data file to the new snapshot.

add

protected void add(DeleteFile file)

Add a delete file to the new snapshot.

add

protected void add(ManifestFile manifest)

Add all files in a manifest to the new snapshot.

validateAddedDataFiles

protected void validateAddedDataFiles(TableMetadata base,
                                      java.lang.Long startingSnapshotId,
                                      Expression conflictDetectionFilter)

Validates that no files matching a filter have been added to the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

conflictDetectionFilter - an expression used to find new conflicting data files

validateNoNewDeletesForDataFiles

protected void validateNoNewDeletesForDataFiles(TableMetadata base,
                                                java.lang.Long startingSnapshotId,
                                                java.lang.Iterable<DataFile> dataFiles)

Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

dataFiles - data files to validate have no new row deletes

validateNoNewDeletesForDataFiles

protected void validateNoNewDeletesForDataFiles(TableMetadata base,
                                                java.lang.Long startingSnapshotId,
                                                Expression dataFilter,
                                                java.lang.Iterable<DataFile> dataFiles)

Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

dataFilter - a data filter

dataFiles - data files to validate have no new row deletes

validateNoNewDeleteFiles

protected void validateNoNewDeleteFiles(TableMetadata base,
                                        java.lang.Long startingSnapshotId,
                                        Expression dataFilter)

Validates that no delete files matching a filter have been added to the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

dataFilter - an expression used to find new conflicting delete files

setNewFilesSequenceNumber

protected void setNewFilesSequenceNumber(long sequenceNumber)

validateDataFilesExist

protected void validateDataFilesExist(TableMetadata base,
                                      java.lang.Long startingSnapshotId,
                                      CharSequenceSet requiredDataFiles,
                                      boolean skipDeletes,
                                      Expression conflictDetectionFilter)

summary

protected java.util.Map<java.lang.String,java.lang.String> summary()

apply

public java.util.List<ManifestFile> apply(TableMetadata base)

Apply the update's changes to the base table metadata and return the new manifest list.

Parameters:

base - the base table metadata to apply changes to

Returns:

a manifest list for the new snapshot.

cleanUncommitted

protected void cleanUncommitted(java.util.Set<ManifestFile> committed)

Clean up any uncommitted manifests that were created.

Manifests may not be committed if apply is called more because a commit conflict has occurred. Implementations may keep around manifests because the same changes will be made by both apply calls. This method instructs the implementation to clean up those manifests and passes the paths of the manifests that were actually committed.

Parameters:

committed - a set of manifest paths that were actually committed

stageOnly

public ThisT stageOnly()

Description copied from interface: SnapshotUpdate

Called to stage a snapshot in table metadata, but not update the current snapshot id.

Specified by:

stageOnly in interface SnapshotUpdate<ThisT>

Returns:

this for method chaining

deleteWith

public ThisT deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback)

Description copied from interface: SnapshotUpdate

Set a callback to delete files instead of the table's default.

Specified by:

deleteWith in interface SnapshotUpdate<ThisT>

Parameters:

deleteCallback - a String consumer used to delete locations.

Returns:

this for method chaining

current

protected TableMetadata current()

refresh

protected TableMetadata refresh()

commit

public void commit()

Description copied from interface: PendingUpdate

Apply the pending changes and commit.

Changes are committed by calling the underlying table's commit method.

Once the commit is successful, the updated table will be refreshed.

Specified by:

commit in interface PendingUpdate<ThisT>

cleanAll

protected void cleanAll()

deleteFile

protected void deleteFile(java.lang.String path)

manifestListPath

protected OutputFile manifestListPath()

newManifestOutput

protected OutputFile newManifestOutput()

newManifestWriter

protected ManifestWriter<DataFile> newManifestWriter(PartitionSpec spec)

newDeleteManifestWriter

protected ManifestWriter<DeleteFile> newDeleteManifestWriter(PartitionSpec spec)

newManifestReader

protected ManifestReader<DataFile> newManifestReader(ManifestFile manifest)

newDeleteManifestReader

protected ManifestReader<DeleteFile> newDeleteManifestReader(ManifestFile manifest)

snapshotId

protected long snapshotId()