org.apache.iceberg

Class BaseOverwriteFiles

java.lang.Object

org.apache.iceberg.BaseOverwriteFiles

All Implemented Interfaces:

OverwriteFiles, PendingUpdate<Snapshot>, SnapshotUpdate<OverwriteFiles>

public class BaseOverwriteFiles
extends java.lang.Object
implements OverwriteFiles

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseOverwriteFiles(java.lang.String tableName, TableOperations ops)

Method Summary

Modifier and Type	Method and Description
protected void	add(DataFile file) Add a data file to the new snapshot.
protected void	add(DeleteFile file) Add a delete file to the new snapshot.
protected void	add(DeleteFile file, long dataSequenceNumber) Add a delete file to the new snapshot.
protected void	add(ManifestFile manifest) Add all files in a manifest to the new snapshot.
protected java.util.List<DataFile>	addedDataFiles()
protected org.apache.iceberg.DeleteFileIndex	addedDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, PartitionSet partitionSet, Snapshot parent) Returns matching delete files have been added to the table since a starting snapshot.
OverwriteFiles	addFile(DataFile file) Add a DataFile to the table.
protected boolean	addsDataFiles()
protected boolean	addsDeleteFiles()
Snapshot	apply() Apply the pending changes and return the uncommitted changes for validation.
java.util.List<ManifestFile>	apply(TableMetadata base, Snapshot snapshot) Apply the update's changes to the given metadata and snapshot.
protected boolean	canInheritSnapshotId()
ThisT	caseSensitive(boolean isCaseSensitive)
protected void	cleanAll()
protected void	cleanUncommitted(java.util.Set<ManifestFile> committed) Clean up any uncommitted manifests that were created.
void	commit() Apply the pending changes and commit.
protected CommitMetrics	commitMetrics()
OverwriteFiles	conflictDetectionFilter(Expression newConflictDetectionFilter) Sets a conflict detection filter used to validate concurrently added data and delete files.
protected TableMetadata	current()
protected PartitionSpec	dataSpec()
protected void	delete(java.lang.CharSequence path) Add a specific data path to be deleted in the new snapshot.
protected void	delete(DataFile file) Add a specific data file to be deleted in the new snapshot.
protected void	delete(DeleteFile file) Add a specific delete file to be deleted in the new snapshot.
protected void	deleteByRowFilter(Expression expr) Add a filter to match files to delete.
OverwriteFiles	deleteFile(DataFile file) Delete a DataFile from the table.
protected void	deleteFile(java.lang.String path)
protected boolean	deletesDataFiles()
protected boolean	deletesDeleteFiles()
ThisT	deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback) Set a callback to delete files instead of the table's default.
protected void	dropPartition(int specId, StructLike partition) Add a partition tuple to drop from the table during the delete phase.
protected void	failAnyDelete()
protected void	failMissingDeletePaths()
protected boolean	isCaseSensitive()
protected OutputFile	manifestListPath()
protected ManifestReader<DeleteFile>	newDeleteManifestReader(ManifestFile manifest)
protected ManifestWriter<DeleteFile>	newDeleteManifestWriter(PartitionSpec spec)
protected OutputFile	newManifestOutput()
protected ManifestReader<DataFile>	newManifestReader(ManifestFile manifest)
protected ManifestWriter<DataFile>	newManifestWriter(PartitionSpec spec)
protected RollingManifestWriter<DeleteFile>	newRollingDeleteManifestWriter(PartitionSpec spec)
protected RollingManifestWriter<DataFile>	newRollingManifestWriter(PartitionSpec spec)
protected java.lang.String	operation() A string that describes the action that produced the new snapshot.
OverwriteFiles	overwriteByRowFilter(Expression expr) Delete files that match an Expression on data rows from the table.
protected TableMetadata	refresh()
protected ThisT	reportWith(MetricsReporter newReporter)
protected Expression	rowFilter()
ThisT	scanManifestsWith(java.util.concurrent.ExecutorService executorService) Use a particular executor to scan manifests.
protected OverwriteFiles	self()
ThisT	set(java.lang.String property, java.lang.String value) Set a summary property in the snapshot produced by this update.
protected void	setNewDataFilesDataSequenceNumber(long sequenceNumber)
protected long	snapshotId()
ThisT	stageOnly() Called to stage a snapshot in table metadata, but not update the current snapshot id.
protected java.util.Map<java.lang.String,java.lang.String>	summary()
protected java.lang.String	targetBranch()
protected void	targetBranch(java.lang.String branch) A setter for the target branch on which snapshot producer operation should be performed
BaseOverwriteFiles	toBranch(java.lang.String branch) Perform operations on a particular branch
java.lang.Object	updateEvent() Generates update event to notify about metadata changes
protected void	validate(TableMetadata base, Snapshot parent) Validate the current metadata.
protected void	validateAddedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression conflictDetectionFilter, Snapshot parent) Validates that no files matching a filter have been added to the table since a starting snapshot.
protected void	validateAddedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent) Validates that no files matching given partitions have been added to the table since a starting snapshot.
OverwriteFiles	validateAddedFilesMatchOverwriteFilter() Signal that each file added to the table must match the overwrite expression.
protected void	validateDataFilesExist(TableMetadata base, java.lang.Long startingSnapshotId, CharSequenceSet requiredDataFiles, boolean skipDeletes, Expression conflictDetectionFilter, Snapshot parent)
protected void	validateDeletedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, Snapshot parent) Validates that no files matching a filter have been deleted from the table since a starting snapshot.
protected void	validateDeletedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent) Validates that no files matching a filter have been deleted from the table since a starting snapshot.
OverwriteFiles	validateFromSnapshot(long snapshotId) Set the snapshot ID used in any reads for this operation.
OverwriteFiles	validateNoConflictingData() Enables validation that data added concurrently does not conflict with this commit's operation.
OverwriteFiles	validateNoConflictingDeletes() Enables validation that deletes that happened concurrently do not conflict with this commit's operation.
protected void	validateNoNewDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, Snapshot parent) Validates that no delete files matching a filter have been added to the table since a starting snapshot.
protected void	validateNoNewDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent) Validates that no delete files matching a partition set have been added to the table since a starting snapshot.
protected void	validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, java.lang.Iterable<DataFile> dataFiles, Snapshot parent) Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.
protected void	validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, java.lang.Iterable<DataFile> dataFiles, Snapshot parent) Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.
protected java.util.concurrent.ExecutorService	workerPool()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.OverwriteFiles

caseSensitive

Methods inherited from interface org.apache.iceberg.SnapshotUpdate

deleteWith, scanManifestsWith, set, stageOnly

Methods inherited from interface org.apache.iceberg.PendingUpdate

apply, commit, updateEvent

Constructor Detail

BaseOverwriteFiles

protected BaseOverwriteFiles(java.lang.String tableName,
                             TableOperations ops)

Method Detail

self

protected OverwriteFiles self()

operation

protected java.lang.String operation()

A string that describes the action that produced the new snapshot.

Returns:

a string operation

overwriteByRowFilter

public OverwriteFiles overwriteByRowFilter(Expression expr)

Description copied from interface: OverwriteFiles

Delete files that match an Expression on data rows from the table.

A file is selected to be deleted by the expression if it could contain any rows that match the expression (candidate files are selected using an inclusive projection). These candidate files are deleted if all of the rows in the file must match the expression (the partition data matches the expression's Projections.strict(PartitionSpec) strict projection}). This guarantees that files are deleted if and only if all rows in the file must match the expression.

Files that may contain some rows that match the expression and some rows that do not will result in a ValidationException.

Specified by:

overwriteByRowFilter in interface OverwriteFiles

Parameters:

expr - an expression on rows in the table

Returns:

this for method chaining

addFile

public OverwriteFiles addFile(DataFile file)

Description copied from interface: OverwriteFiles

Add a DataFile to the table.

Specified by:

addFile in interface OverwriteFiles

Parameters:

file - a data file

Returns:

this for method chaining

deleteFile

public OverwriteFiles deleteFile(DataFile file)

Description copied from interface: OverwriteFiles

Delete a DataFile from the table.

Specified by:

deleteFile in interface OverwriteFiles

Parameters:

file - a data file

Returns:

this for method chaining

validateAddedFilesMatchOverwriteFilter

public OverwriteFiles validateAddedFilesMatchOverwriteFilter()

Description copied from interface: OverwriteFiles

Signal that each file added to the table must match the overwrite expression.

If this method is called, each added file is validated on commit to ensure that it matches the overwrite row filter. This is used to ensure that writes are idempotent: that files cannot be added during a commit that would not be removed if the operation were run a second time.

Specified by:

validateAddedFilesMatchOverwriteFilter in interface OverwriteFiles

Returns:

this for method chaining

validateFromSnapshot

public OverwriteFiles validateFromSnapshot(long snapshotId)

Description copied from interface: OverwriteFiles

Set the snapshot ID used in any reads for this operation.

Validations will check changes after this snapshot ID. If the from snapshot is not set, all ancestor snapshots through the table's initial snapshot are validated.

Specified by:

validateFromSnapshot in interface OverwriteFiles

Parameters:

snapshotId - a snapshot ID

Returns:

this for method chaining

conflictDetectionFilter

public OverwriteFiles conflictDetectionFilter(Expression newConflictDetectionFilter)

Description copied from interface: OverwriteFiles

Sets a conflict detection filter used to validate concurrently added data and delete files.

Specified by:

conflictDetectionFilter in interface OverwriteFiles

Parameters:

newConflictDetectionFilter - an expression on rows in the table

Returns:

this for method chaining

validateNoConflictingData

public OverwriteFiles validateNoConflictingData()

Description copied from interface: OverwriteFiles

Enables validation that data added concurrently does not conflict with this commit's operation.

This method should be called while committing non-idempotent overwrite operations. If a concurrent operation commits a new file after the data was read and that file might contain rows matching the specified conflict detection filter, the overwrite operation will detect this and fail.

Calling this method with a correct conflict detection filter is required to maintain isolation for non-idempotent overwrite operations.

Validation uses the conflict detection filter passed to OverwriteFiles.conflictDetectionFilter(Expression) and applies to operations that happened after the snapshot passed to OverwriteFiles.validateFromSnapshot(long). If the conflict detection filter is not set, any new data added concurrently will fail this overwrite operation.

Specified by:

validateNoConflictingData in interface OverwriteFiles

Returns:

this for method chaining

validateNoConflictingDeletes

public OverwriteFiles validateNoConflictingDeletes()

Description copied from interface: OverwriteFiles

Enables validation that deletes that happened concurrently do not conflict with this commit's operation.

Validating concurrent deletes is required during non-idempotent overwrite operations. If a concurrent operation deletes data in one of the files being overwritten, the overwrite operation must be aborted as it may undelete rows that were removed concurrently.

Calling this method with a correct conflict detection filter is required to maintain isolation for non-idempotent overwrite operations.

Validation uses the conflict detection filter passed to OverwriteFiles.conflictDetectionFilter(Expression) and applies to operations that happened after the snapshot passed to OverwriteFiles.validateFromSnapshot(long). If the conflict detection filter is not set, this operation will use the row filter provided in OverwriteFiles.overwriteByRowFilter(Expression) to check for new delete files and will ensure there are no conflicting deletes for data files removed via OverwriteFiles.deleteFile(DataFile).

Specified by:

validateNoConflictingDeletes in interface OverwriteFiles

Returns:

this for method chaining

toBranch

public BaseOverwriteFiles toBranch(java.lang.String branch)

Description copied from interface: SnapshotUpdate

Perform operations on a particular branch

Specified by:

toBranch in interface SnapshotUpdate<OverwriteFiles>

Parameters:

branch - which is name of SnapshotRef of type branch.

validate

protected void validate(TableMetadata base,
                        Snapshot parent)

Validate the current metadata.

Child operations can override this to add custom validation.

Parameters:

base - current table metadata to validate

parent - ending snapshot on the lineage which is being validated

set

public ThisT set(java.lang.String property,
                 java.lang.String value)

Description copied from interface: SnapshotUpdate

Set a summary property in the snapshot produced by this update.

Parameters:

property - a String property name

value - a String property value

Returns:

this for method chaining

caseSensitive

public ThisT caseSensitive(boolean isCaseSensitive)

isCaseSensitive

protected boolean isCaseSensitive()

dataSpec

protected PartitionSpec dataSpec()

rowFilter

protected Expression rowFilter()

addedDataFiles

protected java.util.List<DataFile> addedDataFiles()

failAnyDelete

protected void failAnyDelete()

failMissingDeletePaths

protected void failMissingDeletePaths()

deleteByRowFilter

protected void deleteByRowFilter(Expression expr)

Add a filter to match files to delete. A file will be deleted if all of the rows it contains match this or any other filter passed to this method.

Parameters:

expr - an expression to match rows.

dropPartition

protected void dropPartition(int specId,
                             StructLike partition)

Add a partition tuple to drop from the table during the delete phase.

delete

protected void delete(DataFile file)

Add a specific data file to be deleted in the new snapshot.

delete

protected void delete(DeleteFile file)

Add a specific delete file to be deleted in the new snapshot.

delete

protected void delete(java.lang.CharSequence path)

Add a specific data path to be deleted in the new snapshot.

deletesDataFiles

protected boolean deletesDataFiles()

deletesDeleteFiles

protected boolean deletesDeleteFiles()

addsDataFiles

protected boolean addsDataFiles()

addsDeleteFiles

protected boolean addsDeleteFiles()

add

protected void add(DataFile file)

Add a data file to the new snapshot.

add

protected void add(DeleteFile file)

Add a delete file to the new snapshot.

add

protected void add(DeleteFile file,
                   long dataSequenceNumber)

Add a delete file to the new snapshot.

add

protected void add(ManifestFile manifest)

Add all files in a manifest to the new snapshot.

validateAddedDataFiles

protected void validateAddedDataFiles(TableMetadata base,
                                      java.lang.Long startingSnapshotId,
                                      PartitionSet partitionSet,
                                      Snapshot parent)

Validates that no files matching given partitions have been added to the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

partitionSet - a set of partitions to filter new conflicting data files

parent - ending snapshot on the lineage being validated

validateAddedDataFiles

protected void validateAddedDataFiles(TableMetadata base,
                                      java.lang.Long startingSnapshotId,
                                      Expression conflictDetectionFilter,
                                      Snapshot parent)

Validates that no files matching a filter have been added to the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

conflictDetectionFilter - an expression used to find new conflicting data files

validateNoNewDeletesForDataFiles

protected void validateNoNewDeletesForDataFiles(TableMetadata base,
                                                java.lang.Long startingSnapshotId,
                                                java.lang.Iterable<DataFile> dataFiles,
                                                Snapshot parent)

Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

dataFiles - data files to validate have no new row deletes

parent - ending snapshot on the branch being validated

validateNoNewDeletesForDataFiles

protected void validateNoNewDeletesForDataFiles(TableMetadata base,
                                                java.lang.Long startingSnapshotId,
                                                Expression dataFilter,
                                                java.lang.Iterable<DataFile> dataFiles,
                                                Snapshot parent)

Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

dataFilter - a data filter

dataFiles - data files to validate have no new row deletes

parent - ending snapshot on the branch being validated

validateNoNewDeleteFiles

protected void validateNoNewDeleteFiles(TableMetadata base,
                                        java.lang.Long startingSnapshotId,
                                        Expression dataFilter,
                                        Snapshot parent)

Validates that no delete files matching a filter have been added to the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

dataFilter - an expression used to find new conflicting delete files

parent - ending snapshot on the branch being validated

validateNoNewDeleteFiles

protected void validateNoNewDeleteFiles(TableMetadata base,
                                        java.lang.Long startingSnapshotId,
                                        PartitionSet partitionSet,
                                        Snapshot parent)

Validates that no delete files matching a partition set have been added to the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

partitionSet - a partition set used to find new conflicting delete files

parent - ending snapshot on the branch being validated

addedDeleteFiles

protected org.apache.iceberg.DeleteFileIndex addedDeleteFiles(TableMetadata base,
                                                              java.lang.Long startingSnapshotId,
                                                              Expression dataFilter,
                                                              PartitionSet partitionSet,
                                                              Snapshot parent)

Returns matching delete files have been added to the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

dataFilter - an expression used to find delete files

partitionSet - a partition set used to find delete files

parent - parent snapshot of the branch

validateDeletedDataFiles

protected void validateDeletedDataFiles(TableMetadata base,
                                        java.lang.Long startingSnapshotId,
                                        Expression dataFilter,
                                        Snapshot parent)

Validates that no files matching a filter have been deleted from the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

dataFilter - an expression used to find deleted data files

parent - ending snapshot on the branch being validated

validateDeletedDataFiles

protected void validateDeletedDataFiles(TableMetadata base,
                                        java.lang.Long startingSnapshotId,
                                        PartitionSet partitionSet,
                                        Snapshot parent)

Validates that no files matching a filter have been deleted from the table since a starting snapshot.

Parameters:

base - table metadata to validate

startingSnapshotId - id of the snapshot current at the start of the operation

partitionSet - a partition set used to find deleted data files

parent - ending snapshot on the branch being validated

setNewDataFilesDataSequenceNumber

protected void setNewDataFilesDataSequenceNumber(long sequenceNumber)

validateDataFilesExist

protected void validateDataFilesExist(TableMetadata base,
                                      java.lang.Long startingSnapshotId,
                                      CharSequenceSet requiredDataFiles,
                                      boolean skipDeletes,
                                      Expression conflictDetectionFilter,
                                      Snapshot parent)

summary

protected java.util.Map<java.lang.String,java.lang.String> summary()

apply

public java.util.List<ManifestFile> apply(TableMetadata base,
                                          Snapshot snapshot)

Apply the update's changes to the given metadata and snapshot. Return the new manifest list.

Parameters:

base - the base table metadata to apply changes to

snapshot - snapshot to apply the changes to

Returns:

a manifest list for the new snapshot.

updateEvent

public java.lang.Object updateEvent()

Description copied from interface: PendingUpdate

Generates update event to notify about metadata changes

Returns:

the generated event

cleanUncommitted

protected void cleanUncommitted(java.util.Set<ManifestFile> committed)

Clean up any uncommitted manifests that were created.

Manifests may not be committed if apply is called more because a commit conflict has occurred. Implementations may keep around manifests because the same changes will be made by both apply calls. This method instructs the implementation to clean up those manifests and passes the paths of the manifests that were actually committed.

Parameters:

committed - a set of manifest paths that were actually committed

stageOnly

public ThisT stageOnly()

Description copied from interface: SnapshotUpdate

Called to stage a snapshot in table metadata, but not update the current snapshot id.

Specified by:

stageOnly in interface SnapshotUpdate<ThisT>

Returns:

this for method chaining

scanManifestsWith

public ThisT scanManifestsWith(java.util.concurrent.ExecutorService executorService)

Description copied from interface: SnapshotUpdate

Use a particular executor to scan manifests. The default worker pool will be used by default.

Specified by:

scanManifestsWith in interface SnapshotUpdate<ThisT>

Parameters:

executorService - the provided executor

Returns:

this for method chaining

commitMetrics

protected CommitMetrics commitMetrics()

reportWith

protected ThisT reportWith(MetricsReporter newReporter)

targetBranch

protected void targetBranch(java.lang.String branch)

A setter for the target branch on which snapshot producer operation should be performed

Parameters:

branch - to set as target branch

targetBranch

protected java.lang.String targetBranch()

workerPool

protected java.util.concurrent.ExecutorService workerPool()

deleteWith

public ThisT deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback)

Description copied from interface: SnapshotUpdate

Set a callback to delete files instead of the table's default.

Specified by:

deleteWith in interface SnapshotUpdate<ThisT>

Parameters:

deleteCallback - a String consumer used to delete locations.

Returns:

this for method chaining

apply

public Snapshot apply()

Description copied from interface: PendingUpdate

Apply the pending changes and return the uncommitted changes for validation.

This does not result in a permanent update.

Specified by:

apply in interface PendingUpdate<Snapshot>

Returns:

the uncommitted changes that would be committed by calling PendingUpdate.commit()

current

protected TableMetadata current()

refresh

protected TableMetadata refresh()

commit

public void commit()

Description copied from interface: PendingUpdate

Apply the pending changes and commit.

Changes are committed by calling the underlying table's commit method.

Once the commit is successful, the updated table will be refreshed.

Specified by:

commit in interface PendingUpdate<Snapshot>

cleanAll

protected void cleanAll()

deleteFile

protected void deleteFile(java.lang.String path)

manifestListPath

protected OutputFile manifestListPath()

newManifestOutput

protected OutputFile newManifestOutput()

newManifestWriter

protected ManifestWriter<DataFile> newManifestWriter(PartitionSpec spec)

newDeleteManifestWriter

protected ManifestWriter<DeleteFile> newDeleteManifestWriter(PartitionSpec spec)

newRollingManifestWriter

protected RollingManifestWriter<DataFile> newRollingManifestWriter(PartitionSpec spec)

newRollingDeleteManifestWriter

protected RollingManifestWriter<DeleteFile> newRollingDeleteManifestWriter(PartitionSpec spec)

newManifestReader

protected ManifestReader<DataFile> newManifestReader(ManifestFile manifest)

newDeleteManifestReader

protected ManifestReader<DeleteFile> newDeleteManifestReader(ManifestFile manifest)

snapshotId

protected long snapshotId()

canInheritSnapshotId

protected boolean canInheritSnapshotId()