Class StreamingDelete
- java.lang.Object
-
- org.apache.iceberg.StreamingDelete
-
- All Implemented Interfaces:
DeleteFiles
,PendingUpdate<Snapshot>
,SnapshotUpdate<DeleteFiles>
public class StreamingDelete extends java.lang.Object implements DeleteFiles
Delete
implementation that avoids loading full manifests in memory.This implementation will attempt to commit 5 times before throwing
CommitFailedException
.
-
-
Constructor Summary
Constructors Modifier Constructor Description protected
StreamingDelete(java.lang.String tableName, TableOperations ops)
-
Method Summary
All Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description protected void
add(DataFile file)
Add a data file to the new snapshot.protected void
add(DeleteFile file)
Add a delete file to the new snapshot.protected void
add(ManifestFile manifest)
Add all files in a manifest to the new snapshot.protected org.apache.iceberg.DeleteFileIndex
addedDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, PartitionSet partitionSet)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.addedDeleteFiles(TableMetadata, Long, Expression, PartitionSet, Snapshot)
insteadprotected org.apache.iceberg.DeleteFileIndex
addedDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, PartitionSet partitionSet, Snapshot parent)
Returns matching delete files have been added to the table since a starting snapshot.protected java.util.List<DataFile>
addedFiles()
Snapshot
apply()
Apply the pending changes and return the uncommitted changes for validation.java.util.List<ManifestFile>
apply(TableMetadata base, Snapshot snapshot)
Apply the update's changes to the given metadata and snapshot.ThisT
caseSensitive(boolean isCaseSensitive)
protected void
cleanAll()
protected void
cleanUncommitted(java.util.Set<ManifestFile> committed)
Clean up any uncommitted manifests that were created.void
commit()
Apply the pending changes and commit.protected CommitMetrics
commitMetrics()
protected TableMetadata
current()
protected PartitionSpec
dataSpec()
protected void
delete(java.lang.CharSequence path)
Add a specific data path to be deleted in the new snapshot.protected void
delete(DataFile file)
Add a specific data file to be deleted in the new snapshot.protected void
delete(DeleteFile file)
Add a specific delete file to be deleted in the new snapshot.protected void
deleteByRowFilter(Expression expr)
Add a filter to match files to delete.StreamingDelete
deleteFile(java.lang.CharSequence path)
Delete a file path from the underlying table.protected void
deleteFile(java.lang.String path)
StreamingDelete
deleteFile(DataFile file)
Delete a file tracked by aDataFile
from the underlying table.StreamingDelete
deleteFromRowFilter(Expression expr)
Delete files that match anExpression
on data rows from the table.ThisT
deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback)
Set a callback to delete files instead of the table's default.protected void
dropPartition(int specId, StructLike partition)
Add a partition tuple to drop from the table during the delete phase.protected void
failAnyDelete()
protected void
failMissingDeletePaths()
protected boolean
isCaseSensitive()
protected OutputFile
manifestListPath()
protected ManifestReader<DeleteFile>
newDeleteManifestReader(ManifestFile manifest)
protected ManifestWriter<DeleteFile>
newDeleteManifestWriter(PartitionSpec spec)
protected OutputFile
newManifestOutput()
protected ManifestReader<DataFile>
newManifestReader(ManifestFile manifest)
protected ManifestWriter<DataFile>
newManifestWriter(PartitionSpec spec)
protected java.lang.String
operation()
A string that describes the action that produced the new snapshot.protected TableMetadata
refresh()
protected ThisT
reportWith(MetricsReporter newReporter)
protected Expression
rowFilter()
ThisT
scanManifestsWith(java.util.concurrent.ExecutorService executorService)
Use a particular executor to scan manifests.protected DeleteFiles
self()
ThisT
set(java.lang.String property, java.lang.String value)
Set a summary property in the snapshot produced by this update.protected void
setNewFilesSequenceNumber(long sequenceNumber)
protected long
snapshotId()
ThisT
stageOnly()
Called to stage a snapshot in table metadata, but not update the current snapshot id.protected java.util.Map<java.lang.String,java.lang.String>
summary()
protected java.lang.String
targetBranch()
protected void
targetBranch(java.lang.String branch)
* A setter for the target branch on which snapshot producer operation should be performedStreamingDelete
toBranch(java.lang.String branch)
Perform operations on a particular branchjava.lang.Object
updateEvent()
Generates update event to notify about metadata changesprotected void
validate(TableMetadata currentMetadata, Snapshot snapshot)
Validate the current metadata.protected void
validateAddedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression conflictDetectionFilter)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateAddedDataFiles(TableMetadata, Long, Expression, Snapshot)
insteadprotected void
validateAddedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression conflictDetectionFilter, Snapshot parent)
Validates that no files matching a filter have been added to the table since a starting snapshot.protected void
validateAddedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateAddedDataFiles(TableMetadata, Long, PartitionSet, Snapshot)
insteadprotected void
validateAddedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent)
Validates that no files matching given partitions have been added to the table since a starting snapshot.protected void
validateDataFilesExist(TableMetadata base, java.lang.Long startingSnapshotId, CharSequenceSet requiredDataFiles, boolean skipDeletes, Expression conflictDetectionFilter)
Deprecated.protected void
validateDataFilesExist(TableMetadata base, java.lang.Long startingSnapshotId, CharSequenceSet requiredDataFiles, boolean skipDeletes, Expression conflictDetectionFilter, Snapshot parent)
protected void
validateDeletedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateDeletedDataFiles(TableMetadata, Long, Expression, Snapshot)
insteadprotected void
validateDeletedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, Snapshot parent)
Validates that no files matching a filter have been deleted from the table since a starting snapshot.protected void
validateDeletedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateNoNewDeleteFiles(TableMetadata, Long, PartitionSet, Snapshot)
insteadprotected void
validateDeletedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent)
Validates that no files matching a filter have been deleted from the table since a starting snapshot.protected void
validateNoNewDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateNoNewDeleteFiles(org.apache.iceberg.TableMetadata, java.lang.Long, org.apache.iceberg.expressions.Expression)
insteadprotected void
validateNoNewDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, Snapshot parent)
Validates that no delete files matching a filter have been added to the table since a starting snapshot.protected void
validateNoNewDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateNoNewDeleteFiles(TableMetadata, Long, PartitionSet, Snapshot)
insteadprotected void
validateNoNewDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent)
Validates that no delete files matching a partition set have been added to the table since a starting snapshot.protected void
validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, java.lang.Iterable<DataFile> dataFiles)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateNoNewDeletesForDataFiles(org.apache.iceberg.TableMetadata, java.lang.Long, java.lang.Iterable<org.apache.iceberg.DataFile>, org.apache.iceberg.Snapshot)
insteadprotected void
validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, java.lang.Iterable<DataFile> dataFiles, Snapshot parent)
Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.protected void
validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, java.lang.Iterable<DataFile> dataFiles)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateNoNewDeletesForDataFiles(org.apache.iceberg.TableMetadata, java.lang.Long, java.lang.Iterable<org.apache.iceberg.DataFile>, org.apache.iceberg.Snapshot)
insteadprotected void
validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, java.lang.Iterable<DataFile> dataFiles, Snapshot parent)
Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.protected java.util.concurrent.ExecutorService
workerPool()
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.iceberg.DeleteFiles
caseSensitive
-
Methods inherited from interface org.apache.iceberg.PendingUpdate
apply, commit, updateEvent
-
Methods inherited from interface org.apache.iceberg.SnapshotUpdate
deleteWith, scanManifestsWith, set, stageOnly
-
-
-
-
Constructor Detail
-
StreamingDelete
protected StreamingDelete(java.lang.String tableName, TableOperations ops)
-
-
Method Detail
-
self
protected DeleteFiles self()
-
operation
protected java.lang.String operation()
A string that describes the action that produced the new snapshot.- Returns:
- a string operation
-
deleteFile
public StreamingDelete deleteFile(java.lang.CharSequence path)
Description copied from interface:DeleteFiles
Delete a file path from the underlying table.To remove a file from the table, this path must equal a path in the table's metadata. Paths that are different but equivalent will not be removed. For example, file:/path/file.avro is equivalent to file:///path/file.avro, but would not remove the latter path from the table.
- Specified by:
deleteFile
in interfaceDeleteFiles
- Parameters:
path
- a fully-qualified file path to remove from the table- Returns:
- this for method chaining
-
deleteFile
public StreamingDelete deleteFile(DataFile file)
Description copied from interface:DeleteFiles
Delete a file tracked by aDataFile
from the underlying table.- Specified by:
deleteFile
in interfaceDeleteFiles
- Parameters:
file
- a DataFile to remove from the table- Returns:
- this for method chaining
-
deleteFromRowFilter
public StreamingDelete deleteFromRowFilter(Expression expr)
Description copied from interface:DeleteFiles
Delete files that match anExpression
on data rows from the table.A file is selected to be deleted by the expression if it could contain any rows that match the expression (candidate files are selected using an
inclusive projection
). These candidate files are deleted if all of the rows in the file must match the expression (the partition data matches the expression'sProjections.strict(PartitionSpec)
strict projection}). This guarantees that files are deleted if and only if all rows in the file must match the expression.Files that may contain some rows that match the expression and some rows that do not will result in a
ValidationException
.- Specified by:
deleteFromRowFilter
in interfaceDeleteFiles
- Parameters:
expr
- an expression on rows in the table- Returns:
- this for method chaining
-
toBranch
public StreamingDelete toBranch(java.lang.String branch)
Description copied from interface:SnapshotUpdate
Perform operations on a particular branch- Specified by:
toBranch
in interfaceSnapshotUpdate<DeleteFiles>
- Parameters:
branch
- which is name of SnapshotRef of type branch.
-
set
public ThisT set(java.lang.String property, java.lang.String value)
Description copied from interface:SnapshotUpdate
Set a summary property in the snapshot produced by this update.- Parameters:
property
- a String property namevalue
- a String property value- Returns:
- this for method chaining
-
caseSensitive
public ThisT caseSensitive(boolean isCaseSensitive)
-
isCaseSensitive
protected boolean isCaseSensitive()
-
dataSpec
protected PartitionSpec dataSpec()
-
rowFilter
protected Expression rowFilter()
-
addedFiles
protected java.util.List<DataFile> addedFiles()
-
failAnyDelete
protected void failAnyDelete()
-
failMissingDeletePaths
protected void failMissingDeletePaths()
-
deleteByRowFilter
protected void deleteByRowFilter(Expression expr)
Add a filter to match files to delete. A file will be deleted if all of the rows it contains match this or any other filter passed to this method.- Parameters:
expr
- an expression to match rows.
-
dropPartition
protected void dropPartition(int specId, StructLike partition)
Add a partition tuple to drop from the table during the delete phase.
-
delete
protected void delete(DataFile file)
Add a specific data file to be deleted in the new snapshot.
-
delete
protected void delete(DeleteFile file)
Add a specific delete file to be deleted in the new snapshot.
-
delete
protected void delete(java.lang.CharSequence path)
Add a specific data path to be deleted in the new snapshot.
-
add
protected void add(DataFile file)
Add a data file to the new snapshot.
-
add
protected void add(DeleteFile file)
Add a delete file to the new snapshot.
-
add
protected void add(ManifestFile manifest)
Add all files in a manifest to the new snapshot.
-
validateAddedDataFiles
@Deprecated protected void validateAddedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateAddedDataFiles(TableMetadata, Long, PartitionSet, Snapshot)
instead- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationpartitionSet
- a set of partitions to filter new conflicting data files
-
validateAddedDataFiles
protected void validateAddedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent)
Validates that no files matching given partitions have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationpartitionSet
- a set of partitions to filter new conflicting data filesparent
- ending snapshot on the lineage being validated
-
validateAddedDataFiles
@Deprecated protected void validateAddedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression conflictDetectionFilter)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateAddedDataFiles(TableMetadata, Long, Expression, Snapshot)
instead- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationconflictDetectionFilter
- an expression used to find new conflicting data files
-
validateAddedDataFiles
protected void validateAddedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression conflictDetectionFilter, Snapshot parent)
Validates that no files matching a filter have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationconflictDetectionFilter
- an expression used to find new conflicting data files
-
validateNoNewDeletesForDataFiles
protected void validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, java.lang.Iterable<DataFile> dataFiles, Snapshot parent)
Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationdataFiles
- data files to validate have no new row deletesparent
- ending snapshot on the branch being validated
-
validateNoNewDeletesForDataFiles
@Deprecated protected void validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, java.lang.Iterable<DataFile> dataFiles)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateNoNewDeletesForDataFiles(org.apache.iceberg.TableMetadata, java.lang.Long, java.lang.Iterable<org.apache.iceberg.DataFile>, org.apache.iceberg.Snapshot)
insteadValidates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationdataFiles
- data files to validate have no new row deletes
-
validateNoNewDeletesForDataFiles
@Deprecated protected void validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, java.lang.Iterable<DataFile> dataFiles)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateNoNewDeletesForDataFiles(org.apache.iceberg.TableMetadata, java.lang.Long, java.lang.Iterable<org.apache.iceberg.DataFile>, org.apache.iceberg.Snapshot)
insteadValidates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationdataFilter
- a data filterdataFiles
- data files to validate have no new row deletes
-
validateNoNewDeletesForDataFiles
protected void validateNoNewDeletesForDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, java.lang.Iterable<DataFile> dataFiles, Snapshot parent)
Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationdataFilter
- a data filterdataFiles
- data files to validate have no new row deletesparent
- ending snapshot on the branch being validated
-
validateNoNewDeleteFiles
@Deprecated protected void validateNoNewDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateNoNewDeleteFiles(org.apache.iceberg.TableMetadata, java.lang.Long, org.apache.iceberg.expressions.Expression)
insteadValidates that no delete files matching a filter have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationdataFilter
- an expression used to find new conflicting delete files
-
validateNoNewDeleteFiles
protected void validateNoNewDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, Snapshot parent)
Validates that no delete files matching a filter have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationdataFilter
- an expression used to find new conflicting delete filesparent
- ending snapshot on the branch being validated
-
validateNoNewDeleteFiles
@Deprecated protected void validateNoNewDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateNoNewDeleteFiles(TableMetadata, Long, PartitionSet, Snapshot)
insteadValidates that no delete files matching a partition set have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationpartitionSet
- a partition set used to find new conflicting delete files
-
validateNoNewDeleteFiles
protected void validateNoNewDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent)
Validates that no delete files matching a partition set have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationpartitionSet
- a partition set used to find new conflicting delete filesparent
- ending snapshot on the branch being validated
-
addedDeleteFiles
@Deprecated protected org.apache.iceberg.DeleteFileIndex addedDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, PartitionSet partitionSet)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.addedDeleteFiles(TableMetadata, Long, Expression, PartitionSet, Snapshot)
insteadReturns matching delete files have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationdataFilter
- an expression used to find delete filespartitionSet
- a partition set used to find delete files
-
addedDeleteFiles
protected org.apache.iceberg.DeleteFileIndex addedDeleteFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, PartitionSet partitionSet, Snapshot parent)
Returns matching delete files have been added to the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationdataFilter
- an expression used to find delete filespartitionSet
- a partition set used to find delete filesparent
- parent snapshot of the branch
-
validateDeletedDataFiles
@Deprecated protected void validateDeletedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateDeletedDataFiles(TableMetadata, Long, Expression, Snapshot)
insteadValidates that no files matching a filter have been deleted from the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationdataFilter
- an expression used to find deleted data files
-
validateDeletedDataFiles
protected void validateDeletedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, Expression dataFilter, Snapshot parent)
Validates that no files matching a filter have been deleted from the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationdataFilter
- an expression used to find deleted data filesparent
- ending snapshot on the branch being validated
-
validateDeletedDataFiles
@Deprecated protected void validateDeletedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet)
Deprecated.will be removed in 1.3.0; useMergingSnapshotProducer.validateNoNewDeleteFiles(TableMetadata, Long, PartitionSet, Snapshot)
insteadValidates that no files matching a filter have been deleted from the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationpartitionSet
- a partition set used to find deleted data files
-
validateDeletedDataFiles
protected void validateDeletedDataFiles(TableMetadata base, java.lang.Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent)
Validates that no files matching a filter have been deleted from the table since a starting snapshot.- Parameters:
base
- table metadata to validatestartingSnapshotId
- id of the snapshot current at the start of the operationpartitionSet
- a partition set used to find deleted data filesparent
- ending snapshot on the branch being validated
-
setNewFilesSequenceNumber
protected void setNewFilesSequenceNumber(long sequenceNumber)
-
validateDataFilesExist
@Deprecated protected void validateDataFilesExist(TableMetadata base, java.lang.Long startingSnapshotId, CharSequenceSet requiredDataFiles, boolean skipDeletes, Expression conflictDetectionFilter)
Deprecated.
-
validateDataFilesExist
protected void validateDataFilesExist(TableMetadata base, java.lang.Long startingSnapshotId, CharSequenceSet requiredDataFiles, boolean skipDeletes, Expression conflictDetectionFilter, Snapshot parent)
-
summary
protected java.util.Map<java.lang.String,java.lang.String> summary()
-
apply
public java.util.List<ManifestFile> apply(TableMetadata base, Snapshot snapshot)
Apply the update's changes to the given metadata and snapshot. Return the new manifest list.- Parameters:
base
- the base table metadata to apply changes tosnapshot
- snapshot to apply the changes to- Returns:
- a manifest list for the new snapshot.
-
updateEvent
public java.lang.Object updateEvent()
Description copied from interface:PendingUpdate
Generates update event to notify about metadata changes- Returns:
- the generated event
-
cleanUncommitted
protected void cleanUncommitted(java.util.Set<ManifestFile> committed)
Clean up any uncommitted manifests that were created.Manifests may not be committed if apply is called more because a commit conflict has occurred. Implementations may keep around manifests because the same changes will be made by both apply calls. This method instructs the implementation to clean up those manifests and passes the paths of the manifests that were actually committed.
- Parameters:
committed
- a set of manifest paths that were actually committed
-
stageOnly
public ThisT stageOnly()
Description copied from interface:SnapshotUpdate
Called to stage a snapshot in table metadata, but not update the current snapshot id.- Specified by:
stageOnly
in interfaceSnapshotUpdate<ThisT>
- Returns:
- this for method chaining
-
scanManifestsWith
public ThisT scanManifestsWith(java.util.concurrent.ExecutorService executorService)
Description copied from interface:SnapshotUpdate
Use a particular executor to scan manifests. The default worker pool will be used by default.- Specified by:
scanManifestsWith
in interfaceSnapshotUpdate<ThisT>
- Parameters:
executorService
- the provided executor- Returns:
- this for method chaining
-
commitMetrics
protected CommitMetrics commitMetrics()
-
reportWith
protected ThisT reportWith(MetricsReporter newReporter)
-
targetBranch
protected void targetBranch(java.lang.String branch)
* A setter for the target branch on which snapshot producer operation should be performed- Parameters:
branch
- to set as target branch
-
targetBranch
protected java.lang.String targetBranch()
-
workerPool
protected java.util.concurrent.ExecutorService workerPool()
-
deleteWith
public ThisT deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback)
Description copied from interface:SnapshotUpdate
Set a callback to delete files instead of the table's default.- Specified by:
deleteWith
in interfaceSnapshotUpdate<ThisT>
- Parameters:
deleteCallback
- a String consumer used to delete locations.- Returns:
- this for method chaining
-
validate
protected void validate(TableMetadata currentMetadata, Snapshot snapshot)
Validate the current metadata.Child operations can override this to add custom validation.
- Parameters:
currentMetadata
- current table metadata to validatesnapshot
- ending snapshot on the lineage which is being validated
-
apply
public Snapshot apply()
Description copied from interface:PendingUpdate
Apply the pending changes and return the uncommitted changes for validation.This does not result in a permanent update.
- Specified by:
apply
in interfacePendingUpdate<ThisT>
- Returns:
- the uncommitted changes that would be committed by calling
PendingUpdate.commit()
-
current
protected TableMetadata current()
-
refresh
protected TableMetadata refresh()
-
commit
public void commit()
Description copied from interface:PendingUpdate
Apply the pending changes and commit.Changes are committed by calling the underlying table's commit method.
Once the commit is successful, the updated table will be refreshed.
- Specified by:
commit
in interfacePendingUpdate<ThisT>
-
cleanAll
protected void cleanAll()
-
deleteFile
protected void deleteFile(java.lang.String path)
-
manifestListPath
protected OutputFile manifestListPath()
-
newManifestOutput
protected OutputFile newManifestOutput()
-
newManifestWriter
protected ManifestWriter<DataFile> newManifestWriter(PartitionSpec spec)
-
newDeleteManifestWriter
protected ManifestWriter<DeleteFile> newDeleteManifestWriter(PartitionSpec spec)
-
newManifestReader
protected ManifestReader<DataFile> newManifestReader(ManifestFile manifest)
-
newDeleteManifestReader
protected ManifestReader<DeleteFile> newDeleteManifestReader(ManifestFile manifest)
-
snapshotId
protected long snapshotId()
-
-