org.apache.iceberg

Class OverwriteData

java.lang.Object

org.apache.iceberg.OverwriteData

All Implemented Interfaces:

OverwriteFiles, PendingUpdate<Snapshot>, SnapshotUpdate<OverwriteFiles>

public class OverwriteData
extends java.lang.Object
implements OverwriteFiles

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	org.apache.iceberg.MergingSnapshotProducer.DeleteException

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	OverwriteData(TableOperations ops)

Method Summary

Modifier and Type	Method and Description
protected void	add(DataFile file) Add a file to the new snapshot.
protected void	add(ManifestFile manifest) Add all files in a manifest to the new snapshot.
protected java.util.List<DataFile>	addedFiles()
OverwriteFiles	addFile(DataFile file) Add a DataFile to the table.
Snapshot	apply() Apply the pending changes and return the uncommitted changes for validation.
java.util.List<ManifestFile>	apply(TableMetadata base) Apply the update's changes to the base table metadata and return the new manifest list.
protected void	cleanAll()
protected void	cleanUncommitted(java.util.Set<ManifestFile> committed) Clean up any uncommitted manifests that were created.
void	commit() Apply the pending changes and commit.
protected void	delete(java.lang.CharSequence path) Add a specific path to be deleted in the new snapshot.
protected void	deleteByRowFilter(Expression expr) Add a filter to match files to delete.
protected void	deleteFile(java.lang.String path)
ThisT	deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback) Set a callback to delete files instead of the table's default.
protected void	dropPartition(StructLike partition) Add a partition tuple to drop from the table during the delete phase.
protected void	failAnyDelete()
protected void	failMissingDeletePaths()
protected OutputFile	manifestListPath()
protected OutputFile	manifestPath(int manifestNumber)
protected java.lang.String	operation() A string that describes the action that produced the new snapshot.
OverwriteFiles	overwriteByRowFilter(Expression expr) Delete files that match an Expression on data rows from the table.
protected Expression	rowFilter()
protected OverwriteFiles	self()
ThisT	set(java.lang.String property, java.lang.String value) Set a summary property in the snapshot produced by this update.
protected long	snapshotId()
protected java.util.Map<java.lang.String,java.lang.String>	summary()
OverwriteFiles	validateAddedFiles() Signal that each file added to the table must match the overwrite expression.
protected PartitionSpec	writeSpec()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.SnapshotUpdate

deleteWith, set

Methods inherited from interface org.apache.iceberg.PendingUpdate

apply, commit

Constructor Detail

OverwriteData

protected OverwriteData(TableOperations ops)

Method Detail

self

protected OverwriteFiles self()

operation

protected java.lang.String operation()

A string that describes the action that produced the new snapshot.

Returns:

a string operation

overwriteByRowFilter

public OverwriteFiles overwriteByRowFilter(Expression expr)

Description copied from interface: OverwriteFiles

Delete files that match an Expression on data rows from the table.

A file is selected to be deleted by the expression if it could contain any rows that match the expression (candidate files are selected using an inclusive projection). These candidate files are deleted if all of the rows in the file must match the expression (the partition data matches the expression's Projections.strict(PartitionSpec) strict projection}). This guarantees that files are deleted if and only if all rows in the file must match the expression.

Files that may contain some rows that match the expression and some rows that do not will result in a ValidationException.

Specified by:

overwriteByRowFilter in interface OverwriteFiles

Parameters:

expr - an expression on rows in the table

Returns:

this for method chaining

addFile

public OverwriteFiles addFile(DataFile file)

Description copied from interface: OverwriteFiles

Add a DataFile to the table.

Specified by:

addFile in interface OverwriteFiles

Parameters:

file - a data file

Returns:

this for method chaining

validateAddedFiles

public OverwriteFiles validateAddedFiles()

Description copied from interface: OverwriteFiles

Signal that each file added to the table must match the overwrite expression.

If this method is called, each added file is validated on commit to ensure that it matches the overwrite row filter. This is used to ensure that writes are idempotent: that files cannot be added during a commit that would not be removed if the operation were run a second time.

Specified by:

validateAddedFiles in interface OverwriteFiles

Returns:

this for method chaining

apply

public java.util.List<ManifestFile> apply(TableMetadata base)

Apply the update's changes to the base table metadata and return the new manifest list.

Parameters:

base - the base table metadata to apply changes to

Returns:

a manifest list for the new snapshot.

set

public ThisT set(java.lang.String property,
                 java.lang.String value)

Description copied from interface: SnapshotUpdate

Set a summary property in the snapshot produced by this update.

Parameters:

property - a String property name

value - a String property value

Returns:

this for method chaining

writeSpec

protected PartitionSpec writeSpec()

rowFilter

protected Expression rowFilter()

addedFiles

protected java.util.List<DataFile> addedFiles()

failAnyDelete

protected void failAnyDelete()

failMissingDeletePaths

protected void failMissingDeletePaths()

deleteByRowFilter

protected void deleteByRowFilter(Expression expr)

Add a filter to match files to delete. A file will be deleted if all of the rows it contains match this or any other filter passed to this method.

Parameters:

expr - an expression to match rows.

dropPartition

protected void dropPartition(StructLike partition)

Add a partition tuple to drop from the table during the delete phase.

delete

protected void delete(java.lang.CharSequence path)

Add a specific path to be deleted in the new snapshot.

add

protected void add(DataFile file)

Add a file to the new snapshot.

add

protected void add(ManifestFile manifest)

Add all files in a manifest to the new snapshot.

summary

protected java.util.Map<java.lang.String,java.lang.String> summary()

cleanUncommitted

protected void cleanUncommitted(java.util.Set<ManifestFile> committed)

Clean up any uncommitted manifests that were created.

Manifests may not be committed if apply is called more because a commit conflict has occurred. Implementations may keep around manifests because the same changes will be made by both apply calls. This method instructs the implementation to clean up those manifests and passes the paths of the manifests that were actually committed.

Parameters:

committed - a set of manifest paths that were actually committed

deleteWith

public ThisT deleteWith(java.util.function.Consumer<java.lang.String> deleteCallback)

Description copied from interface: SnapshotUpdate

Set a callback to delete files instead of the table's default.

Specified by:

deleteWith in interface SnapshotUpdate<ThisT>

Parameters:

deleteCallback - a String consumer used to delete locations.

Returns:

this for method chaining

apply

public Snapshot apply()

Description copied from interface: PendingUpdate

Apply the pending changes and return the uncommitted changes for validation.

This does not result in a permanent update.

Specified by:

apply in interface PendingUpdate<Snapshot>

Returns:

the uncommitted changes that would be committed by calling PendingUpdate.commit()

commit

public void commit()

Description copied from interface: PendingUpdate

Apply the pending changes and commit.

Changes are committed by calling the underlying table's commit method.

Once the commit is successful, the updated table will be refreshed.

Specified by:

commit in interface PendingUpdate<Snapshot>

cleanAll

protected void cleanAll()

deleteFile

protected void deleteFile(java.lang.String path)

manifestListPath

protected OutputFile manifestListPath()

manifestPath

protected OutputFile manifestPath(int manifestNumber)

snapshotId

protected long snapshotId()