Package org.apache.iceberg

Class BaseOverwriteFiles

    • Constructor Detail

      • BaseOverwriteFiles

        protected BaseOverwriteFiles​(java.lang.String tableName,
                             TableOperations ops)

    • Method Detail

      • operation

        protected java.lang.String operation()
        A string that describes the action that produced the new snapshot.
        Returns:
        a string operation

      • overwriteByRowFilter

        public OverwriteFiles overwriteByRowFilter​(Expression expr)
        Description copied from interface: OverwriteFiles
        Delete files that match an Expression on data rows from the table.

        A file is selected to be deleted by the expression if it could contain any rows that match the expression (candidate files are selected using an inclusive projection). These candidate files are deleted if all of the rows in the file must match the expression (the partition data matches the expression's Projections.strict(PartitionSpec) strict projection}). This guarantees that files are deleted if and only if all rows in the file must match the expression.

        Files that may contain some rows that match the expression and some rows that do not will result in a ValidationException.

        Specified by:
        overwriteByRowFilter in interface OverwriteFiles
        Parameters:
        expr - an expression on rows in the table
        Returns:
        this for method chaining

      • validateAddedFilesMatchOverwriteFilter

        public OverwriteFiles validateAddedFilesMatchOverwriteFilter()
        Description copied from interface: OverwriteFiles
        Signal that each file added to the table must match the overwrite expression.

        If this method is called, each added file is validated on commit to ensure that it matches the overwrite row filter. This is used to ensure that writes are idempotent: that files cannot be added during a commit that would not be removed if the operation were run a second time.

        Specified by:
        validateAddedFilesMatchOverwriteFilter in interface OverwriteFiles
        Returns:
        this for method chaining

      • validateNoConflictingAppends

        @Deprecated
public OverwriteFiles validateNoConflictingAppends​(java.lang.Long newReadSnapshotId,
                                                   Expression newConflictDetectionFilter)
        Deprecated.
        Description copied from interface: OverwriteFiles
        Enables validation that files added concurrently do not conflict with this commit's operation.

        This method should be called when the table is queried to determine which files to delete/append. If a concurrent operation commits a new file after the data was read and that file might contain rows matching the specified conflict detection filter, the overwrite operation will detect this during retries and fail.

        Calling this method with a correct conflict detection filter is required to maintain serializable isolation for eager update/delete operations. Otherwise, the isolation level will be snapshot isolation.

        Specified by:
        validateNoConflictingAppends in interface OverwriteFiles
        Parameters:
        newReadSnapshotId - the snapshot id that was used to read the data or null if the table was empty
        newConflictDetectionFilter - an expression on rows in the table
        Returns:
        this for method chaining

      • validateFromSnapshot

        public OverwriteFiles validateFromSnapshot​(long snapshotId)
        Description copied from interface: OverwriteFiles
        Set the snapshot ID used in any reads for this operation.

        Validations will check changes after this snapshot ID. If the from snapshot is not set, all ancestor snapshots through the table's initial snapshot are validated.

        Specified by:
        validateFromSnapshot in interface OverwriteFiles
        Parameters:
        snapshotId - a snapshot ID
        Returns:
        this for method chaining

      • caseSensitive

        public OverwriteFiles caseSensitive​(boolean isCaseSensitive)
        Description copied from interface: OverwriteFiles
        Enables or disables case sensitive expression binding for validations that accept expressions.
        Specified by:
        caseSensitive in interface OverwriteFiles
        Parameters:
        isCaseSensitive - whether expression binding should be case sensitive
        Returns:
        this for method chaining

      • conflictDetectionFilter

        public OverwriteFiles conflictDetectionFilter​(Expression newConflictDetectionFilter)
        Description copied from interface: OverwriteFiles
        Sets a conflict detection filter used to validate concurrently added data and delete files.
        Specified by:
        conflictDetectionFilter in interface OverwriteFiles
        Parameters:
        newConflictDetectionFilter - an expression on rows in the table
        Returns:
        this for method chaining

      • validateNoConflictingData

        public OverwriteFiles validateNoConflictingData()
        Description copied from interface: OverwriteFiles
        Enables validation that data added concurrently does not conflict with this commit's operation.

        This method should be called while committing non-idempotent overwrite operations. If a concurrent operation commits a new file after the data was read and that file might contain rows matching the specified conflict detection filter, the overwrite operation will detect this and fail.

        Calling this method with a correct conflict detection filter is required to maintain isolation for non-idempotent overwrite operations.

        Validation uses the conflict detection filter passed to OverwriteFiles.conflictDetectionFilter(Expression) and applies to operations that happened after the snapshot passed to OverwriteFiles.validateFromSnapshot(long). If the conflict detection filter is not set, any new data added concurrently will fail this overwrite operation.

        Specified by:
        validateNoConflictingData in interface OverwriteFiles
        Returns:
        this for method chaining

      • validateNoConflictingDeletes

        public OverwriteFiles validateNoConflictingDeletes()
        Description copied from interface: OverwriteFiles
        Enables validation that deletes that happened concurrently do not conflict with this commit's operation.

        Validating concurrent deletes is required during non-idempotent overwrite operations. If a concurrent operation deletes data in one of the files being overwritten, the overwrite operation must be aborted as it may undelete rows that were removed concurrently.

        Calling this method with a correct conflict detection filter is required to maintain isolation for non-idempotent overwrite operations.

        Validation uses the conflict detection filter passed to OverwriteFiles.conflictDetectionFilter(Expression) and applies to operations that happened after the snapshot passed to OverwriteFiles.validateFromSnapshot(long). If the conflict detection filter is not set, this operation will use the row filter provided in OverwriteFiles.overwriteByRowFilter(Expression) to check for new delete files and will ensure there are no conflicting deletes for data files removed via OverwriteFiles.deleteFile(DataFile).

        Specified by:
        validateNoConflictingDeletes in interface OverwriteFiles
        Returns:
        this for method chaining

      • validate

        protected void validate​(TableMetadata base)
        Validate the current metadata.

        Child operations can override this to add custom validation.

        Parameters:
        base - current table metadata to validate

      • set

        public ThisT set​(java.lang.String property,
                 java.lang.String value)
        Description copied from interface: SnapshotUpdate
        Set a summary property in the snapshot produced by this update.
        Parameters:
        property - a String property name
        value - a String property value
        Returns:
        this for method chaining

      • addedFiles

        protected java.util.List<DataFile> addedFiles()

      • failAnyDelete

        protected void failAnyDelete()

      • failMissingDeletePaths

        protected void failMissingDeletePaths()

      • deleteByRowFilter

        protected void deleteByRowFilter​(Expression expr)
        Add a filter to match files to delete. A file will be deleted if all of the rows it contains match this or any other filter passed to this method.
        Parameters:
        expr - an expression to match rows.

      • dropPartition

        protected void dropPartition​(int specId,
                             StructLike partition)
        Add a partition tuple to drop from the table during the delete phase.

      • delete

        protected void delete​(DataFile file)
        Add a specific data file to be deleted in the new snapshot.

      • delete

        protected void delete​(DeleteFile file)
        Add a specific delete file to be deleted in the new snapshot.

      • delete

        protected void delete​(java.lang.CharSequence path)
        Add a specific data path to be deleted in the new snapshot.

      • add

        protected void add​(DataFile file)
        Add a data file to the new snapshot.

      • add

        protected void add​(DeleteFile file)
        Add a delete file to the new snapshot.

      • add

        protected void add​(ManifestFile manifest)
        Add all files in a manifest to the new snapshot.

      • validateAddedDataFiles

        protected void validateAddedDataFiles​(TableMetadata base,
                                      java.lang.Long startingSnapshotId,
                                      Expression conflictDetectionFilter,
                                      boolean caseSensitive)
        Validates that no files matching a filter have been added to the table since a starting snapshot.
        Parameters:
        base - table metadata to validate
        startingSnapshotId - id of the snapshot current at the start of the operation
        conflictDetectionFilter - an expression used to find new conflicting data files
        caseSensitive - whether expression evaluation should be case sensitive

      • validateNoNewDeletesForDataFiles

        protected void validateNoNewDeletesForDataFiles​(TableMetadata base,
                                                java.lang.Long startingSnapshotId,
                                                java.lang.Iterable<DataFile> dataFiles)
        Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.
        Parameters:
        base - table metadata to validate
        startingSnapshotId - id of the snapshot current at the start of the operation
        dataFiles - data files to validate have no new row deletes

      • validateNoNewDeletesForDataFiles

        protected void validateNoNewDeletesForDataFiles​(TableMetadata base,
                                                java.lang.Long startingSnapshotId,
                                                Expression dataFilter,
                                                java.lang.Iterable<DataFile> dataFiles,
                                                boolean caseSensitive)
        Validates that no new delete files that must be applied to the given data files have been added to the table since a starting snapshot.
        Parameters:
        base - table metadata to validate
        startingSnapshotId - id of the snapshot current at the start of the operation
        dataFilter - a data filter
        dataFiles - data files to validate have no new row deletes
        caseSensitive - whether expression binding should be case-sensitive

      • validateNoNewDeleteFiles

        protected void validateNoNewDeleteFiles​(TableMetadata base,
                                        java.lang.Long startingSnapshotId,
                                        Expression dataFilter,
                                        boolean caseSensitive)
        Validates that no delete files matching a filter have been added to the table since a starting snapshot.
        Parameters:
        base - table metadata to validate
        startingSnapshotId - id of the snapshot current at the start of the operation
        dataFilter - an expression used to find new conflicting delete files
        caseSensitive - whether expression evaluation should be case-sensitive

      • validateDataFilesExist

        protected void validateDataFilesExist​(TableMetadata base,
                                      java.lang.Long startingSnapshotId,
                                      CharSequenceSet requiredDataFiles,
                                      boolean skipDeletes,
                                      Expression conflictDetectionFilter)

      • summary

        protected java.util.Map<java.lang.String,​java.lang.String> summary()

      • apply

        public java.util.List<ManifestFile> apply​(TableMetadata base)
        Apply the update's changes to the base table metadata and return the new manifest list.
        Parameters:
        base - the base table metadata to apply changes to
        Returns:
        a manifest list for the new snapshot.

      • updateEvent

        public java.lang.Object updateEvent()
        Description copied from interface: PendingUpdate
        Generates update event to notify about metadata changes
        Returns:
        the generated event

      • cleanUncommitted

        protected void cleanUncommitted​(java.util.Set<ManifestFile> committed)
        Clean up any uncommitted manifests that were created.

        Manifests may not be committed if apply is called more because a commit conflict has occurred. Implementations may keep around manifests because the same changes will be made by both apply calls. This method instructs the implementation to clean up those manifests and passes the paths of the manifests that were actually committed.

        Parameters:
        committed - a set of manifest paths that were actually committed

      • stageOnly

        public ThisT stageOnly()
        Description copied from interface: SnapshotUpdate
        Called to stage a snapshot in table metadata, but not update the current snapshot id.
        Specified by:
        stageOnly in interface SnapshotUpdate<ThisT>
        Returns:
        this for method chaining

      • deleteWith

        public ThisT deleteWith​(java.util.function.Consumer<java.lang.String> deleteCallback)
        Description copied from interface: SnapshotUpdate
        Set a callback to delete files instead of the table's default.
        Specified by:
        deleteWith in interface SnapshotUpdate<ThisT>
        Parameters:
        deleteCallback - a String consumer used to delete locations.
        Returns:
        this for method chaining

      • apply

        public Snapshot apply()
        Description copied from interface: PendingUpdate
        Apply the pending changes and return the uncommitted changes for validation.

        This does not result in a permanent update.

        Specified by:
        apply in interface PendingUpdate<ThisT>
        Returns:
        the uncommitted changes that would be committed by calling PendingUpdate.commit()

      • commit

        public void commit()
        Description copied from interface: PendingUpdate
        Apply the pending changes and commit.

        Changes are committed by calling the underlying table's commit method.

        Once the commit is successful, the updated table will be refreshed.

        Specified by:
        commit in interface PendingUpdate<ThisT>

      • cleanAll

        protected void cleanAll()

      • deleteFile

        protected void deleteFile​(java.lang.String path)

      • manifestListPath

        protected OutputFile manifestListPath()

      • newManifestOutput

        protected OutputFile newManifestOutput()

      • snapshotId

        protected long snapshotId()