org.apache.iceberg.actions

Class BaseRewriteDataFilesAction<ThisT>

java.lang.Object

org.apache.iceberg.actions.BaseRewriteDataFilesAction<ThisT>

All Implemented Interfaces:

Action<RewriteDataFilesActionResult>, SnapshotUpdateAction<ThisT,RewriteDataFilesActionResult>

Direct Known Subclasses:

RewriteDataFilesAction, RewriteDataFilesAction

public abstract class BaseRewriteDataFilesAction<ThisT>
extends java.lang.Object

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseRewriteDataFilesAction(Table table)

Method Summary

Modifier and Type	Method and Description
protected boolean	caseSensitive()
BaseRewriteDataFilesAction<ThisT>	caseSensitive(boolean newCaseSensitive) Is it case sensitive
protected void	commit(SnapshotUpdate<?> update)
protected EncryptionManager	encryptionManager()
RewriteDataFilesActionResult	execute() Executes this action.
protected abstract FileIO	fileIO()
BaseRewriteDataFilesAction<ThisT>	filter(Expression expr) Pass a row Expression to filter DataFiles to be rewritten.
protected java.util.List<java.lang.String>	getManifestListPaths(java.lang.Iterable<Snapshot> snapshots) Returns all the path locations of all Manifest Lists for a given list of snapshots
protected java.util.List<java.lang.String>	getOtherMetadataFilePaths(TableOperations ops) Returns all Metadata file paths which may not be in the current metadata.
protected java.lang.String	metadataTableName(MetadataTableType type)
protected java.lang.String	metadataTableName(java.lang.String tableName, MetadataTableType type)
BaseRewriteDataFilesAction<ThisT>	outputSpecId(int specId) Pass a PartitionSpec id to specify which PartitionSpec should be used in DataFile rewrite
protected abstract java.util.List<DataFile>	rewriteDataForTasks(java.util.List<CombinedScanTask> combinedScanTask)
protected abstract ThisT	self()
ThisT	set(java.lang.String property, java.lang.String value)
BaseRewriteDataFilesAction<ThisT>	splitLookback(int lookback) Specify the number of "bins" considered when trying to pack the next file split into a task.
BaseRewriteDataFilesAction<ThisT>	splitOpenFileCost(long openFileCost) Specify the minimum file size to count to pack into one "bin".
protected Table	table()
BaseRewriteDataFilesAction<ThisT>	targetSizeInBytes(long targetSize) Specify the target rewrite data file size in bytes

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BaseRewriteDataFilesAction

protected BaseRewriteDataFilesAction(Table table)

Method Detail

table

protected Table table()

encryptionManager

protected EncryptionManager encryptionManager()

caseSensitive

protected boolean caseSensitive()

caseSensitive

public BaseRewriteDataFilesAction<ThisT> caseSensitive(boolean newCaseSensitive)

Is it case sensitive

Parameters:

newCaseSensitive - caseSensitive

Returns:

this for method chaining

outputSpecId

public BaseRewriteDataFilesAction<ThisT> outputSpecId(int specId)

Pass a PartitionSpec id to specify which PartitionSpec should be used in DataFile rewrite

Parameters:

specId - PartitionSpec id to rewrite

Returns:

this for method chaining

targetSizeInBytes

public BaseRewriteDataFilesAction<ThisT> targetSizeInBytes(long targetSize)

Specify the target rewrite data file size in bytes

Parameters:

targetSize - size in bytes of rewrite data file

Returns:

this for method chaining

splitLookback

public BaseRewriteDataFilesAction<ThisT> splitLookback(int lookback)

Specify the number of "bins" considered when trying to pack the next file split into a task. Increasing this usually makes tasks a bit more even by considering more ways to pack file regions into a single task with extra planning cost.

This configuration can reorder the incoming file regions, to preserve order for lower/upper bounds in file metadata, user can use a lookback of 1.

Parameters:

lookback - number of "bins" considered when trying to pack the next file split into a task.

Returns:

this for method chaining

splitOpenFileCost

public BaseRewriteDataFilesAction<ThisT> splitOpenFileCost(long openFileCost)

Specify the minimum file size to count to pack into one "bin". If the read file size is smaller than this specified threshold, Iceberg will use this value to do count.

this configuration controls the number of files to compact for each task, small value would lead to a high compaction, the default value is 4MB.

Parameters:

openFileCost - minimum file size to count to pack into one "bin".

Returns:

this for method chaining

filter

public BaseRewriteDataFilesAction<ThisT> filter(Expression expr)

Pass a row Expression to filter DataFiles to be rewritten. Note that all files that may contain data matching the filter may be rewritten.

Parameters:

expr - Expression to filter out DataFiles

Returns:

this for method chaining

execute

public RewriteDataFilesActionResult execute()

Description copied from interface: Action

Executes this action.

Returns:

the result of this action

fileIO

protected abstract FileIO fileIO()

rewriteDataForTasks

protected abstract java.util.List<DataFile> rewriteDataForTasks(java.util.List<CombinedScanTask> combinedScanTask)

self

protected abstract ThisT self()

set

public ThisT set(java.lang.String property,
                 java.lang.String value)

Specified by:

set in interface SnapshotUpdateAction<ThisT,R>

commit

protected void commit(SnapshotUpdate<?> update)

metadataTableName

protected java.lang.String metadataTableName(MetadataTableType type)

metadataTableName

protected java.lang.String metadataTableName(java.lang.String tableName,
                                             MetadataTableType type)

getManifestListPaths

protected java.util.List<java.lang.String> getManifestListPaths(java.lang.Iterable<Snapshot> snapshots)

Returns all the path locations of all Manifest Lists for a given list of snapshots

Parameters:

snapshots - snapshots

Returns:

the paths of the Manifest Lists

getOtherMetadataFilePaths

protected java.util.List<java.lang.String> getOtherMetadataFilePaths(TableOperations ops)

Returns all Metadata file paths which may not be in the current metadata. Specifically this includes "version-hint" files as well as entries in metadata.previousFiles.

Parameters:

ops - TableOperations for the table we will be getting paths from

Returns:

a list of paths to metadata files