org.apache.iceberg.flink.actions

Class RewriteDataFilesAction

java.lang.Object

org.apache.iceberg.actions.BaseRewriteDataFilesAction<RewriteDataFilesAction>

org.apache.iceberg.flink.actions.RewriteDataFilesAction

All Implemented Interfaces:

Action<RewriteDataFilesActionResult>, SnapshotUpdateAction<RewriteDataFilesAction,RewriteDataFilesActionResult>

public class RewriteDataFilesAction
extends BaseRewriteDataFilesAction<RewriteDataFilesAction>

Constructor Summary

Constructors

Constructor and Description
RewriteDataFilesAction(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env, Table table)

Method Summary

Modifier and Type	Method and Description
protected void	commit(SnapshotUpdate<?> update)
protected FileIO	fileIO()
protected java.util.List<java.lang.String>	getManifestListPaths(java.lang.Iterable<Snapshot> snapshots) Returns all the path locations of all Manifest Lists for a given list of snapshots
protected java.util.List<java.lang.String>	getOtherMetadataFilePaths(TableOperations ops) Returns all Metadata file paths which may not be in the current metadata.
RewriteDataFilesAction	maxParallelism(int parallelism)
protected java.lang.String	metadataTableName(MetadataTableType type)
protected java.lang.String	metadataTableName(java.lang.String tableName, MetadataTableType type)
protected java.util.List<DataFile>	rewriteDataForTasks(java.util.List<CombinedScanTask> combinedScanTasks)
protected RewriteDataFilesAction	self()
ThisT	set(java.lang.String property, java.lang.String value)

Methods inherited from class org.apache.iceberg.actions.BaseRewriteDataFilesAction

caseSensitive, caseSensitive, encryptionManager, execute, filter, outputSpecId, splitLookback, splitOpenFileCost, table, targetSizeInBytes

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RewriteDataFilesAction

public RewriteDataFilesAction(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env,
                              Table table)

Method Detail

fileIO

protected FileIO fileIO()

Specified by:

fileIO in class BaseRewriteDataFilesAction<RewriteDataFilesAction>

rewriteDataForTasks

protected java.util.List<DataFile> rewriteDataForTasks(java.util.List<CombinedScanTask> combinedScanTasks)

Specified by:

rewriteDataForTasks in class BaseRewriteDataFilesAction<RewriteDataFilesAction>

self

protected RewriteDataFilesAction self()

maxParallelism

public RewriteDataFilesAction maxParallelism(int parallelism)

set

public ThisT set(java.lang.String property,
                 java.lang.String value)

Specified by:

set in interface SnapshotUpdateAction<ThisT,R>

commit

protected void commit(SnapshotUpdate<?> update)

metadataTableName

protected java.lang.String metadataTableName(MetadataTableType type)

metadataTableName

protected java.lang.String metadataTableName(java.lang.String tableName,
                                             MetadataTableType type)

getManifestListPaths

protected java.util.List<java.lang.String> getManifestListPaths(java.lang.Iterable<Snapshot> snapshots)

Returns all the path locations of all Manifest Lists for a given list of snapshots

Parameters:

snapshots - snapshots

Returns:

the paths of the Manifest Lists

getOtherMetadataFilePaths

protected java.util.List<java.lang.String> getOtherMetadataFilePaths(TableOperations ops)

Returns all Metadata file paths which may not be in the current metadata. Specifically this includes "version-hint" files as well as entries in metadata.previousFiles.

Parameters:

ops - TableOperations for the table we will be getting paths from

Returns:

a list of paths to metadata files