org.apache.iceberg.actions

Class RewriteManifestsAction

java.lang.Object

org.apache.iceberg.actions.RewriteManifestsAction

All Implemented Interfaces:

Action<RewriteManifestsActionResult>, SnapshotUpdateAction<RewriteManifestsAction,RewriteManifestsActionResult>

public class RewriteManifestsAction
extends java.lang.Object

An action that rewrites manifests in a distributed manner and co-locates metadata for partitions.

By default, this action rewrites all manifests for the current partition spec and writes the result to the metadata folder. The behavior can be modified by passing a custom predicate to rewriteIf(Predicate) and a custom spec id to specId(int). In addition, there is a way to configure a custom location for new manifests via stagingLocation.

Method Summary

Modifier and Type	Method and Description
protected void	commit(SnapshotUpdate<?> update)
RewriteManifestsActionResult	execute() Executes this action.
protected java.util.List<java.lang.String>	getManifestListPaths(java.lang.Iterable<Snapshot> snapshots) Returns all the path locations of all Manifest Lists for a given list of snapshots
protected java.util.List<java.lang.String>	getOtherMetadataFilePaths(TableOperations ops) Returns all Metadata file paths which may not be in the current metadata.
protected java.lang.String	metadataTableName(MetadataTableType type)
protected java.lang.String	metadataTableName(java.lang.String tableName, MetadataTableType type)
RewriteManifestsAction	rewriteIf(java.util.function.Predicate<ManifestFile> newPredicate) Rewrites only manifests that match the given predicate.
protected RewriteManifestsAction	self()
ThisT	set(java.lang.String property, java.lang.String value)
RewriteManifestsAction	specId(int specId)
RewriteManifestsAction	stagingLocation(java.lang.String newStagingLocation) Passes a location where the manifests should be written.
protected Table	table()
RewriteManifestsAction	useCaching(boolean newUseCaching) Configures whether the action should cache manifest entries used in multiple jobs.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

self

protected RewriteManifestsAction self()

table

protected Table table()

specId

public RewriteManifestsAction specId(int specId)

rewriteIf

public RewriteManifestsAction rewriteIf(java.util.function.Predicate<ManifestFile> newPredicate)

Rewrites only manifests that match the given predicate.

Parameters:

newPredicate - a predicate

Returns:

this for method chaining

stagingLocation

public RewriteManifestsAction stagingLocation(java.lang.String newStagingLocation)

Passes a location where the manifests should be written.

Parameters:

newStagingLocation - a staging location

Returns:

this for method chaining

useCaching

public RewriteManifestsAction useCaching(boolean newUseCaching)

Configures whether the action should cache manifest entries used in multiple jobs.

Parameters:

newUseCaching - a flag whether to use caching

Returns:

this for method chaining

execute

public RewriteManifestsActionResult execute()

Description copied from interface: Action

Executes this action.

Returns:

the result of this action

set

public ThisT set(java.lang.String property,
                 java.lang.String value)

Specified by:

set in interface SnapshotUpdateAction<ThisT,R>

commit

protected void commit(SnapshotUpdate<?> update)

metadataTableName

protected java.lang.String metadataTableName(MetadataTableType type)

metadataTableName

protected java.lang.String metadataTableName(java.lang.String tableName,
                                             MetadataTableType type)

getManifestListPaths

protected java.util.List<java.lang.String> getManifestListPaths(java.lang.Iterable<Snapshot> snapshots)

Returns all the path locations of all Manifest Lists for a given list of snapshots

Parameters:

snapshots - snapshots

Returns:

the paths of the Manifest Lists

getOtherMetadataFilePaths

protected java.util.List<java.lang.String> getOtherMetadataFilePaths(TableOperations ops)

Returns all Metadata file paths which may not be in the current metadata. Specifically this includes "version-hint" files as well as entries in metadata.previousFiles.

Parameters:

ops - TableOperations for the table we will be getting paths from

Returns:

a list of paths to metadata files