Package org.apache.iceberg.actions

Class SizeBasedDataRewriter

java.lang.Object

org.apache.iceberg.actions.SizeBasedFileRewriter<FileScanTask,DataFile>

org.apache.iceberg.actions.SizeBasedDataRewriter

All Implemented Interfaces:

FileRewriter<FileScanTask,DataFile>

public abstract class SizeBasedDataRewriter
extends SizeBasedFileRewriter<FileScanTask,DataFile>

Field Summary

Fields

Modifier and Type	Field	Description
static final String	DELETE_FILE_THRESHOLD	The minimum number of deletes that needs to be associated with a data file for it to be considered for rewriting.
static final int	DELETE_FILE_THRESHOLD_DEFAULT
static final String	DELETE_RATIO_THRESHOLD	The minimum deletion ratio that needs to be associated with a data file for it to be considered for rewriting.
static final double	DELETE_RATIO_THRESHOLD_DEFAULT

Fields inherited from class org.apache.iceberg.actions.SizeBasedFileRewriter

MAX_FILE_GROUP_SIZE_BYTES, MAX_FILE_GROUP_SIZE_BYTES_DEFAULT, MAX_FILE_SIZE_BYTES, MAX_FILE_SIZE_DEFAULT_RATIO, MIN_FILE_SIZE_BYTES, MIN_FILE_SIZE_DEFAULT_RATIO, MIN_INPUT_FILES, MIN_INPUT_FILES_DEFAULT, REWRITE_ALL, REWRITE_ALL_DEFAULT, TARGET_FILE_SIZE_BYTES

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	SizeBasedDataRewriter(Table table)

Method Summary

Modifier and Type	Method	Description
protected long	defaultTargetFileSize()
protected Iterable<List<FileScanTask>>	filterFileGroups(List<List<FileScanTask>> groups)
protected Iterable<FileScanTask>	filterFiles(Iterable<FileScanTask> tasks)
void	init(Map<String,String> options)	Initializes this rewriter using provided options.
Set<String>	validOptions()	Returns a set of supported options for this rewriter.

Methods inherited from class org.apache.iceberg.actions.SizeBasedFileRewriter

enoughContent, enoughInputFiles, inputSize, numOutputFiles, outputSpec, outputSpecId, planFileGroups, splitSize, table, tooMuchContent, writeMaxFileSize, wronglySized

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.actions.FileRewriter

description, rewrite

Field Details

DELETE_FILE_THRESHOLD

public static final String DELETE_FILE_THRESHOLD

The minimum number of deletes that needs to be associated with a data file for it to be considered for rewriting. If a data file has this number of deletes or more, it will be rewritten regardless of its file size determined by SizeBasedFileRewriter.MIN_FILE_SIZE_BYTES and SizeBasedFileRewriter.MAX_FILE_SIZE_BYTES. If a file group contains a file that satisfies this condition, the file group will be rewritten regardless of the number of files in the file group determined by SizeBasedFileRewriter.MIN_INPUT_FILES.

Defaults to Integer.MAX_VALUE, which means this feature is not enabled by default.

See Also:

• Constant Field Values

DELETE_FILE_THRESHOLD_DEFAULT

public static final int DELETE_FILE_THRESHOLD_DEFAULT

See Also:

• Constant Field Values

DELETE_RATIO_THRESHOLD

public static final String DELETE_RATIO_THRESHOLD

The minimum deletion ratio that needs to be associated with a data file for it to be considered for rewriting. If the deletion ratio of a data file is greater than or equal to this value, it will be rewritten regardless of its file size determined by SizeBasedFileRewriter.MIN_FILE_SIZE_BYTES and SizeBasedFileRewriter.MAX_FILE_SIZE_BYTES. If a file group contains a file that satisfies this condition, the file group will be rewritten regardless of the number of files in the file group determined by SizeBasedFileRewriter.MIN_INPUT_FILES.

Defaults to 0.3, which means that if the deletion ratio of a file reaches or exceeds 30%, it may trigger the rewriting operation.

See Also:

• Constant Field Values

DELETE_RATIO_THRESHOLD_DEFAULT

public static final double DELETE_RATIO_THRESHOLD_DEFAULT

See Also:

• Constant Field Values

Constructor Details

SizeBasedDataRewriter

protected SizeBasedDataRewriter(Table table)

Method Details

validOptions

public Set<String> validOptions()

Description copied from interface: FileRewriter

Returns a set of supported options for this rewriter. Only options specified in this list will be accepted at runtime. Any other options will be rejected.

Specified by:

validOptions in interface FileRewriter<FileScanTask,DataFile>

Overrides:

validOptions in class SizeBasedFileRewriter<FileScanTask,DataFile>

init

public void init(Map<String,String> options)

Description copied from interface: FileRewriter

Initializes this rewriter using provided options.

Specified by:

init in interface FileRewriter<FileScanTask,DataFile>

Overrides:

init in class SizeBasedFileRewriter<FileScanTask,DataFile>

Parameters:

options - options to initialize this rewriter

filterFiles

protected Iterable<FileScanTask> filterFiles(Iterable<FileScanTask> tasks)

Specified by:

filterFiles in class SizeBasedFileRewriter<FileScanTask,DataFile>

filterFileGroups

protected Iterable<List<FileScanTask>> filterFileGroups(List<List<FileScanTask>> groups)

Specified by:

filterFileGroups in class SizeBasedFileRewriter<FileScanTask,DataFile>

defaultTargetFileSize

protected long defaultTargetFileSize()

Specified by:

defaultTargetFileSize in class SizeBasedFileRewriter<FileScanTask,DataFile>