Package org.apache.iceberg.flink

Class FlinkWriteConf

java.lang.Object

org.apache.iceberg.flink.FlinkWriteConf

public class FlinkWriteConf
extends Object

A class for common Iceberg configs for Flink writes.

If a config is set at multiple levels, the following order of precedence is used (top to bottom):

1. Write options
2. flink ReadableConfig
3. Table metadata

The most specific value is set in write options and takes precedence over all other configs. If no write option is provided, this class checks the flink configuration for any overrides. If no applicable value is found in the write options, this class uses the table metadata.

Note this class is NOT meant to be serialized.

Constructor Summary

Constructors

Constructor	Description
FlinkWriteConf(Map<String,String> writeOptions, org.apache.flink.configuration.ReadableConfig readableConfig)
FlinkWriteConf(Table table, Map<String,String> writeOptions, org.apache.flink.configuration.ReadableConfig readableConfig)

Method Summary

Modifier and Type	Method	Description
String	avroCompressionCodec()
String	avroCompressionLevel()
String	branch()
boolean	compactMode()
FileFormat	dataFileFormat()
DistributionMode	distributionMode()
String	orcCompressionCodec()
String	orcCompressionStrategy()
boolean	overwriteMode()
String	parquetCompressionCodec()
String	parquetCompressionLevel()
double	rangeDistributionSortKeyBaseWeight()
StatisticsType	rangeDistributionStatisticsType()
Duration	tableRefreshInterval()	NOTE: This may be removed or changed in a future release.
long	targetDataFileSize()
boolean	upsertMode()
int	workerPoolSize()
Integer	writeParallelism()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

FlinkWriteConf

public FlinkWriteConf(Table table,
 Map<String,String> writeOptions,
 org.apache.flink.configuration.ReadableConfig readableConfig)

FlinkWriteConf

public FlinkWriteConf(Map<String,String> writeOptions,
 org.apache.flink.configuration.ReadableConfig readableConfig)

Method Details

overwriteMode

public boolean overwriteMode()

upsertMode

public boolean upsertMode()

dataFileFormat

public FileFormat dataFileFormat()

targetDataFileSize

public long targetDataFileSize()

parquetCompressionCodec

public String parquetCompressionCodec()

parquetCompressionLevel

public String parquetCompressionLevel()

avroCompressionCodec

public String avroCompressionCodec()

avroCompressionLevel

public String avroCompressionLevel()

orcCompressionCodec

public String orcCompressionCodec()

orcCompressionStrategy

public String orcCompressionStrategy()

distributionMode

public DistributionMode distributionMode()

rangeDistributionStatisticsType

public StatisticsType rangeDistributionStatisticsType()

rangeDistributionSortKeyBaseWeight

public double rangeDistributionSortKeyBaseWeight()

workerPoolSize

public int workerPoolSize()

branch

public String branch()

writeParallelism

public Integer writeParallelism()

compactMode

public boolean compactMode()

tableRefreshInterval

@Experimental
public Duration tableRefreshInterval()

NOTE: This may be removed or changed in a future release. This value specifies the interval for refreshing the table instances in sink writer subtasks. If not specified then the default behavior is to not refresh the table.

Returns:

the interval for refreshing the table in sink writer subtasks