Package org.apache.iceberg.flink.source

Class IcebergSource.Builder<T>

java.lang.Object

org.apache.iceberg.flink.source.IcebergSource.Builder<T>

Enclosing class:

IcebergSource<T>

public static class IcebergSource.Builder<T>
extends Object

Method Summary

Modifier and Type	Method	Description
IcebergSource.Builder<T>	asOfTimestamp(Long newAsOfTimestamp)
IcebergSource.Builder<T>	assignerFactory(SplitAssignerFactory assignerFactory)
IcebergSource.Builder<T>	branch(String branch)
IcebergSource<T>	build()
org.apache.flink.streaming.api.datastream.DataStream<T>	buildStream(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env)	Build the IcebergSource and create a DataStream from the source.
IcebergSource.Builder<T>	caseSensitive(boolean newCaseSensitive)
IcebergSource.Builder<T>	endSnapshotId(Long newEndSnapshotId)
IcebergSource.Builder<T>	endTag(String endTag)
IcebergSource.Builder<T>	exposeLocality(boolean newExposeLocality)
IcebergSource.Builder<T>	filters(List<Expression> newFilters)
IcebergSource.Builder<T>	flinkConfig(org.apache.flink.configuration.ReadableConfig config)
IcebergSource.Builder<T>	includeColumnStats(boolean newIncludeColumnStats)
IcebergSource.Builder<T>	limit(Long newLimit)
IcebergSource.Builder<T>	maxAllowedPlanningFailures(int maxAllowedPlanningFailures)
IcebergSource.Builder<T>	monitorInterval(Duration newMonitorInterval)
IcebergSource.Builder<T>	nameMapping(String newNameMapping)
IcebergSource.Builder<T>	planParallelism(int planParallelism)
IcebergSource.Builder<T>	project(org.apache.flink.table.catalog.ResolvedSchema newProjectedFlinkSchema)
IcebergSource.Builder<T>	project(org.apache.flink.table.legacy.api.TableSchema newProjectedFlinkSchema)	Deprecated. since 1.10.0, will be removed in 2.0.0.
IcebergSource.Builder<T>	project(Schema newProjectedSchema)
IcebergSource.Builder<T>	properties(Map<String,String> properties)	Deprecated. will be removed in 2.0.0; use setAll(java.util.Map<java.lang.String, java.lang.String>) instead.
IcebergSource.Builder<T>	readerFunction(ReaderFunction<T> newReaderFunction)	Deprecated. since 1.7.0.
IcebergSource.Builder<T>	set(String property, String value)	Set the read properties for Flink source.
IcebergSource.Builder<T>	setAll(Map<String,String> properties)	Set the read properties for Flink source.
IcebergSource.Builder<T>	splitComparator(SerializableComparator<IcebergSourceSplit> newSplitComparator)
IcebergSource.Builder<T>	splitLookback(Integer newSplitLookback)
IcebergSource.Builder<T>	splitOpenFileCost(Long newSplitOpenFileCost)
IcebergSource.Builder<T>	splitSize(Long newSplitSize)
IcebergSource.Builder<T>	startSnapshotId(Long newStartSnapshotId)
IcebergSource.Builder<T>	startSnapshotTimestamp(Long newStartSnapshotTimestamp)
IcebergSource.Builder<T>	startTag(String startTag)
IcebergSource.Builder<T>	streaming(boolean streaming)
IcebergSource.Builder<T>	streamingStartingStrategy(StreamingStartingStrategy newStartingStrategy)
IcebergSource.Builder<T>	table(Table newTable)
IcebergSource.Builder<T>	tableLoader(TableLoader loader)
IcebergSource.Builder<T>	tag(String tag)
IcebergSource.Builder<T>	useSnapshotId(Long newSnapshotId)
IcebergSource.Builder<T>	watermarkColumn(String columnName)	Emits watermarks once per split based on the min value of column statistics from files metadata in the given split.
IcebergSource.Builder<T>	watermarkColumnTimeUnit(TimeUnit timeUnit)	When the type of the watermarkColumn(java.lang.String) is Types.LongType, then sets the TimeUnit to convert the value.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

tableLoader

public IcebergSource.Builder<T> tableLoader(TableLoader loader)

table

public IcebergSource.Builder<T> table(Table newTable)

assignerFactory

public IcebergSource.Builder<T> assignerFactory(SplitAssignerFactory assignerFactory)

splitComparator

public IcebergSource.Builder<T> splitComparator(SerializableComparator<IcebergSourceSplit> newSplitComparator)

readerFunction

@Deprecated
public IcebergSource.Builder<T> readerFunction(ReaderFunction<T> newReaderFunction)

Deprecated.

since 1.7.0. Will be removed in 2.0.0; useIcebergSource.forOutputType(RowDataConverter) instead to produce output type other than RowData.

flinkConfig

public IcebergSource.Builder<T> flinkConfig(org.apache.flink.configuration.ReadableConfig config)

caseSensitive

public IcebergSource.Builder<T> caseSensitive(boolean newCaseSensitive)

useSnapshotId

public IcebergSource.Builder<T> useSnapshotId(Long newSnapshotId)

streamingStartingStrategy

public IcebergSource.Builder<T> streamingStartingStrategy(StreamingStartingStrategy newStartingStrategy)

startSnapshotTimestamp

public IcebergSource.Builder<T> startSnapshotTimestamp(Long newStartSnapshotTimestamp)

startSnapshotId

public IcebergSource.Builder<T> startSnapshotId(Long newStartSnapshotId)

tag

public IcebergSource.Builder<T> tag(String tag)

branch

public IcebergSource.Builder<T> branch(String branch)

startTag

public IcebergSource.Builder<T> startTag(String startTag)

endTag

public IcebergSource.Builder<T> endTag(String endTag)

endSnapshotId

public IcebergSource.Builder<T> endSnapshotId(Long newEndSnapshotId)

asOfTimestamp

public IcebergSource.Builder<T> asOfTimestamp(Long newAsOfTimestamp)

splitSize

public IcebergSource.Builder<T> splitSize(Long newSplitSize)

splitLookback

public IcebergSource.Builder<T> splitLookback(Integer newSplitLookback)

splitOpenFileCost

public IcebergSource.Builder<T> splitOpenFileCost(Long newSplitOpenFileCost)

streaming

public IcebergSource.Builder<T> streaming(boolean streaming)

monitorInterval

public IcebergSource.Builder<T> monitorInterval(Duration newMonitorInterval)

nameMapping

public IcebergSource.Builder<T> nameMapping(String newNameMapping)

project

public IcebergSource.Builder<T> project(Schema newProjectedSchema)

project

@Deprecated
public IcebergSource.Builder<T> project(org.apache.flink.table.legacy.api.TableSchema newProjectedFlinkSchema)

Deprecated.

since 1.10.0, will be removed in 2.0.0. Use project(ResolvedSchema) instead.

project

public IcebergSource.Builder<T> project(org.apache.flink.table.catalog.ResolvedSchema newProjectedFlinkSchema)

filters

public IcebergSource.Builder<T> filters(List<Expression> newFilters)

limit

public IcebergSource.Builder<T> limit(Long newLimit)

includeColumnStats

public IcebergSource.Builder<T> includeColumnStats(boolean newIncludeColumnStats)

planParallelism

public IcebergSource.Builder<T> planParallelism(int planParallelism)

exposeLocality

public IcebergSource.Builder<T> exposeLocality(boolean newExposeLocality)

maxAllowedPlanningFailures

public IcebergSource.Builder<T> maxAllowedPlanningFailures(int maxAllowedPlanningFailures)

set

public IcebergSource.Builder<T> set(String property,
 String value)

Set the read properties for Flink source. View the supported properties in FlinkReadOptions

setAll

public IcebergSource.Builder<T> setAll(Map<String,String> properties)

Set the read properties for Flink source. View the supported properties in FlinkReadOptions

watermarkColumn

public IcebergSource.Builder<T> watermarkColumn(String columnName)

Emits watermarks once per split based on the min value of column statistics from files metadata in the given split. The generated watermarks are also used for ordering the splits for read. Accepted column types are timestamp/timestamptz/long. For long columns consider setting watermarkColumnTimeUnit(TimeUnit).

Consider setting `read.split.open-file-cost` to prevent combining small files to a single split when the watermark is used for watermark alignment.

watermarkColumnTimeUnit

public IcebergSource.Builder<T> watermarkColumnTimeUnit(TimeUnit timeUnit)

When the type of the watermarkColumn(java.lang.String) is Types.LongType, then sets the TimeUnit to convert the value. The default value is TimeUnit.MICROSECONDS.

properties

@Deprecated
public IcebergSource.Builder<T> properties(Map<String,String> properties)

Deprecated.

will be removed in 2.0.0; use setAll(java.util.Map<java.lang.String, java.lang.String>) instead.

build

public IcebergSource<T> build()

buildStream

public org.apache.flink.streaming.api.datastream.DataStream<T> buildStream(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env)

Build the IcebergSource and create a DataStream from the source. Watermark strategy is set to WatermarkStrategy.noWatermarks().

Returns:

data stream from the Iceberg source