org.apache.iceberg.flink.source

Class StreamingMonitorFunction

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<FlinkInputSplit>

org.apache.iceberg.flink.source.StreamingMonitorFunction

All Implemented Interfaces:

java.io.Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.SourceFunction<FlinkInputSplit>

public class StreamingMonitorFunction
extends org.apache.flink.streaming.api.functions.source.RichSourceFunction<FlinkInputSplit>
implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

This is the single (non-parallel) monitoring task which takes a FlinkInputFormat, it is responsible for:

1. Monitoring snapshots of the Iceberg table.
2. Creating the splits corresponding to the incremental files
3. Assigning them to downstream tasks for further processing.

The splits to be read are forwarded to the downstream StreamingReaderOperator which can have parallelism greater than one.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Constructor Summary

Constructors

Constructor and Description
StreamingMonitorFunction(TableLoader tableLoader, org.apache.iceberg.flink.source.ScanContext scanContext)

Method Summary

Modifier and Type	Method and Description
void	cancel()
void	close()
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<FlinkInputSplit> ctx)
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamingMonitorFunction

public StreamingMonitorFunction(TableLoader tableLoader,
                                org.apache.iceberg.flink.source.ScanContext scanContext)

Method Detail

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws java.lang.Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

java.lang.Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws java.lang.Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

java.lang.Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<FlinkInputSplit> ctx)
         throws java.lang.Exception

Specified by:

run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<FlinkInputSplit>

Throws:

java.lang.Exception

cancel

public void cancel()

Specified by:

cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<FlinkInputSplit>

close

public void close()

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction