Package org.apache.iceberg.flink.source
Class StreamingReaderOperator
- java.lang.Object
-
- org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
-
- org.apache.iceberg.flink.source.StreamingReaderOperator
-
- All Implemented Interfaces:
java.io.Serializable
,org.apache.flink.api.common.state.CheckpointListener
,org.apache.flink.streaming.api.operators.Input<FlinkInputSplit>
,org.apache.flink.streaming.api.operators.KeyContext
,org.apache.flink.streaming.api.operators.OneInputStreamOperator<FlinkInputSplit,org.apache.flink.table.data.RowData>
,org.apache.flink.streaming.api.operators.SetupableStreamOperator<org.apache.flink.table.data.RowData>
,org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>
,org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
public class StreamingReaderOperator extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData> implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<FlinkInputSplit,org.apache.flink.table.data.RowData>
The operator that reads thesplits
received from the precedingStreamingMonitorFunction
. Contrary to theStreamingMonitorFunction
which has a parallelism of 1, this operator can have multiple parallelism.As soon as a split descriptor is received, it is put in a queue, and use
MailboxExecutor
read the actual data of the split. This architecture allows the separation of the reading thread from the one split processing the checkpoint barriers, thus removing any potential back-pressure.- See Also:
- Serialized Form
-
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
void
finish()
void
initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
void
processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<FlinkInputSplit> element)
void
processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
void
snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
-
Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator
getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, registerCounterOnOutput, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener
notifyCheckpointAborted, notifyCheckpointComplete
-
Methods inherited from interface org.apache.flink.streaming.api.operators.Input
processLatencyMarker, processWatermarkStatus
-
Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext
getCurrentKey, setCurrentKey
-
-
-
-
Method Detail
-
initializeState
public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context) throws java.lang.Exception
- Specified by:
initializeState
in interfaceorg.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
- Overrides:
initializeState
in classorg.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
- Throws:
java.lang.Exception
-
snapshotState
public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context) throws java.lang.Exception
- Specified by:
snapshotState
in interfaceorg.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
- Overrides:
snapshotState
in classorg.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
- Throws:
java.lang.Exception
-
processElement
public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<FlinkInputSplit> element)
- Specified by:
processElement
in interfaceorg.apache.flink.streaming.api.operators.Input<FlinkInputSplit>
-
processWatermark
public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
- Specified by:
processWatermark
in interfaceorg.apache.flink.streaming.api.operators.Input<FlinkInputSplit>
- Overrides:
processWatermark
in classorg.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
-
close
public void close() throws java.lang.Exception
- Specified by:
close
in interfaceorg.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>
- Overrides:
close
in classorg.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
- Throws:
java.lang.Exception
-
finish
public void finish() throws java.lang.Exception
- Specified by:
finish
in interfaceorg.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>
- Overrides:
finish
in classorg.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
- Throws:
java.lang.Exception
-
-