Package org.apache.iceberg.flink.source

Class StreamingReaderOperator

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<FlinkInputSplit>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<FlinkInputSplit,​org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
    public class StreamingReaderOperator
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<FlinkInputSplit,​org.apache.flink.table.data.RowData>
    The operator that reads the splits received from the preceding StreamingMonitorFunction. Contrary to the StreamingMonitorFunction which has a parallelism of 1, this operator can have multiple parallelism.

    As soon as a split descriptor is received, it is put in a queue, and use MailboxExecutor read the actual data of the split. This architecture allows the separation of the reading thread from the one split processing the checkpoint barriers, thus removing any potential back-pressure.

    See Also:
    Serialized Form

    • Field Summary

      • Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        chainingStrategy, config, latencyStats, metrics, output, processingTimeService

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      void finish()  
      void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)  
      void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<FlinkInputSplit> element)  
      void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)  
      void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)  

      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted, notifyCheckpointComplete

      • Methods inherited from interface org.apache.flink.streaming.api.operators.Input

        processLatencyMarker, processWatermarkStatus

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

        getCurrentKey, setCurrentKey

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

        hasKeyContext

      • Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

        setKeyContextElement

      • Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

        getMetricGroup, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

    • Method Detail

      • initializeState

        public void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws java.lang.Exception
        Specified by:
        initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
        Throws:
        java.lang.Exception

      • snapshotState

        public void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws java.lang.Exception
        Specified by:
        snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
        Throws:
        java.lang.Exception

      • processElement

        public void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<FlinkInputSplit> element)
        Specified by:
        processElement in interface org.apache.flink.streaming.api.operators.Input<FlinkInputSplit>

      • processWatermark

        public void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)
        Specified by:
        processWatermark in interface org.apache.flink.streaming.api.operators.Input<FlinkInputSplit>
        Overrides:
        processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>

      • close

        public void close()
           throws java.lang.Exception
        Specified by:
        close in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>
        Overrides:
        close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
        Throws:
        java.lang.Exception

      • finish

        public void finish()
            throws java.lang.Exception
        Specified by:
        finish in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>
        Overrides:
        finish in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>
        Throws:
        java.lang.Exception