Package org.apache.iceberg.flink.source

Class StreamingMonitorFunction

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.SourceFunction<FlinkInputSplit>
    public class StreamingMonitorFunction
extends org.apache.flink.streaming.api.functions.source.RichSourceFunction<FlinkInputSplit>
implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
    This is the single (non-parallel) monitoring task which takes a FlinkInputFormat, it is responsible for:
    1. Monitoring snapshots of the Iceberg table.
    2. Creating the splits corresponding to the incremental files
    3. Assigning them to downstream tasks for further processing.

    The splits to be read are forwarded to the downstream StreamingReaderOperator which can have parallelism greater than one.

    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

        org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T extends java.lang.Object>

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void cancel()  
      void close()  
      void initializeState​(org.apache.flink.runtime.state.FunctionInitializationContext context)  
      void run​(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<FlinkInputSplit> ctx)  
      void snapshotState​(org.apache.flink.runtime.state.FunctionSnapshotContext context)  

      • Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

        getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • StreamingMonitorFunction

        public StreamingMonitorFunction​(TableLoader tableLoader,
                                org.apache.iceberg.flink.source.ScanContext scanContext)

    • Method Detail

      • initializeState

        public void initializeState​(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws java.lang.Exception
        Specified by:
        initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
        Throws:
        java.lang.Exception

      • snapshotState

        public void snapshotState​(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws java.lang.Exception
        Specified by:
        snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
        Throws:
        java.lang.Exception

      • run

        public void run​(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<FlinkInputSplit> ctx)
         throws java.lang.Exception
        Specified by:
        run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<FlinkInputSplit>
        Throws:
        java.lang.Exception

      • cancel

        public void cancel()
        Specified by:
        cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<FlinkInputSplit>

      • close

        public void close()
        Specified by:
        close in interface org.apache.flink.api.common.functions.RichFunction
        Overrides:
        close in class org.apache.flink.api.common.functions.AbstractRichFunction