Package org.apache.iceberg.flink.source

Class StreamingMonitorFunction

java.lang.Object
org.apache.flink.api.common.functions.AbstractRichFunction
org.apache.flink.streaming.api.functions.source.RichSourceFunction<FlinkInputSplit>
org.apache.iceberg.flink.source.StreamingMonitorFunction
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.SourceFunction<FlinkInputSplit>
public class StreamingMonitorFunction extends org.apache.flink.streaming.api.functions.source.RichSourceFunction<FlinkInputSplit> implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
This is the single (non-parallel) monitoring task which takes a FlinkInputFormat, it is responsible for:
  1. Monitoring snapshots of the Iceberg table.
  2. Creating the splits corresponding to the incremental files
  3. Assigning them to downstream tasks for further processing.

The splits to be read are forwarded to the downstream StreamingReaderOperator which can have parallelism greater than one.

See Also:

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

    org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

  • Constructor Summary

    Constructors
    Constructor
    Description
    StreamingMonitorFunction(TableLoader tableLoader, ScanContext scanContext)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    cancel()
     
    void
    close()
     
    void
    initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
     
    void
    open(org.apache.flink.configuration.Configuration parameters)
     
    void
    run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<FlinkInputSplit> ctx)
     
    void
    snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
     

    Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

    getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.flink.api.common.functions.RichFunction

    open

  • Constructor Details

    • StreamingMonitorFunction

      public StreamingMonitorFunction(TableLoader tableLoader, ScanContext scanContext)

  • Method Details

    • open

      public void open(org.apache.flink.configuration.Configuration parameters) throws Exception
      Specified by:
      open in interface org.apache.flink.api.common.functions.RichFunction
      Overrides:
      open in class org.apache.flink.api.common.functions.AbstractRichFunction
      Throws:
      Exception

    • initializeState

      public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context) throws Exception
      Specified by:
      initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
      Throws:
      Exception

    • snapshotState

      public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context) throws Exception
      Specified by:
      snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
      Throws:
      Exception

    • run

      public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<FlinkInputSplit> ctx) throws Exception
      Specified by:
      run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<FlinkInputSplit>
      Throws:
      Exception

    • cancel

      public void cancel()
      Specified by:
      cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<FlinkInputSplit>

    • close

      public void close()
      Specified by:
      close in interface org.apache.flink.api.common.functions.RichFunction
      Overrides:
      close in class org.apache.flink.api.common.functions.AbstractRichFunction