Package org.apache.iceberg.flink.maintenance.operator

Class TriggerManager

java.lang.Object
org.apache.flink.api.common.functions.AbstractRichFunction
org.apache.flink.streaming.api.functions.KeyedProcessFunction<Boolean,TableChange,Trigger>
org.apache.iceberg.flink.maintenance.operator.TriggerManager
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
@Internal public class TriggerManager extends org.apache.flink.streaming.api.functions.KeyedProcessFunction<Boolean,TableChange,Trigger> implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
TriggerManager starts the Maintenance Tasks by emitting Trigger messages which are calculated based on the incoming TableChange messages. The TriggerManager keeps track of the changes since the last run of the Maintenance Tasks and triggers a new run based on the result of the TriggerEvaluator.

The TriggerManager prevents overlapping Maintenance Task runs using TriggerLockFactory.Lock. The current implementation only handles conflicts within a single job. Users should avoid scheduling maintenance for the same table in different Flink jobs.

The TriggerManager should run as a global operator. KeyedProcessFunction is used, so the timer functions are available, but the key is not used.

See Also:

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.KeyedProcessFunction

    org.apache.flink.streaming.api.functions.KeyedProcessFunction.Context, org.apache.flink.streaming.api.functions.KeyedProcessFunction.OnTimerContext

  • Constructor Summary

    Constructors
    Constructor
    Description
    TriggerManager(TableLoader tableLoader, TriggerLockFactory lockFactory, List<String> maintenanceTaskNames, List<TriggerEvaluator> evaluators, long minFireDelayMs, long lockCheckDelayMs)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    close()
     
    void
    initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
     
    void
    onTimer(long timestamp, org.apache.flink.streaming.api.functions.KeyedProcessFunction<Boolean,TableChange,Trigger>.org.apache.flink.streaming.api.functions.KeyedProcessFunction.OnTimerContext ctx, org.apache.flink.util.Collector<Trigger> out)
     
    void
    open(org.apache.flink.api.common.functions.OpenContext parameters)
     
    void
    processElement(TableChange change, org.apache.flink.streaming.api.functions.KeyedProcessFunction<Boolean,TableChange,Trigger>.org.apache.flink.streaming.api.functions.KeyedProcessFunction.Context ctx, org.apache.flink.util.Collector<Trigger> out)
     
    void
    snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
     

    Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

    getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

  • Method Details

    • open

      public void open(org.apache.flink.api.common.functions.OpenContext parameters) throws Exception
      Specified by:
      open in interface org.apache.flink.api.common.functions.RichFunction
      Overrides:
      open in class org.apache.flink.api.common.functions.AbstractRichFunction
      Throws:
      Exception

    • snapshotState

      public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context) throws Exception
      Specified by:
      snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
      Throws:
      Exception

    • initializeState

      public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context) throws Exception
      Specified by:
      initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
      Throws:
      Exception

    • processElement

      public void processElement(TableChange change, org.apache.flink.streaming.api.functions.KeyedProcessFunction<Boolean,TableChange,Trigger>.org.apache.flink.streaming.api.functions.KeyedProcessFunction.Context ctx, org.apache.flink.util.Collector<Trigger> out) throws Exception
      Specified by:
      processElement in class org.apache.flink.streaming.api.functions.KeyedProcessFunction<Boolean,TableChange,Trigger>
      Throws:
      Exception

    • onTimer

      public void onTimer(long timestamp, org.apache.flink.streaming.api.functions.KeyedProcessFunction<Boolean,TableChange,Trigger>.org.apache.flink.streaming.api.functions.KeyedProcessFunction.OnTimerContext ctx, org.apache.flink.util.Collector<Trigger> out) throws Exception
      Overrides:
      onTimer in class org.apache.flink.streaming.api.functions.KeyedProcessFunction<Boolean,TableChange,Trigger>
      Throws:
      Exception

    • close

      public void close() throws IOException
      Specified by:
      close in interface org.apache.flink.api.common.functions.RichFunction
      Overrides:
      close in class org.apache.flink.api.common.functions.AbstractRichFunction
      Throws:
      IOException