Package org.apache.iceberg.flink.maintenance.operator

Class TaskResultAggregator

java.lang.Object
org.apache.flink.streaming.api.operators.AbstractStreamOperator<TaskResult>
org.apache.iceberg.flink.maintenance.operator.TaskResultAggregator
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.StreamOperator<TaskResult>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.TwoInputStreamOperator<Trigger,Exception,TaskResult>, org.apache.flink.streaming.api.operators.YieldingOperator<TaskResult>
@Internal public class TaskResultAggregator extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<TaskResult> implements org.apache.flink.streaming.api.operators.TwoInputStreamOperator<Trigger,Exception,TaskResult>
Aggregates results of the operators for a given maintenance task.
  • Input 1 is used:
    • To provide the TaskResult.startEpoch() - should be chained to the task input
    • To mark that the task is finished - should be chained at the end of the task, so an incoming watermark will signal that the task is finished
  • Input 2 expects an Exception which caused the failure - should be chained to the ERROR_STREAM of the operators
The operator emits a TaskResult with the overall result on Watermark.
See Also:

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final org.apache.flink.util.OutputTag<Exception>
    ERROR_STREAM
     

    Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

  • Constructor Summary

    Constructors
    Constructor
    Description
    TaskResultAggregator(String tableName, String taskName, int taskIndex)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    processElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<Trigger> streamRecord)
     
    void
    processElement2(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<Exception> streamRecord)
     
    void
    processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
     

    Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    beforeInitializeStateHandler, close, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isAsyncStateProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useSplittableTimers

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

    notifyCheckpointAborted, notifyCheckpointComplete

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

    getCurrentKey, setCurrentKey

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

    hasKeyContext

    Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

    close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

    Methods inherited from interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator

    processLatencyMarker1, processLatencyMarker2, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus1, processWatermarkStatus2

  • Field Details

    • ERROR_STREAM

      public static final org.apache.flink.util.OutputTag<Exception> ERROR_STREAM

  • Constructor Details

    • TaskResultAggregator

      public TaskResultAggregator(String tableName, String taskName, int taskIndex)

  • Method Details

    • processElement1

      public void processElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<Trigger> streamRecord)
      Specified by:
      processElement1 in interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator<Trigger,Exception,TaskResult>

    • processElement2

      public void processElement2(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<Exception> streamRecord)
      Specified by:
      processElement2 in interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator<Trigger,Exception,TaskResult>

    • processWatermark

      public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark) throws Exception
      Overrides:
      processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<TaskResult>
      Throws:
      Exception