Class DefaultSplitAssigner

  • All Implemented Interfaces:, java.lang.AutoCloseable, SplitAssigner

    public class DefaultSplitAssigner
    extends java.lang.Object
    implements SplitAssigner
    Since all methods are called in the source coordinator thread by enumerator, there is no need for locking.
    • Method Detail

      • isAvailable

        public java.util.concurrent.CompletableFuture<java.lang.Void> isAvailable()
        Description copied from interface: SplitAssigner
        Enumerator can get a notification via CompletableFuture when the assigner has more splits available later. Enumerator should schedule assignment in the thenAccept action of the future.

        Assigner will return the same future if this method is called again before the previous future is completed.

        The future can be completed from other thread, e.g. the coordinator thread from another thread for event time alignment.

        If enumerator need to trigger action upon the future completion, it may want to run it in the coordinator thread using SplitEnumeratorContext.runInCoordinatorThread(Runnable).

        Specified by:
        isAvailable in interface SplitAssigner
      • pendingSplitCount

        public int pendingSplitCount()
        Description copied from interface: SplitAssigner
        Return the number of pending splits that haven't been assigned yet.

        The enumerator can poll this API to publish a metric on the number of pending splits.

        The enumerator can also use this information to throttle split discovery for streaming read. If there are already many pending splits tracked by the assigner, it is undesirable to discover more splits and track them in the assigner. That will increase the memory footprint and enumerator checkpoint size.

        Throttling works better together with ScanContext.maxPlanningSnapshotCount(). Otherwise, the next split discovery after throttling will just discover all non-enumerated snapshots and splits, which defeats the purpose of throttling.

        Specified by:
        pendingSplitCount in interface SplitAssigner
      • pendingRecords

        public long pendingRecords()
        Description copied from interface: SplitAssigner
        Return the number of pending records, which can act as a measure of the source lag. This value could be an estimation if the exact number of records cannot be accurately computed.
        Specified by:
        pendingRecords in interface SplitAssigner