Class SimpleSplitAssigner
- java.lang.Object
-
- org.apache.iceberg.flink.source.assigner.SimpleSplitAssigner
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,SplitAssigner
@Internal public class SimpleSplitAssigner extends java.lang.Object implements SplitAssigner
Since all methods are called in the source coordinator thread by enumerator, there is no need for locking.
-
-
Constructor Summary
Constructors Constructor Description SimpleSplitAssigner()
SimpleSplitAssigner(java.util.Collection<IcebergSourceSplitState> assignerState)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description GetSplitResult
getNext(java.lang.String hostname)
Request a new split from the assigner when enumerator trying to assign splits to awaiting readers.java.util.concurrent.CompletableFuture<java.lang.Void>
isAvailable()
Enumerator can get a notification via CompletableFuture when the assigner has more splits available later.void
onDiscoveredSplits(java.util.Collection<IcebergSourceSplit> splits)
Add new splits discovered by enumeratorvoid
onUnassignedSplits(java.util.Collection<IcebergSourceSplit> splits)
Forward addSplitsBack event (for failed reader) to assignerint
pendingSplitCount()
Return the number of pending splits that haven't been assigned yet.java.util.Collection<IcebergSourceSplitState>
state()
Simple assigner only tracks unassigned splits-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.iceberg.flink.source.assigner.SplitAssigner
close, onCompletedSplits, start
-
-
-
-
Constructor Detail
-
SimpleSplitAssigner
public SimpleSplitAssigner()
-
SimpleSplitAssigner
public SimpleSplitAssigner(java.util.Collection<IcebergSourceSplitState> assignerState)
-
-
Method Detail
-
getNext
public GetSplitResult getNext(@Nullable java.lang.String hostname)
Description copied from interface:SplitAssigner
Request a new split from the assigner when enumerator trying to assign splits to awaiting readers.If enumerator wasn't able to assign the split (e.g., reader disconnected), enumerator should call
SplitAssigner.onUnassignedSplits(java.util.Collection<org.apache.iceberg.flink.source.split.IcebergSourceSplit>)
to return the split.- Specified by:
getNext
in interfaceSplitAssigner
-
onDiscoveredSplits
public void onDiscoveredSplits(java.util.Collection<IcebergSourceSplit> splits)
Description copied from interface:SplitAssigner
Add new splits discovered by enumerator- Specified by:
onDiscoveredSplits
in interfaceSplitAssigner
-
onUnassignedSplits
public void onUnassignedSplits(java.util.Collection<IcebergSourceSplit> splits)
Description copied from interface:SplitAssigner
Forward addSplitsBack event (for failed reader) to assigner- Specified by:
onUnassignedSplits
in interfaceSplitAssigner
-
state
public java.util.Collection<IcebergSourceSplitState> state()
Simple assigner only tracks unassigned splits- Specified by:
state
in interfaceSplitAssigner
-
isAvailable
public java.util.concurrent.CompletableFuture<java.lang.Void> isAvailable()
Description copied from interface:SplitAssigner
Enumerator can get a notification via CompletableFuture when the assigner has more splits available later. Enumerator should schedule assignment in the thenAccept action of the future.Assigner will return the same future if this method is called again before the previous future is completed.
The future can be completed from other thread, e.g. the coordinator thread from another thread for event time alignment.
If enumerator need to trigger action upon the future completion, it may want to run it in the coordinator thread using
SplitEnumeratorContext.runInCoordinatorThread(Runnable)
.- Specified by:
isAvailable
in interfaceSplitAssigner
-
pendingSplitCount
public int pendingSplitCount()
Description copied from interface:SplitAssigner
Return the number of pending splits that haven't been assigned yet.The enumerator can poll this API to publish a metric on the number of pending splits.
The enumerator can also use this information to throttle split discovery for streaming read. If there are already many pending splits tracked by the assigner, it is undesirable to discover more splits and track them in the assigner. That will increase the memory footprint and enumerator checkpoint size.
Throttling works better together with
ScanContext.maxPlanningSnapshotCount()
. Otherwise, the next split discovery after throttling will just discover all non-enumerated snapshots and splits, which defeats the purpose of throttling.- Specified by:
pendingSplitCount
in interfaceSplitAssigner
-
-