org.apache.iceberg.flink.source.assigner

Class DefaultSplitAssigner

java.lang.Object

org.apache.iceberg.flink.source.assigner.DefaultSplitAssigner

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, SplitAssigner

@Internal
public class DefaultSplitAssigner
extends java.lang.Object
implements SplitAssigner

Since all methods are called in the source coordinator thread by enumerator, there is no need for locking.

Constructor Summary

Constructors

Constructor and Description
DefaultSplitAssigner(SerializableComparator<IcebergSourceSplit> comparator)
DefaultSplitAssigner(SerializableComparator<IcebergSourceSplit> comparator, java.util.Collection<IcebergSourceSplitState> assignerState)

Method Summary

Modifier and Type	Method and Description
GetSplitResult	getNext(java.lang.String hostname) Request a new split from the assigner when enumerator trying to assign splits to awaiting readers.
java.util.concurrent.CompletableFuture<java.lang.Void>	isAvailable() Enumerator can get a notification via CompletableFuture when the assigner has more splits available later.
void	onDiscoveredSplits(java.util.Collection<IcebergSourceSplit> splits) Add new splits discovered by enumerator
void	onUnassignedSplits(java.util.Collection<IcebergSourceSplit> splits) Forward addSplitsBack event (for failed reader) to assigner
long	pendingRecords() Return the number of pending records, which can act as a measure of the source lag.
int	pendingSplitCount() Return the number of pending splits that haven't been assigned yet.
java.util.Collection<IcebergSourceSplitState>	state() Simple assigner only tracks unassigned splits

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.flink.source.assigner.SplitAssigner

close, onCompletedSplits, start

Constructor Detail

DefaultSplitAssigner

public DefaultSplitAssigner(SerializableComparator<IcebergSourceSplit> comparator)

DefaultSplitAssigner

public DefaultSplitAssigner(SerializableComparator<IcebergSourceSplit> comparator,
                            java.util.Collection<IcebergSourceSplitState> assignerState)

Method Detail

getNext

public GetSplitResult getNext(@Nullable
                              java.lang.String hostname)

Description copied from interface: SplitAssigner

Request a new split from the assigner when enumerator trying to assign splits to awaiting readers.

If enumerator wasn't able to assign the split (e.g., reader disconnected), enumerator should call SplitAssigner.onUnassignedSplits(java.util.Collection<org.apache.iceberg.flink.source.split.IcebergSourceSplit>) to return the split.

Specified by:

getNext in interface SplitAssigner

onDiscoveredSplits

public void onDiscoveredSplits(java.util.Collection<IcebergSourceSplit> splits)

Description copied from interface: SplitAssigner

Add new splits discovered by enumerator

Specified by:

onDiscoveredSplits in interface SplitAssigner

onUnassignedSplits

public void onUnassignedSplits(java.util.Collection<IcebergSourceSplit> splits)

Description copied from interface: SplitAssigner

Forward addSplitsBack event (for failed reader) to assigner

Specified by:

onUnassignedSplits in interface SplitAssigner

state

public java.util.Collection<IcebergSourceSplitState> state()

Simple assigner only tracks unassigned splits

Specified by:

state in interface SplitAssigner

isAvailable

public java.util.concurrent.CompletableFuture<java.lang.Void> isAvailable()

Description copied from interface: SplitAssigner

Enumerator can get a notification via CompletableFuture when the assigner has more splits available later. Enumerator should schedule assignment in the thenAccept action of the future.

Assigner will return the same future if this method is called again before the previous future is completed.

The future can be completed from other thread, e.g. the coordinator thread from another thread for event time alignment.

If enumerator need to trigger action upon the future completion, it may want to run it in the coordinator thread using SplitEnumeratorContext.runInCoordinatorThread(Runnable).

Specified by:

isAvailable in interface SplitAssigner

pendingSplitCount

public int pendingSplitCount()

Description copied from interface: SplitAssigner

Return the number of pending splits that haven't been assigned yet.

The enumerator can poll this API to publish a metric on the number of pending splits.

The enumerator can also use this information to throttle split discovery for streaming read. If there are already many pending splits tracked by the assigner, it is undesirable to discover more splits and track them in the assigner. That will increase the memory footprint and enumerator checkpoint size.

Throttling works better together with ScanContext.maxPlanningSnapshotCount(). Otherwise, the next split discovery after throttling will just discover all non-enumerated snapshots and splits, which defeats the purpose of throttling.

Specified by:

pendingSplitCount in interface SplitAssigner

pendingRecords

public long pendingRecords()

Description copied from interface: SplitAssigner

Return the number of pending records, which can act as a measure of the source lag. This value could be an estimation if the exact number of records cannot be accurately computed.

Specified by:

pendingRecords in interface SplitAssigner