org.apache.iceberg.spark

Class ChangelogIterator

java.lang.Object

org.apache.iceberg.spark.ChangelogIterator

All Implemented Interfaces:

java.util.Iterator<org.apache.spark.sql.Row>

Direct Known Subclasses:

ComputeUpdateIterator

public abstract class ChangelogIterator
extends java.lang.Object
implements java.util.Iterator<org.apache.spark.sql.Row>

An iterator that transforms rows from changelog tables within a single Spark task.

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.lang.String	DELETE
protected static java.lang.String	INSERT
protected static java.lang.String	UPDATE_AFTER
protected static java.lang.String	UPDATE_BEFORE

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	ChangelogIterator(java.util.Iterator<org.apache.spark.sql.Row> rowIterator, org.apache.spark.sql.types.StructType rowType)

Method Summary

Modifier and Type	Method and Description
protected int	changeTypeIndex()
static java.util.Iterator<org.apache.spark.sql.Row>	computeUpdates(java.util.Iterator<org.apache.spark.sql.Row> rowIterator, org.apache.spark.sql.types.StructType rowType, java.lang.String[] identifierFields) Creates an iterator composing RemoveCarryoverIterator and ComputeUpdateIterator to remove carry-over rows and compute update rows
protected boolean	isDifferentValue(org.apache.spark.sql.Row currentRow, org.apache.spark.sql.Row nextRow, int idx)
static java.util.Iterator<org.apache.spark.sql.Row>	removeCarryovers(java.util.Iterator<org.apache.spark.sql.Row> rowIterator, org.apache.spark.sql.types.StructType rowType) Creates an iterator that removes carry-over rows from a changelog table.
protected java.util.Iterator<org.apache.spark.sql.Row>	rowIterator()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining, hasNext, next, remove

Field Detail

DELETE

protected static final java.lang.String DELETE

INSERT

protected static final java.lang.String INSERT

UPDATE_BEFORE

protected static final java.lang.String UPDATE_BEFORE

UPDATE_AFTER

protected static final java.lang.String UPDATE_AFTER

Constructor Detail

ChangelogIterator

protected ChangelogIterator(java.util.Iterator<org.apache.spark.sql.Row> rowIterator,
                            org.apache.spark.sql.types.StructType rowType)

Method Detail

changeTypeIndex

protected int changeTypeIndex()

rowIterator

protected java.util.Iterator<org.apache.spark.sql.Row> rowIterator()

computeUpdates

public static java.util.Iterator<org.apache.spark.sql.Row> computeUpdates(java.util.Iterator<org.apache.spark.sql.Row> rowIterator,
                                                                          org.apache.spark.sql.types.StructType rowType,
                                                                          java.lang.String[] identifierFields)

Creates an iterator composing RemoveCarryoverIterator and ComputeUpdateIterator to remove carry-over rows and compute update rows

Parameters:

rowIterator - the iterator of rows from a changelog table

rowType - the schema of the rows

identifierFields - the names of the identifier columns, which determine if rows are the same

Returns:

a new iterator instance

removeCarryovers

public static java.util.Iterator<org.apache.spark.sql.Row> removeCarryovers(java.util.Iterator<org.apache.spark.sql.Row> rowIterator,
                                                                            org.apache.spark.sql.types.StructType rowType)

Creates an iterator that removes carry-over rows from a changelog table.

Parameters:

rowIterator - the iterator of rows from a changelog table

rowType - the schema of the rows

Returns:

a new iterator instance

isDifferentValue

protected boolean isDifferentValue(org.apache.spark.sql.Row currentRow,
                                   org.apache.spark.sql.Row nextRow,
                                   int idx)