org.apache.iceberg.spark

Class ChangelogIterator

java.lang.Object

org.apache.iceberg.spark.ChangelogIterator

All Implemented Interfaces:

java.util.Iterator<org.apache.spark.sql.Row>

Direct Known Subclasses:

ComputeUpdateIterator, RemoveNetCarryoverIterator

public abstract class ChangelogIterator
extends java.lang.Object
implements java.util.Iterator<org.apache.spark.sql.Row>

An iterator that transforms rows from changelog tables within a single Spark task.

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.lang.String	DELETE
protected static java.lang.String	INSERT
protected static java.lang.String	UPDATE_AFTER
protected static java.lang.String	UPDATE_BEFORE

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	ChangelogIterator(java.util.Iterator<org.apache.spark.sql.Row> rowIterator, org.apache.spark.sql.types.StructType rowType)

Method Summary

Modifier and Type	Method and Description
protected java.lang.String	changeType(org.apache.spark.sql.Row row)
protected int	changeTypeIndex()
static java.util.Iterator<org.apache.spark.sql.Row>	computeUpdates(java.util.Iterator<org.apache.spark.sql.Row> rowIterator, org.apache.spark.sql.types.StructType rowType, java.lang.String[] identifierFields) Creates an iterator composing RemoveCarryoverIterator and ComputeUpdateIterator to remove carry-over rows and compute update rows
protected static int[]	generateIndicesToIdentifySameRow(int totalColumnCount, java.util.Set<java.lang.Integer> metadataColumnIndices)
protected boolean	isDifferentValue(org.apache.spark.sql.Row currentRow, org.apache.spark.sql.Row nextRow, int idx)
protected boolean	isSameRecord(org.apache.spark.sql.Row currentRow, org.apache.spark.sql.Row nextRow, int[] indicesToIdentifySameRow)
static java.util.Iterator<org.apache.spark.sql.Row>	removeCarryovers(java.util.Iterator<org.apache.spark.sql.Row> rowIterator, org.apache.spark.sql.types.StructType rowType) Creates an iterator that removes carry-over rows from a changelog table.
static java.util.Iterator<org.apache.spark.sql.Row>	removeNetCarryovers(java.util.Iterator<org.apache.spark.sql.Row> rowIterator, org.apache.spark.sql.types.StructType rowType)
protected java.util.Iterator<org.apache.spark.sql.Row>	rowIterator()
protected org.apache.spark.sql.types.StructType	rowType()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining, hasNext, next, remove

Field Detail

DELETE

protected static final java.lang.String DELETE

INSERT

protected static final java.lang.String INSERT

UPDATE_BEFORE

protected static final java.lang.String UPDATE_BEFORE

UPDATE_AFTER

protected static final java.lang.String UPDATE_AFTER

Constructor Detail

ChangelogIterator

protected ChangelogIterator(java.util.Iterator<org.apache.spark.sql.Row> rowIterator,
                            org.apache.spark.sql.types.StructType rowType)

Method Detail

changeTypeIndex

protected int changeTypeIndex()

rowType

protected org.apache.spark.sql.types.StructType rowType()

changeType

protected java.lang.String changeType(org.apache.spark.sql.Row row)

rowIterator

protected java.util.Iterator<org.apache.spark.sql.Row> rowIterator()

computeUpdates

public static java.util.Iterator<org.apache.spark.sql.Row> computeUpdates(java.util.Iterator<org.apache.spark.sql.Row> rowIterator,
                                                                          org.apache.spark.sql.types.StructType rowType,
                                                                          java.lang.String[] identifierFields)

Creates an iterator composing RemoveCarryoverIterator and ComputeUpdateIterator to remove carry-over rows and compute update rows

Parameters:

rowIterator - the iterator of rows from a changelog table

rowType - the schema of the rows

identifierFields - the names of the identifier columns, which determine if rows are the same

Returns:

a new iterator instance

removeCarryovers

public static java.util.Iterator<org.apache.spark.sql.Row> removeCarryovers(java.util.Iterator<org.apache.spark.sql.Row> rowIterator,
                                                                            org.apache.spark.sql.types.StructType rowType)

Creates an iterator that removes carry-over rows from a changelog table.

Parameters:

rowIterator - the iterator of rows from a changelog table

rowType - the schema of the rows

Returns:

a new iterator instance

removeNetCarryovers

public static java.util.Iterator<org.apache.spark.sql.Row> removeNetCarryovers(java.util.Iterator<org.apache.spark.sql.Row> rowIterator,
                                                                               org.apache.spark.sql.types.StructType rowType)

isSameRecord

protected boolean isSameRecord(org.apache.spark.sql.Row currentRow,
                               org.apache.spark.sql.Row nextRow,
                               int[] indicesToIdentifySameRow)

isDifferentValue

protected boolean isDifferentValue(org.apache.spark.sql.Row currentRow,
                                   org.apache.spark.sql.Row nextRow,
                                   int idx)

generateIndicesToIdentifySameRow

protected static int[] generateIndicesToIdentifySameRow(int totalColumnCount,
                                                        java.util.Set<java.lang.Integer> metadataColumnIndices)