Package org.apache.iceberg.arrow.vectorized

Class ArrowReader

java.lang.Object

org.apache.iceberg.io.CloseableGroup

org.apache.iceberg.arrow.vectorized.ArrowReader

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public class ArrowReader
extends CloseableGroup

Vectorized reader that returns an iterator of ColumnarBatch. See open(CloseableIterable) ()} to learn about the behavior of the iterator.

The following Iceberg data types are supported and have been tested:

• Iceberg: Types.BooleanType, Arrow: Types.MinorType.BIT
• Iceberg: Types.IntegerType, Arrow: Types.MinorType.INT
• Iceberg: Types.LongType, Arrow: Types.MinorType.BIGINT
• Iceberg: Types.FloatType, Arrow: Types.MinorType.FLOAT4
• Iceberg: Types.DoubleType, Arrow: Types.MinorType.FLOAT8
• Iceberg: Types.StringType, Arrow: Types.MinorType.VARCHAR
• Iceberg: Types.TimestampType (both with and without timezone), Arrow: Types.MinorType.TIMEMICRO
• Iceberg: Types.BinaryType, Arrow: Types.MinorType.VARBINARY
• Iceberg: Types.DateType, Arrow: Types.MinorType.DATEDAY

Features that don't work in this implementation:

• Type promotion: In case of type promotion, the Arrow vector corresponding to the data type in the parquet file is returned instead of the data type in the latest schema. See https://github.com/apache/iceberg/issues/2483.
• Columns with constant values are physically encoded as a dictionary. The Arrow vector type is int32 instead of the type as per the schema. See https://github.com/apache/iceberg/issues/2484.
• Data types: Types.ListType, Types.MapType, Types.StructType, Types.FixedType and Types.DecimalType See https://github.com/apache/iceberg/issues/2485 and https://github.com/apache/iceberg/issues/2486.
• Iceberg v2 spec is not supported. See https://github.com/apache/iceberg/issues/2487.

Constructor Summary

Constructors

Constructor	Description
ArrowReader(TableScan scan, int batchSize, boolean reuseContainers)	Create a new instance of the reader.

Method Summary

Modifier and Type	Method	Description
void	close()
CloseableIterator<ColumnarBatch>	open(CloseableIterable<CombinedScanTask> tasks)	Returns a new iterator of ColumnarBatch objects.

Methods inherited from class org.apache.iceberg.io.CloseableGroup

addCloseable

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ArrowReader

public ArrowReader(TableScan scan,
                   int batchSize,
                   boolean reuseContainers)

Create a new instance of the reader.

Parameters:

scan - the table scan object.

batchSize - the maximum number of rows per Arrow batch.

reuseContainers - whether to reuse Arrow vectors when iterating through the data. If set to false, every Iterator.next() call creates new instances of Arrow vectors. If set to true, the Arrow vectors in the previous Iterator.next() may be reused for the data returned in the current Iterator.next(). This option avoids allocating memory again and again. Irrespective of the value of reuseContainers, the Arrow vectors in the previous Iterator.next() call are closed before creating new instances if the current Iterator.next().

Method Detail

open

public CloseableIterator<ColumnarBatch> open(CloseableIterable<CombinedScanTask> tasks)

Returns a new iterator of ColumnarBatch objects.

Note that the reader owns the ColumnarBatch objects and takes care of closing them. The caller should not hold onto a ColumnarBatch or try to close them.

If reuseContainers is false, the Arrow vectors in the previous ColumnarBatch are closed before returning the next ColumnarBatch object. This implies that the caller should either use the ColumnarBatch or transfer the ownership of ColumnarBatch before getting the next ColumnarBatch.

If reuseContainers is true, the Arrow vectors in the previous ColumnarBatch may be reused for the next ColumnarBatch. This implies that the caller should either use the ColumnarBatch or deep copy the ColumnarBatch before getting the next ColumnarBatch.

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Overrides:

close in class CloseableGroup

Throws:

java.io.IOException