Class ArrowReader

  • All Implemented Interfaces:, java.lang.AutoCloseable

    public class ArrowReader
    extends CloseableGroup
    Vectorized reader that returns an iterator of ColumnarBatch. See open(CloseableIterable) ()} to learn about the behavior of the iterator.

    The following Iceberg data types are supported and have been tested:

    Features that don't work in this implementation:

    • Type promotion: In case of type promotion, the Arrow vector corresponding to the data type in the parquet file is returned instead of the data type in the latest schema. See
    • Columns with constant values are physically encoded as a dictionary. The Arrow vector type is int32 instead of the type as per the schema. See
    • Data types: Types.ListType, Types.MapType, Types.StructType, Types.FixedType and Types.DecimalType See and
    • Delete files are not supported. See
    • Constructor Detail

      • ArrowReader

        public ArrowReader​(TableScan scan,
                           int batchSize,
                           boolean reuseContainers)
        Create a new instance of the reader.
        scan - the table scan object.
        batchSize - the maximum number of rows per Arrow batch.
        reuseContainers - whether to reuse Arrow vectors when iterating through the data. If set to false, every call creates new instances of Arrow vectors. If set to true, the Arrow vectors in the previous may be reused for the data returned in the current This option avoids allocating memory again and again. Irrespective of the value of reuseContainers, the Arrow vectors in the previous call are closed before creating new instances if the current
    • Method Detail

      • close

        public void close()
        Description copied from class: CloseableGroup
        Close all the registered resources. Close method of each resource will only be called once. Checked exception from AutoCloseable will be wrapped to runtime exception.
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface
        close in class CloseableGroup