org.apache.iceberg.spark

Class SparkReadConf

java.lang.Object

org.apache.iceberg.spark.SparkReadConf

public class SparkReadConf
extends java.lang.Object

A class for common Iceberg configs for Spark reads.

If a config is set at multiple levels, the following order of precedence is used (top to bottom):

1. Read options
2. Session configuration
3. Table metadata

The most specific value is set in read options and takes precedence over all other configs. If no read option is provided, this class checks the session configuration for any overrides. If no applicable value is found in the session configuration, this class uses the table metadata.

Note this class is NOT meant to be serialized and sent to executors.

Constructor Summary

Constructors

Constructor and Description
SparkReadConf(org.apache.spark.sql.SparkSession spark, Table table, java.util.Map<java.lang.String,java.lang.String> readOptions)

Method Summary

Modifier and Type	Method and Description
java.lang.Long	asOfTimestamp()
boolean	caseSensitive()
java.lang.Long	endSnapshotId()
java.lang.String	fileScanTaskSetId()
boolean	handleTimestampWithoutZone() Enables reading a timestamp without time zone as a timestamp with time zone.
boolean	localityEnabled()
int	orcBatchSize()
boolean	orcVectorizationEnabled()
int	parquetBatchSize()
boolean	parquetVectorizationEnabled()
java.lang.Long	snapshotId()
int	splitLookback()
java.lang.Integer	splitLookbackOption()
long	splitOpenFileCost()
java.lang.Long	splitOpenFileCostOption()
long	splitSize()
java.lang.Long	splitSizeOption()
java.lang.Long	startSnapshotId()
java.lang.Long	streamFromTimestamp()
boolean	streamingSkipDeleteSnapshots()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkReadConf

public SparkReadConf(org.apache.spark.sql.SparkSession spark,
                     Table table,
                     java.util.Map<java.lang.String,java.lang.String> readOptions)

Method Detail

caseSensitive

public boolean caseSensitive()

localityEnabled

public boolean localityEnabled()

snapshotId

public java.lang.Long snapshotId()

asOfTimestamp

public java.lang.Long asOfTimestamp()

startSnapshotId

public java.lang.Long startSnapshotId()

endSnapshotId

public java.lang.Long endSnapshotId()

fileScanTaskSetId

public java.lang.String fileScanTaskSetId()

streamingSkipDeleteSnapshots

public boolean streamingSkipDeleteSnapshots()

parquetVectorizationEnabled

public boolean parquetVectorizationEnabled()

parquetBatchSize

public int parquetBatchSize()

orcVectorizationEnabled

public boolean orcVectorizationEnabled()

orcBatchSize

public int orcBatchSize()

splitSizeOption

public java.lang.Long splitSizeOption()

splitSize

public long splitSize()

splitLookbackOption

public java.lang.Integer splitLookbackOption()

splitLookback

public int splitLookback()

splitOpenFileCostOption

public java.lang.Long splitOpenFileCostOption()

splitOpenFileCost

public long splitOpenFileCost()

handleTimestampWithoutZone

public boolean handleTimestampWithoutZone()

Enables reading a timestamp without time zone as a timestamp with time zone.

Generally, this is not safe as a timestamp without time zone is supposed to represent the wall-clock time, i.e. no matter the reader/writer timezone 3PM should always be read as 3PM, but a timestamp with time zone represents instant semantics, i.e. the timestamp is adjusted so that the corresponding time in the reader timezone is displayed.

When set to false (default), an exception must be thrown while reading a timestamp without time zone.

Returns:

boolean indicating if reading timestamps without timezone is allowed

streamFromTimestamp

public java.lang.Long streamFromTimestamp()