Package org.apache.iceberg.spark
Class SparkReadConf
- java.lang.Object
-
- org.apache.iceberg.spark.SparkReadConf
-
public class SparkReadConf extends java.lang.Object
A class for common Iceberg configs for Spark reads.If a config is set at multiple levels, the following order of precedence is used (top to bottom):
- Read options
- Session configuration
- Table metadata
Note this class is NOT meant to be serialized and sent to executors.
-
-
Constructor Summary
Constructors Constructor Description SparkReadConf(org.apache.spark.sql.SparkSession spark, Table table, java.lang.String branch, java.util.Map<java.lang.String,java.lang.String> readOptions)
SparkReadConf(org.apache.spark.sql.SparkSession spark, Table table, java.util.Map<java.lang.String,java.lang.String> readOptions)
-
Method Summary
All Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description boolean
aggregatePushDownEnabled()
java.lang.Long
asOfTimestamp()
java.lang.String
branch()
boolean
caseSensitive()
java.lang.Long
endSnapshotId()
java.lang.Long
endTimestamp()
java.lang.String
fileScanTaskSetId()
Deprecated.will be removed in 1.3.0, usescanTaskSetId()
insteadboolean
handleTimestampWithoutZone()
Enables reading a timestamp without time zone as a timestamp with time zone.boolean
localityEnabled()
int
orcBatchSize()
boolean
orcVectorizationEnabled()
int
parquetBatchSize()
boolean
parquetVectorizationEnabled()
boolean
preserveDataGrouping()
java.lang.String
scanTaskSetId()
java.lang.Long
snapshotId()
int
splitLookback()
java.lang.Integer
splitLookbackOption()
long
splitOpenFileCost()
java.lang.Long
splitOpenFileCostOption()
long
splitSize()
java.lang.Long
splitSizeOption()
java.lang.Long
startSnapshotId()
java.lang.Long
startTimestamp()
java.lang.Long
streamFromTimestamp()
boolean
streamingSkipDeleteSnapshots()
boolean
streamingSkipOverwriteSnapshots()
java.lang.String
tag()
-
-
-
Constructor Detail
-
SparkReadConf
public SparkReadConf(org.apache.spark.sql.SparkSession spark, Table table, java.util.Map<java.lang.String,java.lang.String> readOptions)
-
SparkReadConf
public SparkReadConf(org.apache.spark.sql.SparkSession spark, Table table, java.lang.String branch, java.util.Map<java.lang.String,java.lang.String> readOptions)
-
-
Method Detail
-
caseSensitive
public boolean caseSensitive()
-
localityEnabled
public boolean localityEnabled()
-
snapshotId
public java.lang.Long snapshotId()
-
asOfTimestamp
public java.lang.Long asOfTimestamp()
-
startSnapshotId
public java.lang.Long startSnapshotId()
-
endSnapshotId
public java.lang.Long endSnapshotId()
-
branch
public java.lang.String branch()
-
tag
public java.lang.String tag()
-
fileScanTaskSetId
@Deprecated public java.lang.String fileScanTaskSetId()
Deprecated.will be removed in 1.3.0, usescanTaskSetId()
instead
-
scanTaskSetId
public java.lang.String scanTaskSetId()
-
streamingSkipDeleteSnapshots
public boolean streamingSkipDeleteSnapshots()
-
streamingSkipOverwriteSnapshots
public boolean streamingSkipOverwriteSnapshots()
-
parquetVectorizationEnabled
public boolean parquetVectorizationEnabled()
-
parquetBatchSize
public int parquetBatchSize()
-
orcVectorizationEnabled
public boolean orcVectorizationEnabled()
-
orcBatchSize
public int orcBatchSize()
-
splitSizeOption
public java.lang.Long splitSizeOption()
-
splitSize
public long splitSize()
-
splitLookbackOption
public java.lang.Integer splitLookbackOption()
-
splitLookback
public int splitLookback()
-
splitOpenFileCostOption
public java.lang.Long splitOpenFileCostOption()
-
splitOpenFileCost
public long splitOpenFileCost()
-
handleTimestampWithoutZone
public boolean handleTimestampWithoutZone()
Enables reading a timestamp without time zone as a timestamp with time zone.Generally, this is not safe as a timestamp without time zone is supposed to represent the wall-clock time, i.e. no matter the reader/writer timezone 3PM should always be read as 3PM, but a timestamp with time zone represents instant semantics, i.e. the timestamp is adjusted so that the corresponding time in the reader timezone is displayed.
When set to false (default), an exception must be thrown while reading a timestamp without time zone.
- Returns:
- boolean indicating if reading timestamps without timezone is allowed
-
streamFromTimestamp
public java.lang.Long streamFromTimestamp()
-
startTimestamp
public java.lang.Long startTimestamp()
-
endTimestamp
public java.lang.Long endTimestamp()
-
preserveDataGrouping
public boolean preserveDataGrouping()
-
aggregatePushDownEnabled
public boolean aggregatePushDownEnabled()
-
-