Package org.apache.iceberg.spark

Class SparkWriteConf

java.lang.Object

org.apache.iceberg.spark.SparkWriteConf

public class SparkWriteConf
extends java.lang.Object

A class for common Iceberg configs for Spark writes.

If a config is set at multiple levels, the following order of precedence is used (top to bottom):

1. Write options
2. Session configuration
3. Table metadata

The most specific value is set in write options and takes precedence over all other configs. If no write option is provided, this class checks the session configuration for any overrides. If no applicable value is found in the session configuration, this class uses the table metadata.

Note this class is NOT meant to be serialized and sent to executors.

Constructor Summary

Constructors

Constructor	Description
SparkWriteConf(org.apache.spark.sql.SparkSession spark, Table table, java.lang.String branch, java.util.Map<java.lang.String,java.lang.String> writeOptions)
SparkWriteConf(org.apache.spark.sql.SparkSession spark, Table table, java.util.Map<java.lang.String,java.lang.String> writeOptions)

Method Summary

Modifier and Type	Method	Description
java.lang.String	branch()
boolean	caseSensitive()
boolean	checkNullability()
boolean	checkOrdering()
DistributionMode	copyOnWriteMergeDistributionMode()
FileFormat	dataFileFormat()
DistributionMode	deleteDistributionMode()
FileFormat	deleteFileFormat()
DistributionMode	distributionMode()
java.util.Map<java.lang.String,java.lang.String>	extraSnapshotMetadata()
boolean	fanoutWriterEnabled()
boolean	handleTimestampWithoutZone()	Enables writing a timestamp with time zone as a timestamp without time zone.
IsolationLevel	isolationLevel()
boolean	mergeSchema()
java.lang.String	overwriteMode()
DistributionMode	positionDeltaMergeDistributionMode()
java.lang.String	rewrittenFileSetId()
long	targetDataFileSize()
long	targetDeleteFileSize()
DistributionMode	updateDistributionMode()
boolean	useTableDistributionAndOrdering()
java.lang.Long	validateFromSnapshotId()
boolean	wapEnabled()
java.lang.String	wapId()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkWriteConf

public SparkWriteConf(org.apache.spark.sql.SparkSession spark,
                      Table table,
                      java.util.Map<java.lang.String,java.lang.String> writeOptions)

SparkWriteConf

public SparkWriteConf(org.apache.spark.sql.SparkSession spark,
                      Table table,
                      java.lang.String branch,
                      java.util.Map<java.lang.String,java.lang.String> writeOptions)

Method Detail

checkNullability

public boolean checkNullability()

checkOrdering

public boolean checkOrdering()

handleTimestampWithoutZone

public boolean handleTimestampWithoutZone()

Enables writing a timestamp with time zone as a timestamp without time zone.

Generally, this is not safe as a timestamp without time zone is supposed to represent the wall-clock time, i.e. no matter the reader/writer timezone 3PM should always be read as 3PM, but a timestamp with time zone represents instant semantics, i.e. the timestamp is adjusted so that the corresponding time in the reader timezone is displayed.

When set to false (default), an exception must be thrown if the table contains a timestamp without time zone.

Returns:

boolean indicating if writing timestamps without timezone is allowed

overwriteMode

public java.lang.String overwriteMode()

wapEnabled

public boolean wapEnabled()

wapId

public java.lang.String wapId()

mergeSchema

public boolean mergeSchema()

dataFileFormat

public FileFormat dataFileFormat()

targetDataFileSize

public long targetDataFileSize()

fanoutWriterEnabled

public boolean fanoutWriterEnabled()

deleteFileFormat

public FileFormat deleteFileFormat()

targetDeleteFileSize

public long targetDeleteFileSize()

extraSnapshotMetadata

public java.util.Map<java.lang.String,java.lang.String> extraSnapshotMetadata()

rewrittenFileSetId

public java.lang.String rewrittenFileSetId()

distributionMode

public DistributionMode distributionMode()

deleteDistributionMode

public DistributionMode deleteDistributionMode()

updateDistributionMode

public DistributionMode updateDistributionMode()

copyOnWriteMergeDistributionMode

public DistributionMode copyOnWriteMergeDistributionMode()

positionDeltaMergeDistributionMode

public DistributionMode positionDeltaMergeDistributionMode()

useTableDistributionAndOrdering

public boolean useTableDistributionAndOrdering()

validateFromSnapshotId

public java.lang.Long validateFromSnapshotId()

isolationLevel

public IsolationLevel isolationLevel()

caseSensitive

public boolean caseSensitive()

branch

public java.lang.String branch()