Package org.apache.iceberg.spark

Class SparkWriteConf

java.lang.Object

org.apache.iceberg.spark.SparkWriteConf

public class SparkWriteConf
extends java.lang.Object

A class for common Iceberg configs for Spark writes.

If a config is set at multiple levels, the following order of precedence is used (top to bottom):

1. Write options
2. Session configuration
3. Table metadata

The most specific value is set in write options and takes precedence over all other configs. If no write option is provided, this class checks the session configuration for any overrides. If no applicable value is found in the session configuration, this class uses the table metadata.

Note this class is NOT meant to be serialized and sent to executors.

Constructor Summary

Constructors

Constructor	Description
SparkWriteConf(org.apache.spark.sql.SparkSession spark, Table table, java.util.Map<java.lang.String,java.lang.String> writeOptions)

Method Summary

Modifier and Type	Method	Description
boolean	checkNullability()
boolean	checkOrdering()
DistributionMode	copyOnWriteDeleteDistributionMode()
DistributionMode	copyOnWriteMergeDistributionMode()
DistributionMode	copyOnWriteUpdateDistributionMode()
FileFormat	dataFileFormat()
FileFormat	deleteFileFormat()
DistributionMode	distributionMode()
java.util.Map<java.lang.String,java.lang.String>	extraSnapshotMetadata()
boolean	fanoutWriterEnabled()
boolean	handleTimestampWithoutZone()	Enables writing a timestamp with time zone as a timestamp without time zone.
java.lang.String	overwriteMode()
DistributionMode	positionDeleteDistributionMode()
java.lang.String	rewrittenFileSetId()
long	targetDataFileSize()
long	targetDeleteFileSize()
boolean	useTableDistributionAndOrdering()
boolean	wapEnabled()
java.lang.String	wapId()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkWriteConf

public SparkWriteConf(org.apache.spark.sql.SparkSession spark,
                      Table table,
                      java.util.Map<java.lang.String,java.lang.String> writeOptions)

Method Detail

checkNullability

public boolean checkNullability()

checkOrdering

public boolean checkOrdering()

handleTimestampWithoutZone

public boolean handleTimestampWithoutZone()

Enables writing a timestamp with time zone as a timestamp without time zone.

Generally, this is not safe as a timestamp without time zone is supposed to represent the wall-clock time, i.e. no matter the reader/writer timezone 3PM should always be read as 3PM, but a timestamp with time zone represents instant semantics, i.e. the timestamp is adjusted so that the corresponding time in the reader timezone is displayed.

When set to false (default), an exception must be thrown if the table contains a timestamp without time zone.

Returns:

boolean indicating if writing timestamps without timezone is allowed

overwriteMode

public java.lang.String overwriteMode()

wapEnabled

public boolean wapEnabled()

wapId

public java.lang.String wapId()

dataFileFormat

public FileFormat dataFileFormat()

targetDataFileSize

public long targetDataFileSize()

fanoutWriterEnabled

public boolean fanoutWriterEnabled()

deleteFileFormat

public FileFormat deleteFileFormat()

targetDeleteFileSize

public long targetDeleteFileSize()

extraSnapshotMetadata

public java.util.Map<java.lang.String,java.lang.String> extraSnapshotMetadata()

rewrittenFileSetId

public java.lang.String rewrittenFileSetId()

distributionMode

public DistributionMode distributionMode()

copyOnWriteDeleteDistributionMode

public DistributionMode copyOnWriteDeleteDistributionMode()

copyOnWriteUpdateDistributionMode

public DistributionMode copyOnWriteUpdateDistributionMode()

copyOnWriteMergeDistributionMode

public DistributionMode copyOnWriteMergeDistributionMode()

positionDeleteDistributionMode

public DistributionMode positionDeleteDistributionMode()

useTableDistributionAndOrdering

public boolean useTableDistributionAndOrdering()