Class SparkWriteConf


  • public class SparkWriteConf
    extends java.lang.Object
    A class for common Iceberg configs for Spark writes.

    If a config is set at multiple levels, the following order of precedence is used (top to bottom):

    1. Write options
    2. Session configuration
    3. Table metadata
    The most specific value is set in write options and takes precedence over all other configs. If no write option is provided, this class checks the session configuration for any overrides. If no applicable value is found in the session configuration, this class uses the table metadata.

    Note this class is NOT meant to be serialized and sent to executors.

    • Constructor Detail

      • SparkWriteConf

        public SparkWriteConf​(org.apache.spark.sql.SparkSession spark,
                              Table table,
                              java.util.Map<java.lang.String,​java.lang.String> writeOptions)
    • Method Detail

      • checkNullability

        public boolean checkNullability()
      • checkOrdering

        public boolean checkOrdering()
      • handleTimestampWithoutZone

        public boolean handleTimestampWithoutZone()
        Enables writing a timestamp with time zone as a timestamp without time zone.

        Generally, this is not safe as a timestamp without time zone is supposed to represent the wall-clock time, i.e. no matter the reader/writer timezone 3PM should always be read as 3PM, but a timestamp with time zone represents instant semantics, i.e. the timestamp is adjusted so that the corresponding time in the reader timezone is displayed.

        When set to false (default), an exception must be thrown if the table contains a timestamp without time zone.

        Returns:
        boolean indicating if writing timestamps without timezone is allowed
      • overwriteMode

        public java.lang.String overwriteMode()
      • wapEnabled

        public boolean wapEnabled()
      • wapId

        public java.lang.String wapId()
      • dataFileFormat

        public FileFormat dataFileFormat()
      • targetDataFileSize

        public long targetDataFileSize()
      • fanoutWriterEnabled

        public boolean fanoutWriterEnabled()
      • deleteFileFormat

        public FileFormat deleteFileFormat()
      • targetDeleteFileSize

        public long targetDeleteFileSize()
      • extraSnapshotMetadata

        public java.util.Map<java.lang.String,​java.lang.String> extraSnapshotMetadata()
      • rewrittenFileSetId

        public java.lang.String rewrittenFileSetId()
      • copyOnWriteDeleteDistributionMode

        public DistributionMode copyOnWriteDeleteDistributionMode()
      • copyOnWriteUpdateDistributionMode

        public DistributionMode copyOnWriteUpdateDistributionMode()
      • copyOnWriteMergeDistributionMode

        public DistributionMode copyOnWriteMergeDistributionMode()
      • positionDeleteDistributionMode

        public DistributionMode positionDeleteDistributionMode()
      • useTableDistributionAndOrdering

        public boolean useTableDistributionAndOrdering()