Class SparkWriteConf

  • public class SparkWriteConf
    extends java.lang.Object
    A class for common Iceberg configs for Spark writes.

    If a config is set at multiple levels, the following order of precedence is used (top to bottom):

    1. Write options
    2. Session configuration
    3. Table metadata
    The most specific value is set in write options and takes precedence over all other configs. If no write option is provided, this class checks the session configuration for any overrides. If no applicable value is found in the session configuration, this class uses the table metadata.

    Note this class is NOT meant to be serialized and sent to executors.

    • Constructor Detail

      • SparkWriteConf

        public SparkWriteConf​(org.apache.spark.sql.SparkSession spark,
                              Table table,
                              java.util.Map<java.lang.String,​java.lang.String> writeOptions)
      • SparkWriteConf

        public SparkWriteConf​(org.apache.spark.sql.SparkSession spark,
                              Table table,
                              java.lang.String branch,
                              java.util.Map<java.lang.String,​java.lang.String> writeOptions)
    • Method Detail

      • checkNullability

        public boolean checkNullability()
      • checkOrdering

        public boolean checkOrdering()
      • overwriteMode

        public java.lang.String overwriteMode()
      • wapEnabled

        public boolean wapEnabled()
      • wapId

        public java.lang.String wapId()
      • mergeSchema

        public boolean mergeSchema()
      • outputSpecId

        public int outputSpecId()
      • dataFileFormat

        public FileFormat dataFileFormat()
      • targetDataFileSize

        public long targetDataFileSize()
      • deleteFileFormat

        public FileFormat deleteFileFormat()
      • targetDeleteFileSize

        public long targetDeleteFileSize()
      • extraSnapshotMetadata

        public java.util.Map<java.lang.String,​java.lang.String> extraSnapshotMetadata()
      • rewrittenFileSetId

        public java.lang.String rewrittenFileSetId()
      • copyOnWriteRequirements

        public SparkWriteRequirements copyOnWriteRequirements​(org.apache.spark.sql.connector.write.RowLevelOperation.Command command)
      • positionDeltaRequirements

        public SparkWriteRequirements positionDeltaRequirements​(org.apache.spark.sql.connector.write.RowLevelOperation.Command command)
      • validateFromSnapshotId

        public java.lang.Long validateFromSnapshotId()
      • caseSensitive

        public boolean caseSensitive()
      • branch

        public java.lang.String branch()
      • writeProperties

        public java.util.Map<java.lang.String,​java.lang.String> writeProperties()