Package org.apache.iceberg.spark

Class SparkWriteConf

java.lang.Object
org.apache.iceberg.spark.SparkWriteConf
public class SparkWriteConf extends Object
A class for common Iceberg configs for Spark writes.

If a config is set at multiple levels, the following order of precedence is used (top to bottom):

  1. Write options
  2. Session configuration
  3. Table metadata
The most specific value is set in write options and takes precedence over all other configs. If no write option is provided, this class checks the session configuration for any overrides. If no applicable value is found in the session configuration, this class uses the table metadata.

Note this class is NOT meant to be serialized and sent to executors.

  • Constructor Details

    • SparkWriteConf

      public SparkWriteConf(org.apache.spark.sql.SparkSession spark, Table table)

    • SparkWriteConf

      public SparkWriteConf(org.apache.spark.sql.SparkSession spark, Table table, org.apache.spark.sql.util.CaseInsensitiveStringMap options)

    • SparkWriteConf

      @Deprecated public SparkWriteConf(org.apache.spark.sql.SparkSession spark, Table table, String branch, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
      Deprecated.
      since 1.11.0, will be removed in 1.12.0. Use SparkWriteConf(SparkSession, Table, CaseInsensitiveStringMap) instead.

  • Method Details

    • checkNullability

      public boolean checkNullability()

    • checkOrdering

      public boolean checkOrdering()

    • overwriteMode

      public String overwriteMode()

    • wapEnabled

      public boolean wapEnabled()

    • wapId

      public String wapId()

    • mergeSchema

      public boolean mergeSchema()

    • outputSpecId

      public int outputSpecId()

    • outputSortOrderId

      public int outputSortOrderId(SparkWriteRequirements writeRequirements)

    • dataFileFormat

      public FileFormat dataFileFormat()

    • targetDataFileSize

      public long targetDataFileSize()

    • useFanoutWriter

      public boolean useFanoutWriter(SparkWriteRequirements writeRequirements)

    • deleteFileFormat

      public FileFormat deleteFileFormat()

    • targetDeleteFileSize

      public long targetDeleteFileSize()

    • extraSnapshotMetadata

      public Map<String,String> extraSnapshotMetadata()

    • writeRequirements

      public SparkWriteRequirements writeRequirements()

    • copyOnWriteRequirements

      public SparkWriteRequirements copyOnWriteRequirements(org.apache.spark.sql.connector.write.RowLevelOperation.Command command)

    • positionDeltaRequirements

      public SparkWriteRequirements positionDeltaRequirements(org.apache.spark.sql.connector.write.RowLevelOperation.Command command)

    • validateFromSnapshotId

      public Long validateFromSnapshotId()

    • isolationLevel

      public IsolationLevel isolationLevel()

    • caseSensitive

      public boolean caseSensitive()

    • writeProperties

      public Map<String,String> writeProperties()

    • deleteGranularity

      public DeleteGranularity deleteGranularity()

    • shredVariants

      public boolean shredVariants()

    • variantInferenceBufferSize

      public int variantInferenceBufferSize()