org.apache.iceberg.spark

Class SparkWriteUtil

java.lang.Object

org.apache.iceberg.spark.SparkWriteUtil

public class SparkWriteUtil
extends java.lang.Object

A utility that contains helper methods for working with Spark writes.

Note it is an evolving internal API that is subject to change even in minor releases.

Method Summary

Modifier and Type	Method and Description
static SparkWriteRequirements	copyOnWriteRequirements(Table table, org.apache.spark.sql.connector.write.RowLevelOperation.Command command, DistributionMode mode, boolean fanoutEnabled, long advisoryPartitionSize) Builds requirements for copy-on-write DELETE, UPDATE, MERGE operations.
static SparkWriteRequirements	positionDeltaRequirements(Table table, org.apache.spark.sql.connector.write.RowLevelOperation.Command command, DistributionMode mode, boolean fanoutEnabled, long advisoryPartitionSize) Builds requirements for merge-on-read DELETE, UPDATE, MERGE operations.
static SparkWriteRequirements	writeRequirements(Table table, DistributionMode mode, boolean fanoutEnabled, long advisoryPartitionSize) Builds requirements for batch and micro-batch writes such as append or overwrite.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

writeRequirements

public static SparkWriteRequirements writeRequirements(Table table,
                                                       DistributionMode mode,
                                                       boolean fanoutEnabled,
                                                       long advisoryPartitionSize)

Builds requirements for batch and micro-batch writes such as append or overwrite.

copyOnWriteRequirements

public static SparkWriteRequirements copyOnWriteRequirements(Table table,
                                                             org.apache.spark.sql.connector.write.RowLevelOperation.Command command,
                                                             DistributionMode mode,
                                                             boolean fanoutEnabled,
                                                             long advisoryPartitionSize)

Builds requirements for copy-on-write DELETE, UPDATE, MERGE operations.

positionDeltaRequirements

public static SparkWriteRequirements positionDeltaRequirements(Table table,
                                                               org.apache.spark.sql.connector.write.RowLevelOperation.Command command,
                                                               DistributionMode mode,
                                                               boolean fanoutEnabled,
                                                               long advisoryPartitionSize)

Builds requirements for merge-on-read DELETE, UPDATE, MERGE operations.