org.apache.iceberg.spark

Class Spark3Util

java.lang.Object

org.apache.iceberg.spark.Spark3Util

public class Spark3Util
extends java.lang.Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Spark3Util.DescribeSchemaVisitor

Method Summary

Modifier and Type	Method and Description
static UpdateProperties	applyPropertyChanges(UpdateProperties pendingUpdate, java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes) Applies a list of Spark table changes to an UpdateProperties operation.
static UpdateSchema	applySchemaChanges(UpdateSchema pendingUpdate, java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes) Applies a list of Spark table changes to an UpdateSchema operation.
static int	batchSize(java.util.Map<java.lang.String,java.lang.String> properties, org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)
static java.lang.String	describe(Expression expr)
static java.lang.String	describe(Schema schema)
static java.lang.String	describe(Type type)
static boolean	isLocalityEnabled(FileIO io, java.lang.String location, org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)
static boolean	isVectorizationEnabled(java.util.Map<java.lang.String,java.lang.String> properties, org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)
static java.lang.Integer	propertyAsInt(org.apache.spark.sql.util.CaseInsensitiveStringMap options, java.lang.String property, java.lang.Integer defaultValue)
static java.lang.Long	propertyAsLong(org.apache.spark.sql.util.CaseInsensitiveStringMap options, java.lang.String property, java.lang.Long defaultValue)
static java.util.Map<java.lang.String,java.lang.String>	rebuildCreateProperties(java.util.Map<java.lang.String,java.lang.String> createProperties)
static PartitionSpec	toPartitionSpec(Schema schema, org.apache.spark.sql.connector.expressions.Transform[] partitioning) Converts Spark transforms into a PartitionSpec.
static org.apache.spark.sql.connector.expressions.Transform[]	toTransforms(PartitionSpec spec) Converts a PartitionSpec to Spark transforms.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

rebuildCreateProperties

public static java.util.Map<java.lang.String,java.lang.String> rebuildCreateProperties(java.util.Map<java.lang.String,java.lang.String> createProperties)

applyPropertyChanges

public static UpdateProperties applyPropertyChanges(UpdateProperties pendingUpdate,
                                                    java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes)

Applies a list of Spark table changes to an UpdateProperties operation.

Parameters:

pendingUpdate - an uncommitted UpdateProperties operation to configure

changes - a list of Spark table changes

Returns:

the UpdateProperties operation configured with the changes

applySchemaChanges

public static UpdateSchema applySchemaChanges(UpdateSchema pendingUpdate,
                                              java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes)

Applies a list of Spark table changes to an UpdateSchema operation.

Parameters:

pendingUpdate - an uncommitted UpdateSchema operation to configure

changes - a list of Spark table changes

Returns:

the UpdateSchema operation configured with the changes

toTransforms

public static org.apache.spark.sql.connector.expressions.Transform[] toTransforms(PartitionSpec spec)

Converts a PartitionSpec to Spark transforms.

Parameters:

spec - a PartitionSpec

Returns:

an array of Transforms

toPartitionSpec

public static PartitionSpec toPartitionSpec(Schema schema,
                                            org.apache.spark.sql.connector.expressions.Transform[] partitioning)

Converts Spark transforms into a PartitionSpec.

Parameters:

schema - the table schema

partitioning - Spark Transforms

Returns:

a PartitionSpec

describe

public static java.lang.String describe(Expression expr)

describe

public static java.lang.String describe(Schema schema)

describe

public static java.lang.String describe(Type type)

isLocalityEnabled

public static boolean isLocalityEnabled(FileIO io,
                                        java.lang.String location,
                                        org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)

isVectorizationEnabled

public static boolean isVectorizationEnabled(java.util.Map<java.lang.String,java.lang.String> properties,
                                             org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)

batchSize

public static int batchSize(java.util.Map<java.lang.String,java.lang.String> properties,
                            org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)

propertyAsLong

public static java.lang.Long propertyAsLong(org.apache.spark.sql.util.CaseInsensitiveStringMap options,
                                            java.lang.String property,
                                            java.lang.Long defaultValue)

propertyAsInt

public static java.lang.Integer propertyAsInt(org.apache.spark.sql.util.CaseInsensitiveStringMap options,
                                              java.lang.String property,
                                              java.lang.Integer defaultValue)