org.apache.iceberg.spark

Class Spark3Util

java.lang.Object

org.apache.iceberg.spark.Spark3Util

public class Spark3Util
extends java.lang.Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Spark3Util.CatalogAndIdentifier This mimics a class inside of Spark which is private inside of LookupCatalog.
static class	Spark3Util.DescribeSchemaVisitor

Method Summary

Modifier and Type	Method and Description
static UpdateProperties	applyPropertyChanges(UpdateProperties pendingUpdate, java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes) Applies a list of Spark table changes to an UpdateProperties operation.
static UpdateSchema	applySchemaChanges(UpdateSchema pendingUpdate, java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes) Applies a list of Spark table changes to an UpdateSchema operation.
static int	batchSize(java.util.Map<java.lang.String,java.lang.String> properties, org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)
static Spark3Util.CatalogAndIdentifier	catalogAndIdentifier(org.apache.spark.sql.SparkSession spark, java.util.List<java.lang.String> nameParts)
static Spark3Util.CatalogAndIdentifier	catalogAndIdentifier(org.apache.spark.sql.SparkSession spark, java.util.List<java.lang.String> nameParts, org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog) A modified version of Spark's LookupCatalog.CatalogAndIdentifier.unapply Attempts to find the catalog and identifier a multipart identifier represents
static Spark3Util.CatalogAndIdentifier	catalogAndIdentifier(org.apache.spark.sql.SparkSession spark, java.lang.String name)
static Spark3Util.CatalogAndIdentifier	catalogAndIdentifier(org.apache.spark.sql.SparkSession spark, java.lang.String name, org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog)
static Spark3Util.CatalogAndIdentifier	catalogAndIdentifier(java.lang.String description, org.apache.spark.sql.SparkSession spark, java.lang.String name)
static Spark3Util.CatalogAndIdentifier	catalogAndIdentifier(java.lang.String description, org.apache.spark.sql.SparkSession spark, java.lang.String name, org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog)
static org.apache.iceberg.spark.OrderField[]	convert(SortOrder sortOrder)
static java.lang.String	describe(Expression expr)
static java.lang.String	describe(Schema schema)
static java.lang.String	describe(Type type)
static TableIdentifier	identifierToTableIdentifier(org.apache.spark.sql.connector.catalog.Identifier identifier)
static boolean	isLocalityEnabled(FileIO io, java.lang.String location, org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)
static boolean	isVectorizationEnabled(java.util.Map<java.lang.String,java.lang.String> properties, org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)
static java.lang.Integer	propertyAsInt(org.apache.spark.sql.util.CaseInsensitiveStringMap options, java.lang.String property, java.lang.Integer defaultValue)
static java.lang.Long	propertyAsLong(org.apache.spark.sql.util.CaseInsensitiveStringMap options, java.lang.String property, java.lang.Long defaultValue)
static java.util.Map<java.lang.String,java.lang.String>	rebuildCreateProperties(java.util.Map<java.lang.String,java.lang.String> createProperties)
static Distribution	toClusteredDistribution(PartitionSpec spec)
static Term	toIcebergTerm(org.apache.spark.sql.connector.expressions.Transform transform)
static Distribution	toOrderedDistribution(PartitionSpec spec, SortOrder sortOrder, boolean inferFromSpec)
static PartitionSpec	toPartitionSpec(Schema schema, org.apache.spark.sql.connector.expressions.Transform[] partitioning) Converts Spark transforms into a PartitionSpec.
static org.apache.spark.sql.connector.expressions.Transform[]	toTransforms(PartitionSpec spec) Converts a PartitionSpec to Spark transforms.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

rebuildCreateProperties

public static java.util.Map<java.lang.String,java.lang.String> rebuildCreateProperties(java.util.Map<java.lang.String,java.lang.String> createProperties)

applyPropertyChanges

public static UpdateProperties applyPropertyChanges(UpdateProperties pendingUpdate,
                                                    java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes)

Applies a list of Spark table changes to an UpdateProperties operation.

Parameters:

pendingUpdate - an uncommitted UpdateProperties operation to configure

changes - a list of Spark table changes

Returns:

the UpdateProperties operation configured with the changes

applySchemaChanges

public static UpdateSchema applySchemaChanges(UpdateSchema pendingUpdate,
                                              java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes)

Applies a list of Spark table changes to an UpdateSchema operation.

Parameters:

pendingUpdate - an uncommitted UpdateSchema operation to configure

changes - a list of Spark table changes

Returns:

the UpdateSchema operation configured with the changes

toTransforms

public static org.apache.spark.sql.connector.expressions.Transform[] toTransforms(PartitionSpec spec)

Converts a PartitionSpec to Spark transforms.

Parameters:

spec - a PartitionSpec

Returns:

an array of Transforms

toOrderedDistribution

public static Distribution toOrderedDistribution(PartitionSpec spec,
                                                 SortOrder sortOrder,
                                                 boolean inferFromSpec)

toClusteredDistribution

public static Distribution toClusteredDistribution(PartitionSpec spec)

convert

public static org.apache.iceberg.spark.OrderField[] convert(SortOrder sortOrder)

toIcebergTerm

public static Term toIcebergTerm(org.apache.spark.sql.connector.expressions.Transform transform)

toPartitionSpec

public static PartitionSpec toPartitionSpec(Schema schema,
                                            org.apache.spark.sql.connector.expressions.Transform[] partitioning)

Converts Spark transforms into a PartitionSpec.

Parameters:

schema - the table schema

partitioning - Spark Transforms

Returns:

a PartitionSpec

describe

public static java.lang.String describe(Expression expr)

describe

public static java.lang.String describe(Schema schema)

describe

public static java.lang.String describe(Type type)

isLocalityEnabled

public static boolean isLocalityEnabled(FileIO io,
                                        java.lang.String location,
                                        org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)

isVectorizationEnabled

public static boolean isVectorizationEnabled(java.util.Map<java.lang.String,java.lang.String> properties,
                                             org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)

batchSize

public static int batchSize(java.util.Map<java.lang.String,java.lang.String> properties,
                            org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)

propertyAsLong

public static java.lang.Long propertyAsLong(org.apache.spark.sql.util.CaseInsensitiveStringMap options,
                                            java.lang.String property,
                                            java.lang.Long defaultValue)

propertyAsInt

public static java.lang.Integer propertyAsInt(org.apache.spark.sql.util.CaseInsensitiveStringMap options,
                                              java.lang.String property,
                                              java.lang.Integer defaultValue)

catalogAndIdentifier

public static Spark3Util.CatalogAndIdentifier catalogAndIdentifier(org.apache.spark.sql.SparkSession spark,
                                                                   java.lang.String name)
                                                            throws org.apache.spark.sql.catalyst.parser.ParseException

Throws:

org.apache.spark.sql.catalyst.parser.ParseException

catalogAndIdentifier

public static Spark3Util.CatalogAndIdentifier catalogAndIdentifier(org.apache.spark.sql.SparkSession spark,
                                                                   java.lang.String name,
                                                                   org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog)
                                                            throws org.apache.spark.sql.catalyst.parser.ParseException

Throws:

org.apache.spark.sql.catalyst.parser.ParseException

catalogAndIdentifier

public static Spark3Util.CatalogAndIdentifier catalogAndIdentifier(java.lang.String description,
                                                                   org.apache.spark.sql.SparkSession spark,
                                                                   java.lang.String name)

catalogAndIdentifier

public static Spark3Util.CatalogAndIdentifier catalogAndIdentifier(java.lang.String description,
                                                                   org.apache.spark.sql.SparkSession spark,
                                                                   java.lang.String name,
                                                                   org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog)

catalogAndIdentifier

public static Spark3Util.CatalogAndIdentifier catalogAndIdentifier(org.apache.spark.sql.SparkSession spark,
                                                                   java.util.List<java.lang.String> nameParts)

catalogAndIdentifier

public static Spark3Util.CatalogAndIdentifier catalogAndIdentifier(org.apache.spark.sql.SparkSession spark,
                                                                   java.util.List<java.lang.String> nameParts,
                                                                   org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog)

A modified version of Spark's LookupCatalog.CatalogAndIdentifier.unapply Attempts to find the catalog and identifier a multipart identifier represents

Parameters:

spark - Spark session to use for resolution

nameParts - Multipart identifier representing a table

defaultCatalog - Catalog to use if none is specified

Returns:

The CatalogPlugin and Identifier for the table

identifierToTableIdentifier

public static TableIdentifier identifierToTableIdentifier(org.apache.spark.sql.connector.catalog.Identifier identifier)