Package org.apache.iceberg.spark

Class Spark3Util

  • public class Spark3Util
extends java.lang.Object

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static UpdateProperties applyPropertyChanges​(UpdateProperties pendingUpdate, java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes)
      Applies a list of Spark table changes to an UpdateProperties operation.
      static UpdateSchema applySchemaChanges​(UpdateSchema pendingUpdate, java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes)
      Applies a list of Spark table changes to an UpdateSchema operation.
      static int batchSize​(java.util.Map<java.lang.String,​java.lang.String> properties, org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)  
      static java.lang.String describe​(Expression expr)  
      static java.lang.String describe​(Schema schema)  
      static java.lang.String describe​(Type type)  
      static boolean isLocalityEnabled​(FileIO io, java.lang.String location, org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)  
      static boolean isVectorizationEnabled​(java.util.Map<java.lang.String,​java.lang.String> properties, org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)  
      static java.lang.Integer propertyAsInt​(org.apache.spark.sql.util.CaseInsensitiveStringMap options, java.lang.String property, java.lang.Integer defaultValue)  
      static java.lang.Long propertyAsLong​(org.apache.spark.sql.util.CaseInsensitiveStringMap options, java.lang.String property, java.lang.Long defaultValue)  
      static java.util.Map<java.lang.String,​java.lang.String> rebuildCreateProperties​(java.util.Map<java.lang.String,​java.lang.String> createProperties)  
      static PartitionSpec toPartitionSpec​(Schema schema, org.apache.spark.sql.connector.expressions.Transform[] partitioning)
      Converts Spark transforms into a PartitionSpec.
      static org.apache.spark.sql.connector.expressions.Transform[] toTransforms​(PartitionSpec spec)
      Converts a PartitionSpec to Spark transforms.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • rebuildCreateProperties

        public static java.util.Map<java.lang.String,​java.lang.String> rebuildCreateProperties​(java.util.Map<java.lang.String,​java.lang.String> createProperties)

      • applyPropertyChanges

        public static UpdateProperties applyPropertyChanges​(UpdateProperties pendingUpdate,
                                                    java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes)
        Applies a list of Spark table changes to an UpdateProperties operation.
        Parameters:
        pendingUpdate - an uncommitted UpdateProperties operation to configure
        changes - a list of Spark table changes
        Returns:
        the UpdateProperties operation configured with the changes

      • applySchemaChanges

        public static UpdateSchema applySchemaChanges​(UpdateSchema pendingUpdate,
                                              java.util.List<org.apache.spark.sql.connector.catalog.TableChange> changes)
        Applies a list of Spark table changes to an UpdateSchema operation.
        Parameters:
        pendingUpdate - an uncommitted UpdateSchema operation to configure
        changes - a list of Spark table changes
        Returns:
        the UpdateSchema operation configured with the changes

      • toTransforms

        public static org.apache.spark.sql.connector.expressions.Transform[] toTransforms​(PartitionSpec spec)
        Converts a PartitionSpec to Spark transforms.
        Parameters:
        spec - a PartitionSpec
        Returns:
        an array of Transforms

      • toPartitionSpec

        public static PartitionSpec toPartitionSpec​(Schema schema,
                                            org.apache.spark.sql.connector.expressions.Transform[] partitioning)
        Converts Spark transforms into a PartitionSpec.
        Parameters:
        schema - the table schema
        partitioning - Spark Transforms
        Returns:
        a PartitionSpec

      • describe

        public static java.lang.String describe​(Expression expr)

      • describe

        public static java.lang.String describe​(Schema schema)

      • describe

        public static java.lang.String describe​(Type type)

      • isLocalityEnabled

        public static boolean isLocalityEnabled​(FileIO io,
                                        java.lang.String location,
                                        org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)

      • isVectorizationEnabled

        public static boolean isVectorizationEnabled​(java.util.Map<java.lang.String,​java.lang.String> properties,
                                             org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)

      • batchSize

        public static int batchSize​(java.util.Map<java.lang.String,​java.lang.String> properties,
                            org.apache.spark.sql.util.CaseInsensitiveStringMap readOptions)

      • propertyAsLong

        public static java.lang.Long propertyAsLong​(org.apache.spark.sql.util.CaseInsensitiveStringMap options,
                                            java.lang.String property,
                                            java.lang.Long defaultValue)

      • propertyAsInt

        public static java.lang.Integer propertyAsInt​(org.apache.spark.sql.util.CaseInsensitiveStringMap options,
                                              java.lang.String property,
                                              java.lang.Integer defaultValue)