Package org.apache.iceberg.types

Class TypeUtil

  • public class TypeUtil
extends java.lang.Object

    • Method Detail

      • project

        public static Schema project​(Schema schema,
                             java.util.Set<java.lang.Integer> fieldIds)
        Project extracts particular fields from a schema by ID.

        Unlike select(Schema, Set), project will pick out only the fields enumerated. Structs that are explicitly projected are empty unless sub-fields are explicitly projected. Maps and lists cannot be explicitly selected in fieldIds.

        Parameters:
        schema - to project fields from
        fieldIds - list of explicit fields to extract
        Returns:
        the schema with all fields fields not selected removed

      • select

        public static Schema select​(Schema schema,
                            java.util.Set<java.lang.Integer> fieldIds)

      • getProjectedIds

        public static java.util.Set<java.lang.Integer> getProjectedIds​(Schema schema)

      • getProjectedIds

        public static java.util.Set<java.lang.Integer> getProjectedIds​(Type type)

      • selectNot

        public static Schema selectNot​(Schema schema,
                               java.util.Set<java.lang.Integer> fieldIds)

      • indexByName

        public static java.util.Map<java.lang.String,​java.lang.Integer> indexByName​(Types.StructType struct)

      • indexNameById

        public static java.util.Map<java.lang.Integer,​java.lang.String> indexNameById​(Types.StructType struct)

      • indexQuotedNameById

        public static java.util.Map<java.lang.Integer,​java.lang.String> indexQuotedNameById​(Types.StructType struct,
                                                                                          java.util.function.Function<java.lang.String,​java.lang.String> quotingFunc)

      • indexByLowerCaseName

        public static java.util.Map<java.lang.String,​java.lang.Integer> indexByLowerCaseName​(Types.StructType struct)

      • indexParents

        public static java.util.Map<java.lang.Integer,​java.lang.Integer> indexParents​(Types.StructType struct)

      • assignFreshIds

        public static Type assignFreshIds​(Type type,
                                  TypeUtil.NextID nextId)
        Assigns fresh ids from the nextId function for all fields in a type.
        Parameters:
        type - a type
        nextId - an id assignment function
        Returns:
        an structurally identical type with new ids assigned by the nextId function

      • assignFreshIds

        public static Schema assignFreshIds​(Schema schema,
                                    TypeUtil.NextID nextId)
        Assigns fresh ids from the nextId function for all fields in a schema.
        Parameters:
        schema - a schema
        nextId - an id assignment function
        Returns:
        a structurally identical schema with new ids assigned by the nextId function

      • assignFreshIds

        public static Schema assignFreshIds​(int schemaId,
                                    Schema schema,
                                    TypeUtil.NextID nextId)
        Assigns fresh ids from the nextId function for all fields in a schema.
        Parameters:
        schemaId - an ID assigned to this schema
        schema - a schema
        nextId - an id assignment function
        Returns:
        a structurally identical schema with new ids assigned by the nextId function

      • assignFreshIds

        public static Schema assignFreshIds​(Schema schema,
                                    Schema baseSchema,
                                    TypeUtil.NextID nextId)
        Assigns ids to match a given schema, and fresh ids from the nextId function for all other fields.
        Parameters:
        schema - a schema
        baseSchema - a schema with existing IDs to copy by name
        nextId - an id assignment function
        Returns:
        a structurally identical schema with new ids assigned by the nextId function

      • refreshIdentifierFields

        public static java.util.Set<java.lang.Integer> refreshIdentifierFields​(Types.StructType freshSchema,
                                                                       Schema baseSchema)
        Get the identifier fields in the fresh schema based on the identifier fields in the base schema.
        Parameters:
        freshSchema - fresh schema
        baseSchema - base schema
        Returns:
        identifier fields in the fresh schema

      • assignIncreasingFreshIds

        public static Schema assignIncreasingFreshIds​(Schema schema)
        Assigns strictly increasing fresh ids for all fields in a schema, starting from 1.
        Parameters:
        schema - a schema
        Returns:
        a structurally identical schema with new ids assigned strictly increasing from 1

      • reassignIds

        public static Schema reassignIds​(Schema schema,
                                 Schema idSourceSchema)
        Reassigns ids in a schema from another schema.

        Ids are determined by field names. If a field in the schema cannot be found in the source schema, this will throw IllegalArgumentException.

        This will not alter a schema's structure, nullability, or types.

        Parameters:
        schema - the schema to have ids reassigned
        idSourceSchema - the schema from which field ids will be used
        Returns:
        an structurally identical schema with field ids matching the source schema
        Throws:
        java.lang.IllegalArgumentException - if a field cannot be found (by name) in the source schema

      • reassignDoc

        public static Schema reassignDoc​(Schema schema,
                                 Schema docSourceSchema)
        Reassigns doc in a schema from another schema.

        Doc are determined by field id. If a field in the schema cannot be found in the source schema, this will throw IllegalArgumentException.

        This will not alter a schema's structure, nullability, or types.

        Parameters:
        schema - the schema to have doc reassigned
        docSourceSchema - the schema from which field doc will be used
        Returns:
        an structurally identical schema with field ids matching the source schema
        Throws:
        java.lang.IllegalArgumentException - if a field cannot be found (by id) in the source schema

      • reassignIds

        public static Schema reassignIds​(Schema schema,
                                 Schema idSourceSchema,
                                 boolean caseSensitive)
        Reassigns ids in a schema from another schema.

        Ids are determined by field names. If a field in the schema cannot be found in the source schema, this will throw IllegalArgumentException.

        This will not alter a schema's structure, nullability, or types.

        Parameters:
        schema - the schema to have ids reassigned
        idSourceSchema - the schema from which field ids will be used
        Returns:
        an structurally identical schema with field ids matching the source schema
        Throws:
        java.lang.IllegalArgumentException - if a field cannot be found (by name) in the source schema

      • reassignOrRefreshIds

        public static Schema reassignOrRefreshIds​(Schema schema,
                                          Schema idSourceSchema)

      • reassignOrRefreshIds

        public static Schema reassignOrRefreshIds​(Schema schema,
                                          Schema idSourceSchema,
                                          boolean caseSensitive)

      • find

        public static Type find​(Schema schema,
                        java.util.function.Predicate<Type> predicate)

      • validateWriteSchema

        public static void validateWriteSchema​(Schema tableSchema,
                                       Schema writeSchema,
                                       java.lang.Boolean checkNullability,
                                       java.lang.Boolean checkOrdering)
        Check whether we could write the iceberg table with the user-provided write schema.
        Parameters:
        tableSchema - the table schema written in iceberg meta data.
        writeSchema - the user-provided write schema.
        checkNullability - If true, not allow to write optional values to a required field.
        checkOrdering - If true, not allow input schema to have different ordering than table schema.

      • validateSchema

        public static void validateSchema​(java.lang.String context,
                                  Schema expectedSchema,
                                  Schema providedSchema,
                                  boolean checkNullability,
                                  boolean checkOrdering)
        Validates whether the provided schema is compatible with the expected schema.
        Parameters:
        context - the schema context (e.g. row ID)
        expectedSchema - the expected schema
        providedSchema - the provided schema
        checkNullability - whether to check field nullability
        checkOrdering - whether to check field ordering

      • visit

        public static <T> T visit​(Type type,
                          TypeUtil.CustomOrderSchemaVisitor<T> visitor)
        Used to traverse types with traversals other than post-order.

        This passes a Supplier to each visitor method that returns the result of traversing child types. Structs are passed an Iterable that traverses child fields during iteration.

        An example use is assigning column IDs, which should be done with a pre-order traversal.

        Type Parameters:
        T - the type returned by the visitor
        Parameters:
        type - a type to traverse with a visitor
        visitor - a custom order visitor
        Returns:
        the result of traversing the given type with the visitor

      • decimalRequiredBytes

        public static int decimalRequiredBytes​(int precision)