Package org.apache.iceberg.flink.sink.dynamic

Class EvolveSchemaVisitor

java.lang.Object

org.apache.iceberg.schema.SchemaWithPartnerVisitor<Integer,Boolean>

org.apache.iceberg.flink.sink.dynamic.EvolveSchemaVisitor

public class EvolveSchemaVisitor
extends SchemaWithPartnerVisitor<Integer,Boolean>

Visitor class that accumulates the set of changes needed to evolve an existing schema into the target schema. Changes are applied to an UpdateSchema operation.

We support:

• Adding new columns
• Widening the type of existing columsn
• Reordering columns

We don't support:

• Dropping columns
• Renaming columns

The reason is that dropping columns would create issues with late / out of order data. Once we drop fields, we wouldn't be able to easily add them back later without losing the associated data. Renaming columns is not supported because we compare schemas by name, which doesn't allow for renaming without additional hints.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.iceberg.schema.SchemaWithPartnerVisitor

SchemaWithPartnerVisitor.PartnerAccessors<P>

Method Summary

Modifier and Type	Method	Description
Boolean	field(Types.NestedField field, Integer partnerId, Boolean isFieldMissing)
Boolean	list(Types.ListType list, Integer partnerId, Boolean isElementMissing)
Boolean	map(Types.MapType map, Integer partnerId, Boolean isKeyMissing, Boolean isValueMissing)
Boolean	primitive(Type.PrimitiveType primitive, Integer partnerId)
Boolean	struct(Types.StructType struct, Integer partnerId, List<Boolean> existingFields)
static void	visit(UpdateSchema api, Schema existingSchema, Schema targetSchema)	Adds changes needed to produce the target schema to an UpdateSchema operation.

Methods inherited from class org.apache.iceberg.schema.SchemaWithPartnerVisitor

afterField, afterListElement, afterMapKey, afterMapValue, beforeField, beforeListElement, beforeMapKey, beforeMapValue, schema, variant, visit, visit

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

visit

public static void visit(UpdateSchema api,
 Schema existingSchema,
 Schema targetSchema)

Adds changes needed to produce the target schema to an UpdateSchema operation.

Changes are accumulated to evolve the existingSchema into a targetSchema.

Parameters:

api - an UpdateSchema for adding changes

existingSchema - an existing schema

targetSchema - a new schema to compare with the existing

struct

public Boolean struct(Types.StructType struct,
 Integer partnerId,
 List<Boolean> existingFields)

Overrides:

struct in class SchemaWithPartnerVisitor<Integer,Boolean>

field

public Boolean field(Types.NestedField field,
 Integer partnerId,
 Boolean isFieldMissing)

Overrides:

field in class SchemaWithPartnerVisitor<Integer,Boolean>

list

public Boolean list(Types.ListType list,
 Integer partnerId,
 Boolean isElementMissing)

Overrides:

list in class SchemaWithPartnerVisitor<Integer,Boolean>

map

public Boolean map(Types.MapType map,
 Integer partnerId,
 Boolean isKeyMissing,
 Boolean isValueMissing)

Overrides:

map in class SchemaWithPartnerVisitor<Integer,Boolean>

primitive

public Boolean primitive(Type.PrimitiveType primitive,
 Integer partnerId)

Overrides:

primitive in class SchemaWithPartnerVisitor<Integer,Boolean>