org.apache.iceberg.spark

Class SparkDataFile

java.lang.Object

org.apache.iceberg.spark.SparkContentFile<DataFile>

org.apache.iceberg.spark.SparkDataFile

All Implemented Interfaces:

ContentFile<DataFile>, DataFile

public class SparkDataFile
extends SparkContentFile<DataFile>
implements DataFile

Field Summary

Fields inherited from interface org.apache.iceberg.DataFile

COLUMN_SIZES, CONTENT, EQUALITY_IDS, FILE_FORMAT, FILE_PATH, FILE_SIZE, KEY_METADATA, LOWER_BOUNDS, NAN_VALUE_COUNTS, NULL_VALUE_COUNTS, PARTITION_DOC, PARTITION_ID, PARTITION_NAME, RECORD_COUNT, SORT_ORDER_ID, SPEC_ID, SPLIT_OFFSETS, UPPER_BOUNDS, VALUE_COUNTS

Constructor Summary

Constructors

Constructor and Description
SparkDataFile(Types.StructType type, org.apache.spark.sql.types.StructType sparkType)
SparkDataFile(Types.StructType type, Types.StructType projectedType, org.apache.spark.sql.types.StructType sparkType)

Method Summary

Modifier and Type	Method and Description
protected DataFile	asFile()
java.util.List<java.lang.Integer>	equalityFieldIds() Returns the set of field IDs used for equality comparison, in equality delete files.

Methods inherited from class org.apache.iceberg.spark.SparkContentFile

columnSizes, content, copy, copyWithoutStats, fileSizeInBytes, format, keyMetadata, lowerBounds, nanValueCounts, nullValueCounts, partition, path, pos, recordCount, sortOrderId, specId, splitOffsets, upperBounds, valueCounts, wrap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.DataFile

content, getType

Methods inherited from interface org.apache.iceberg.ContentFile

columnSizes, copy, copy, copyWithoutStats, copyWithStats, dataSequenceNumber, fileSequenceNumber, fileSizeInBytes, format, keyMetadata, lowerBounds, nanValueCounts, nullValueCounts, partition, path, pos, recordCount, sortOrderId, specId, splitOffsets, upperBounds, valueCounts

Constructor Detail

SparkDataFile

public SparkDataFile(Types.StructType type,
                     org.apache.spark.sql.types.StructType sparkType)

SparkDataFile

public SparkDataFile(Types.StructType type,
                     Types.StructType projectedType,
                     org.apache.spark.sql.types.StructType sparkType)

Method Detail

asFile

protected DataFile asFile()

Specified by:

asFile in class SparkContentFile<DataFile>

equalityFieldIds

public java.util.List<java.lang.Integer> equalityFieldIds()

Description copied from interface: ContentFile

Returns the set of field IDs used for equality comparison, in equality delete files.

An equality delete file may contain additional data fields that are not used by equality comparison. The subset of columns in a delete file to be used in equality comparison are tracked by ID. Extra columns can be used to reconstruct changes and metrics from extra columns are used during job planning.

Specified by:

equalityFieldIds in interface ContentFile<DataFile>

Specified by:

equalityFieldIds in interface DataFile

Overrides:

equalityFieldIds in class SparkContentFile<DataFile>

Returns:

IDs of the fields used in equality comparison with the records in this delete file