Class SparkDataFile

    • Constructor Detail

      • SparkDataFile

        public SparkDataFile​(Types.StructType type,
                             org.apache.spark.sql.types.StructType sparkType)
    • Method Detail

      • equalityFieldIds

        public java.util.List<java.lang.Integer> equalityFieldIds()
        Description copied from interface: ContentFile
        Returns the set of field IDs used for equality comparison, in equality delete files.

        An equality delete file may contain additional data fields that are not used by equality comparison. The subset of columns in a delete file to be used in equality comparison are tracked by ID. Extra columns can be used to reconstruct changes and metrics from extra columns are used during job planning.

        Specified by:
        equalityFieldIds in interface ContentFile<DataFile>
        Specified by:
        equalityFieldIds in interface DataFile
        equalityFieldIds in class SparkContentFile<DataFile>
        IDs of the fields used in equality comparison with the records in this delete file