Class SparkDataFile

    • Constructor Detail

      • SparkDataFile

        public SparkDataFile​(Types.StructType type,
                             org.apache.spark.sql.types.StructType sparkType)
    • Method Detail

      • wrap

        public SparkDataFile wrap​(org.apache.spark.sql.Row row)
      • pos

        public java.lang.Long pos()
        Description copied from interface: ContentFile
        Returns the ordinal position of the file in a manifest, or null if it was not read from a manifest.
        Specified by:
        pos in interface ContentFile<DataFile>
      • specId

        public int specId()
        Description copied from interface: ContentFile
        Returns id of the partition spec used for partition metadata.
        Specified by:
        specId in interface ContentFile<DataFile>
      • path

        public java.lang.CharSequence path()
        Description copied from interface: ContentFile
        Returns fully qualified path to the file, suitable for constructing a Hadoop Path.
        Specified by:
        path in interface ContentFile<DataFile>
      • recordCount

        public long recordCount()
        Description copied from interface: ContentFile
        Returns the number of top-level records in the file.
        Specified by:
        recordCount in interface ContentFile<DataFile>
      • columnSizes

        public java.util.Map<java.lang.Integer,​java.lang.Long> columnSizes()
        Description copied from interface: ContentFile
        Returns if collected, map from column ID to the size of the column in bytes, null otherwise.
        Specified by:
        columnSizes in interface ContentFile<DataFile>
      • valueCounts

        public java.util.Map<java.lang.Integer,​java.lang.Long> valueCounts()
        Description copied from interface: ContentFile
        Returns if collected, map from column ID to the count of its values (including null and NaN values), null otherwise.
        Specified by:
        valueCounts in interface ContentFile<DataFile>
      • nullValueCounts

        public java.util.Map<java.lang.Integer,​java.lang.Long> nullValueCounts()
        Description copied from interface: ContentFile
        Returns if collected, map from column ID to its null value count, null otherwise.
        Specified by:
        nullValueCounts in interface ContentFile<DataFile>
      • nanValueCounts

        public java.util.Map<java.lang.Integer,​java.lang.Long> nanValueCounts()
        Description copied from interface: ContentFile
        Returns if collected, map from column ID to its NaN value count, null otherwise.
        Specified by:
        nanValueCounts in interface ContentFile<DataFile>
      • lowerBounds

        public java.util.Map<java.lang.Integer,​java.nio.ByteBuffer> lowerBounds()
        Description copied from interface: ContentFile
        Returns if collected, map from column ID to value lower bounds, null otherwise.
        Specified by:
        lowerBounds in interface ContentFile<DataFile>
      • upperBounds

        public java.util.Map<java.lang.Integer,​java.nio.ByteBuffer> upperBounds()
        Description copied from interface: ContentFile
        Returns if collected, map from column ID to value upper bounds, null otherwise.
        Specified by:
        upperBounds in interface ContentFile<DataFile>
      • keyMetadata

        public java.nio.ByteBuffer keyMetadata()
        Description copied from interface: ContentFile
        Returns metadata about how this file is encrypted, or null if the file is stored in plain text.
        Specified by:
        keyMetadata in interface ContentFile<DataFile>
      • copy

        public DataFile copy()
        Description copied from interface: ContentFile
        Copies this file. Manifest readers can reuse file instances; use this method to copy data when collecting files from tasks.
        Specified by:
        copy in interface ContentFile<DataFile>
        Returns:
        a copy of this data file
      • copyWithoutStats

        public DataFile copyWithoutStats()
        Description copied from interface: ContentFile
        Copies this file without file stats. Manifest readers can reuse file instances; use this method to copy data without stats when collecting files.
        Specified by:
        copyWithoutStats in interface ContentFile<DataFile>
        Returns:
        a copy of this data file, without lower bounds, upper bounds, value counts, null value counts, or nan value counts
      • splitOffsets

        public java.util.List<java.lang.Long> splitOffsets()
        Description copied from interface: ContentFile
        Returns list of recommended split locations, if applicable, null otherwise.

        When available, this information is used for planning scan tasks whose boundaries are determined by these offsets. The returned list must be sorted in ascending order.

        Specified by:
        splitOffsets in interface ContentFile<DataFile>
      • sortOrderId

        public java.lang.Integer sortOrderId()
        Description copied from interface: ContentFile
        Returns the sort order id of this file, which describes how the file is ordered. This information will be useful for merging data and equality delete files more efficiently when they share the same sort order id.
        Specified by:
        sortOrderId in interface ContentFile<DataFile>