Package org.apache.iceberg.hadoop

Class HadoopFileIO

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.hadoop.conf.Configuration conf()  
      void deleteFile​(java.lang.String path)
      Delete the file at the given path.
      void deletePrefix​(java.lang.String prefix)
      Delete all files under a prefix.
      org.apache.hadoop.conf.Configuration getConf()  
      java.lang.Iterable<FileInfo> listPrefix​(java.lang.String prefix)
      Return an iterable of all files under a prefix.
      InputFile newInputFile​(java.lang.String path)
      Get a InputFile instance to read bytes from the file at the given path.
      InputFile newInputFile​(java.lang.String path, long length)
      Get a InputFile instance to read bytes from the file at the given path, with a known file length.
      OutputFile newOutputFile​(java.lang.String path)
      Get a OutputFile instance to write bytes to the file at the given path.
      java.util.Map<java.lang.String,​java.lang.String> properties()
      Returns the property map used to configure this FileIO
      void serializeConfWith​(java.util.function.Function<org.apache.hadoop.conf.Configuration,​SerializableSupplier<org.apache.hadoop.conf.Configuration>> confSerializer)
      Take a function that serializes Hadoop configuration into a supplier.
      void setConf​(org.apache.hadoop.conf.Configuration conf)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • HadoopFileIO

        public HadoopFileIO()
        Constructor used for dynamic FileIO loading.

        Hadoop configuration must be set through setConf(Configuration)

      • HadoopFileIO

        public HadoopFileIO​(org.apache.hadoop.conf.Configuration hadoopConf)

      • HadoopFileIO

        public HadoopFileIO​(SerializableSupplier<org.apache.hadoop.conf.Configuration> hadoopConf)

    • Method Detail

      • conf

        public org.apache.hadoop.conf.Configuration conf()

      • newInputFile

        public InputFile newInputFile​(java.lang.String path)
        Description copied from interface: FileIO
        Get a InputFile instance to read bytes from the file at the given path.
        Specified by:
        newInputFile in interface FileIO

      • newInputFile

        public InputFile newInputFile​(java.lang.String path,
                              long length)
        Description copied from interface: FileIO
        Get a InputFile instance to read bytes from the file at the given path, with a known file length.
        Specified by:
        newInputFile in interface FileIO

      • newOutputFile

        public OutputFile newOutputFile​(java.lang.String path)
        Description copied from interface: FileIO
        Get a OutputFile instance to write bytes to the file at the given path.
        Specified by:
        newOutputFile in interface FileIO

      • deleteFile

        public void deleteFile​(java.lang.String path)
        Description copied from interface: FileIO
        Delete the file at the given path.
        Specified by:
        deleteFile in interface FileIO

      • properties

        public java.util.Map<java.lang.String,​java.lang.String> properties()
        Description copied from interface: FileIO
        Returns the property map used to configure this FileIO
        Specified by:
        properties in interface FileIO

      • setConf

        public void setConf​(org.apache.hadoop.conf.Configuration conf)
        Specified by:
        setConf in interface org.apache.hadoop.conf.Configurable

      • getConf

        public org.apache.hadoop.conf.Configuration getConf()
        Specified by:
        getConf in interface org.apache.hadoop.conf.Configurable

      • serializeConfWith

        public void serializeConfWith​(java.util.function.Function<org.apache.hadoop.conf.Configuration,​SerializableSupplier<org.apache.hadoop.conf.Configuration>> confSerializer)
        Description copied from interface: HadoopConfigurable
        Take a function that serializes Hadoop configuration into a supplier. An implementation is supposed to pass in its current Hadoop configuration into this function, and the result can be safely serialized for future use.
        Specified by:
        serializeConfWith in interface HadoopConfigurable
        Parameters:
        confSerializer - A function that takes Hadoop configuration and returns a serializable supplier of it.

      • listPrefix

        public java.lang.Iterable<FileInfo> listPrefix​(java.lang.String prefix)
        Description copied from interface: SupportsPrefixOperations
        Return an iterable of all files under a prefix.

        Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.

        Specified by:
        listPrefix in interface SupportsPrefixOperations
        Parameters:
        prefix - prefix to list
        Returns:
        iterable of file information

      • deletePrefix

        public void deletePrefix​(java.lang.String prefix)
        Description copied from interface: SupportsPrefixOperations
        Delete all files under a prefix.

        Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.

        Specified by:
        deletePrefix in interface SupportsPrefixOperations
        Parameters:
        prefix - prefix to delete