Package org.apache.iceberg.io

Class ResolvingFileIO

java.lang.Object
org.apache.iceberg.io.ResolvingFileIO
All Implemented Interfaces:
Closeable, Serializable, AutoCloseable, org.apache.hadoop.conf.Configurable, HadoopConfigurable, DelegateFileIO, FileIO, SupportsBulkOperations, SupportsPrefixOperations, SupportsStorageCredentials
public class ResolvingFileIO extends Object implements HadoopConfigurable, DelegateFileIO, SupportsStorageCredentials
FileIO implementation that uses location scheme to choose the correct FileIO implementation. Delegate FileIO implementations must implement the DelegateFileIO mixin interface, otherwise initialization will fail.
See Also:

  • Constructor Details

    • ResolvingFileIO

      public ResolvingFileIO()
      No-arg constructor to load the FileIO dynamically.

      All fields are initialized by calling initialize(Map) later.

  • Method Details

    • newInputFile

      public InputFile newInputFile(String location)
      Description copied from interface: FileIO
      Get a InputFile instance to read bytes from the file at the given path.
      Specified by:
      newInputFile in interface FileIO

    • newInputFile

      public InputFile newInputFile(String location, long length)
      Description copied from interface: FileIO
      Get a InputFile instance to read bytes from the file at the given path, with a known file length.
      Specified by:
      newInputFile in interface FileIO

    • newOutputFile

      public OutputFile newOutputFile(String location)
      Description copied from interface: FileIO
      Get a OutputFile instance to write bytes to the file at the given path.
      Specified by:
      newOutputFile in interface FileIO

    • deleteFile

      public void deleteFile(String location)
      Description copied from interface: FileIO
      Delete the file at the given path.
      Specified by:
      deleteFile in interface FileIO

    • deleteFiles

      public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailureException
      Description copied from interface: SupportsBulkOperations
      Delete the files at the given paths.
      Specified by:
      deleteFiles in interface SupportsBulkOperations
      Parameters:
      pathsToDelete - The paths to delete
      Throws:
      BulkDeletionFailureException - in case of failure to delete at least 1 file

    • properties

      public Map<String,String> properties()
      Description copied from interface: FileIO
      Returns the property map used to configure this FileIO
      Specified by:
      properties in interface FileIO

    • initialize

      public void initialize(Map<String,String> newProperties)
      Description copied from interface: FileIO
      Initialize File IO from catalog properties.
      Specified by:
      initialize in interface FileIO
      Parameters:
      newProperties - catalog properties

    • close

      public void close()
      Description copied from interface: FileIO
      Close File IO to release underlying resources.

      Calling this method is only required when this FileIO instance is no longer expected to be used, and the resources it holds need to be explicitly released to avoid resource leaks.

      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Specified by:
      close in interface FileIO

    • serializeConfWith

      public void serializeConfWith(Function<org.apache.hadoop.conf.Configuration,SerializableSupplier<org.apache.hadoop.conf.Configuration>> confSerializer)
      Description copied from interface: HadoopConfigurable
      Take a function that serializes Hadoop configuration into a supplier. An implementation is supposed to pass in its current Hadoop configuration into this function, and the result can be safely serialized for future use.
      Specified by:
      serializeConfWith in interface HadoopConfigurable
      Parameters:
      confSerializer - A function that takes Hadoop configuration and returns a serializable supplier of it.

    • setConf

      public void setConf(org.apache.hadoop.conf.Configuration conf)
      Specified by:
      setConf in interface org.apache.hadoop.conf.Configurable

    • getConf

      public org.apache.hadoop.conf.Configuration getConf()
      Specified by:
      getConf in interface org.apache.hadoop.conf.Configurable

    • ioClass

      public Class<?> ioClass(String location)

    • finalize

      protected void finalize() throws Throwable
      Overrides:
      finalize in class Object
      Throws:
      Throwable

    • listPrefix

      public Iterable<FileInfo> listPrefix(String prefix)
      Description copied from interface: SupportsPrefixOperations
      Return an iterable of all files under a prefix.

      Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.

      Specified by:
      listPrefix in interface SupportsPrefixOperations
      Parameters:
      prefix - prefix to list
      Returns:
      iterable of file information

    • deletePrefix

      public void deletePrefix(String prefix)
      Description copied from interface: SupportsPrefixOperations
      Delete all files under a prefix.

      Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.

      Specified by:
      deletePrefix in interface SupportsPrefixOperations
      Parameters:
      prefix - prefix to delete

    • setCredentials

      public void setCredentials(List<StorageCredential> credentials)
      Specified by:
      setCredentials in interface SupportsStorageCredentials

    • credentials

      public List<StorageCredential> credentials()
      Specified by:
      credentials in interface SupportsStorageCredentials