Package org.apache.iceberg.gcp.gcs

Class GCSFileIO

java.lang.Object
org.apache.iceberg.gcp.gcs.GCSFileIO
All Implemented Interfaces:
Closeable, Serializable, AutoCloseable, DelegateFileIO, FileIO, SupportsBulkOperations, SupportsPrefixOperations, SupportsStorageCredentials
public class GCSFileIO extends Object implements DelegateFileIO, SupportsStorageCredentials
FileIO Implementation backed by Google Cloud Storage (GCS)

Locations follow the conventions used by BlobId.fromGsUtilUri that follow the convention 

gs://<bucket>/<blob_path>

See Cloud Storage Overview

See Also:

  • Constructor Details

    • GCSFileIO

      public GCSFileIO()
      No-arg constructor to load the FileIO dynamically.

      All fields are initialized by calling initialize(Map) later.

    • GCSFileIO

      public GCSFileIO(SerializableSupplier<com.google.cloud.storage.Storage> storageSupplier)
      Constructor with custom storage supplier.
      Parameters:
      storageSupplier - storage supplier

    • GCSFileIO

      @Deprecated public GCSFileIO(SerializableSupplier<com.google.cloud.storage.Storage> storageSupplier, GCPProperties gcpProperties)
      Deprecated.
      since 1.10.0, will be removed in 1.11.0; use GCSFileIO(SerializableSupplier) with initialize(Map) instead
      Constructor with custom storage supplier and GCP properties.

      Calling initialize(Map) will overwrite information set in this constructor.

      Parameters:
      storageSupplier - storage supplier
      gcpProperties - gcp properties

  • Method Details

    • newInputFile

      public InputFile newInputFile(String path)
      Description copied from interface: FileIO
      Get a InputFile instance to read bytes from the file at the given path.
      Specified by:
      newInputFile in interface FileIO

    • newInputFile

      public InputFile newInputFile(String path, long length)
      Description copied from interface: FileIO
      Get a InputFile instance to read bytes from the file at the given path, with a known file length.
      Specified by:
      newInputFile in interface FileIO

    • newOutputFile

      public OutputFile newOutputFile(String path)
      Description copied from interface: FileIO
      Get a OutputFile instance to write bytes to the file at the given path.
      Specified by:
      newOutputFile in interface FileIO

    • deleteFile

      public void deleteFile(String path)
      Description copied from interface: FileIO
      Delete the file at the given path.
      Specified by:
      deleteFile in interface FileIO

    • properties

      public Map<String,String> properties()
      Description copied from interface: FileIO
      Returns the property map used to configure this FileIO
      Specified by:
      properties in interface FileIO

    • client

      public com.google.cloud.storage.Storage client()

    • client

      public com.google.cloud.storage.Storage client(String storagePath)

    • initialize

      public void initialize(Map<String,String> props)
      Description copied from interface: FileIO
      Initialize File IO from catalog properties.
      Specified by:
      initialize in interface FileIO
      Parameters:
      props - catalog properties

    • close

      public void close()
      Description copied from interface: FileIO
      Close File IO to release underlying resources.

      Calling this method is only required when this FileIO instance is no longer expected to be used, and the resources it holds need to be explicitly released to avoid resource leaks.

      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Specified by:
      close in interface FileIO

    • listPrefix

      public Iterable<FileInfo> listPrefix(String prefix)
      Description copied from interface: SupportsPrefixOperations
      Return an iterable of all files under a prefix.

      Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.

      Specified by:
      listPrefix in interface SupportsPrefixOperations
      Parameters:
      prefix - prefix to list
      Returns:
      iterable of file information

    • deletePrefix

      public void deletePrefix(String prefix)
      Description copied from interface: SupportsPrefixOperations
      Delete all files under a prefix.

      Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.

      Specified by:
      deletePrefix in interface SupportsPrefixOperations
      Parameters:
      prefix - prefix to delete

    • deleteFiles

      public void deleteFiles(Iterable<String> pathsToDelete) throws BulkDeletionFailureException
      Description copied from interface: SupportsBulkOperations
      Delete the files at the given paths.
      Specified by:
      deleteFiles in interface SupportsBulkOperations
      Parameters:
      pathsToDelete - The paths to delete
      Throws:
      BulkDeletionFailureException - in case of failure to delete at least 1 file

    • setCredentials

      public void setCredentials(List<StorageCredential> credentials)
      Specified by:
      setCredentials in interface SupportsStorageCredentials

    • credentials

      public List<StorageCredential> credentials()
      Specified by:
      credentials in interface SupportsStorageCredentials