Class S3FileIO

    • Constructor Summary

      Constructors 
      Constructor Description
      S3FileIO()
      No-arg constructor to load the FileIO dynamically.
      S3FileIO​(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)
      Constructor with custom s3 supplier and default AWS properties.
      S3FileIO​(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3, AwsProperties awsProperties)
      Constructor with custom s3 supplier and AWS properties.
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()
      Close File IO to release underlying resources.
      void deleteFile​(java.lang.String path)
      Delete the file at the given path.
      void deleteFiles​(java.lang.Iterable<java.lang.String> paths)
      Deletes the given paths in a batched manner.
      void deletePrefix​(java.lang.String prefix)
      This method provides a "best-effort" to delete all objects under the given prefix.
      java.lang.String getCredential()
      Returns the credential string
      void initialize​(java.util.Map<java.lang.String,​java.lang.String> props)
      Initialize File IO from catalog properties.
      java.lang.Iterable<FileInfo> listPrefix​(java.lang.String prefix)
      Return an iterable of all files under a prefix.
      InputFile newInputFile​(java.lang.String path)
      Get a InputFile instance to read bytes from the file at the given path.
      InputFile newInputFile​(java.lang.String path, long length)
      Get a InputFile instance to read bytes from the file at the given path, with a known file length.
      OutputFile newOutputFile​(java.lang.String path)
      Get a OutputFile instance to write bytes to the file at the given path.
      java.util.Map<java.lang.String,​java.lang.String> properties()
      Returns the property map used to configure this FileIO
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Constructor Detail

      • S3FileIO

        public S3FileIO()
        No-arg constructor to load the FileIO dynamically.

        All fields are initialized by calling initialize(Map) later.

      • S3FileIO

        public S3FileIO​(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)
        Constructor with custom s3 supplier and default AWS properties.

        Calling initialize(Map) will overwrite information set in this constructor.

        Parameters:
        s3 - s3 supplier
      • S3FileIO

        public S3FileIO​(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3,
                        AwsProperties awsProperties)
        Constructor with custom s3 supplier and AWS properties.

        Calling initialize(Map) will overwrite information set in this constructor.

        Parameters:
        s3 - s3 supplier
        awsProperties - aws properties
    • Method Detail

      • newInputFile

        public InputFile newInputFile​(java.lang.String path)
        Description copied from interface: FileIO
        Get a InputFile instance to read bytes from the file at the given path.
        Specified by:
        newInputFile in interface FileIO
      • newInputFile

        public InputFile newInputFile​(java.lang.String path,
                                      long length)
        Description copied from interface: FileIO
        Get a InputFile instance to read bytes from the file at the given path, with a known file length.
        Specified by:
        newInputFile in interface FileIO
      • newOutputFile

        public OutputFile newOutputFile​(java.lang.String path)
        Description copied from interface: FileIO
        Get a OutputFile instance to write bytes to the file at the given path.
        Specified by:
        newOutputFile in interface FileIO
      • deleteFile

        public void deleteFile​(java.lang.String path)
        Description copied from interface: FileIO
        Delete the file at the given path.
        Specified by:
        deleteFile in interface FileIO
      • properties

        public java.util.Map<java.lang.String,​java.lang.String> properties()
        Description copied from interface: FileIO
        Returns the property map used to configure this FileIO
        Specified by:
        properties in interface FileIO
      • deleteFiles

        public void deleteFiles​(java.lang.Iterable<java.lang.String> paths)
                         throws BulkDeletionFailureException
        Deletes the given paths in a batched manner.

        The paths are grouped by bucket, and deletion is triggered when we either reach the configured batch size or have a final remainder batch for each bucket.

        Specified by:
        deleteFiles in interface SupportsBulkOperations
        Parameters:
        paths - paths to delete
        Throws:
        BulkDeletionFailureException - in case of failure to delete at least 1 file
      • listPrefix

        public java.lang.Iterable<FileInfo> listPrefix​(java.lang.String prefix)
        Description copied from interface: SupportsPrefixOperations
        Return an iterable of all files under a prefix.

        Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.

        Specified by:
        listPrefix in interface SupportsPrefixOperations
        Parameters:
        prefix - prefix to list
        Returns:
        iterable of file information
      • deletePrefix

        public void deletePrefix​(java.lang.String prefix)
        This method provides a "best-effort" to delete all objects under the given prefix.

        Bulk delete operations are used and no reattempt is made for deletes if they fail, but will log any individual objects that are not deleted as part of the bulk operation.

        Specified by:
        deletePrefix in interface SupportsPrefixOperations
        Parameters:
        prefix - prefix to delete
      • initialize

        public void initialize​(java.util.Map<java.lang.String,​java.lang.String> props)
        Description copied from interface: FileIO
        Initialize File IO from catalog properties.
        Specified by:
        initialize in interface FileIO
        Parameters:
        props - catalog properties
      • close

        public void close()
        Description copied from interface: FileIO
        Close File IO to release underlying resources.

        Calling this method is only required when this FileIO instance is no longer expected to be used, and the resources it holds need to be explicitly released to avoid resource leaks.

        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Specified by:
        close in interface FileIO