org.apache.iceberg.aws.s3

Class S3FileIO

java.lang.Object

org.apache.iceberg.aws.s3.S3FileIO

All Implemented Interfaces:

java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable, CredentialSupplier, DelegateFileIO, FileIO, SupportsBulkOperations, SupportsPrefixOperations

public class S3FileIO
extends java.lang.Object
implements CredentialSupplier, DelegateFileIO

FileIO implementation backed by S3.

Locations used must follow the conventions for S3 URIs (e.g. s3://bucket/path...). URIs with schemes s3a, s3n, https are also treated as s3 file paths. Using this FileIO with other schemes will result in ValidationException.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
S3FileIO() No-arg constructor to load the FileIO dynamically.
S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3) Constructor with custom s3 supplier and S3FileIO properties.
S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3, S3FileIOProperties s3FileIOProperties) Constructor with custom s3 supplier and S3FileIO properties.

Method Summary

Modifier and Type	Method and Description
software.amazon.awssdk.services.s3.S3Client	client()
void	close() Close File IO to release underlying resources.
void	deleteFile(java.lang.String path) Delete the file at the given path.
void	deleteFiles(java.lang.Iterable<java.lang.String> paths) Deletes the given paths in a batched manner.
void	deletePrefix(java.lang.String prefix) This method provides a "best-effort" to delete all objects under the given prefix.
protected void	finalize()
java.lang.String	getCredential() Returns the credential string
void	initialize(java.util.Map<java.lang.String,java.lang.String> props) Initialize File IO from catalog properties.
java.lang.Iterable<FileInfo>	listPrefix(java.lang.String prefix) Return an iterable of all files under a prefix.
InputFile	newInputFile(java.lang.String path) Get a InputFile instance to read bytes from the file at the given path.
InputFile	newInputFile(java.lang.String path, long length) Get a InputFile instance to read bytes from the file at the given path, with a known file length.
OutputFile	newOutputFile(java.lang.String path) Get a OutputFile instance to write bytes to the file at the given path.
java.util.Map<java.lang.String,java.lang.String>	properties() Returns the property map used to configure this FileIO

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.io.FileIO

deleteFile, deleteFile, newInputFile, newInputFile, newInputFile

Constructor Detail

S3FileIO

public S3FileIO()

No-arg constructor to load the FileIO dynamically.

All fields are initialized by calling initialize(Map) later.

S3FileIO

public S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)

Constructor with custom s3 supplier and S3FileIO properties.

Calling initialize(Map) will overwrite information set in this constructor.

Parameters:

s3 - s3 supplier

S3FileIO

public S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3,
                S3FileIOProperties s3FileIOProperties)

Constructor with custom s3 supplier and S3FileIO properties.

Calling initialize(Map) will overwrite information set in this constructor.

Parameters:

s3 - s3 supplier

s3FileIOProperties - S3 FileIO properties

Method Detail

newInputFile

public InputFile newInputFile(java.lang.String path)

Description copied from interface: FileIO

Get a InputFile instance to read bytes from the file at the given path.

Specified by:

newInputFile in interface FileIO

newInputFile

public InputFile newInputFile(java.lang.String path,
                              long length)

Description copied from interface: FileIO

Get a InputFile instance to read bytes from the file at the given path, with a known file length.

Specified by:

newInputFile in interface FileIO

newOutputFile

public OutputFile newOutputFile(java.lang.String path)

Description copied from interface: FileIO

Get a OutputFile instance to write bytes to the file at the given path.

Specified by:

newOutputFile in interface FileIO

deleteFile

public void deleteFile(java.lang.String path)

Description copied from interface: FileIO

Delete the file at the given path.

Specified by:

deleteFile in interface FileIO

properties

public java.util.Map<java.lang.String,java.lang.String> properties()

Description copied from interface: FileIO

Returns the property map used to configure this FileIO

Specified by:

properties in interface FileIO

deleteFiles

public void deleteFiles(java.lang.Iterable<java.lang.String> paths)
                 throws BulkDeletionFailureException

Deletes the given paths in a batched manner.

The paths are grouped by bucket, and deletion is triggered when we either reach the configured batch size or have a final remainder batch for each bucket.

Specified by:

deleteFiles in interface SupportsBulkOperations

Parameters:

paths - paths to delete

Throws:

BulkDeletionFailureException - in case of failure to delete at least 1 file

listPrefix

public java.lang.Iterable<FileInfo> listPrefix(java.lang.String prefix)

Description copied from interface: SupportsPrefixOperations

Return an iterable of all files under a prefix.

Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.

Specified by:

listPrefix in interface SupportsPrefixOperations

Parameters:

prefix - prefix to list

Returns:

iterable of file information

deletePrefix

public void deletePrefix(java.lang.String prefix)

This method provides a "best-effort" to delete all objects under the given prefix.

Bulk delete operations are used and no reattempt is made for deletes if they fail, but will log any individual objects that are not deleted as part of the bulk operation.

Specified by:

deletePrefix in interface SupportsPrefixOperations

Parameters:

prefix - prefix to delete

client

public software.amazon.awssdk.services.s3.S3Client client()

getCredential

public java.lang.String getCredential()

Description copied from interface: CredentialSupplier

Returns the credential string

Specified by:

getCredential in interface CredentialSupplier

initialize

public void initialize(java.util.Map<java.lang.String,java.lang.String> props)

Description copied from interface: FileIO

Initialize File IO from catalog properties.

Specified by:

initialize in interface FileIO

Parameters:

props - catalog properties

close

public void close()

Description copied from interface: FileIO

Close File IO to release underlying resources.

Calling this method is only required when this FileIO instance is no longer expected to be used, and the resources it holds need to be explicitly released to avoid resource leaks.

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface FileIO

finalize

protected void finalize()
                 throws java.lang.Throwable

Overrides:

finalize in class java.lang.Object

Throws:

java.lang.Throwable