Class S3FileIO
- java.lang.Object
-
- org.apache.iceberg.aws.s3.S3FileIO
-
- All Implemented Interfaces:
java.io.Closeable
,java.io.Serializable
,java.lang.AutoCloseable
,CredentialSupplier
,FileIO
,SupportsBulkOperations
,SupportsPrefixOperations
public class S3FileIO extends java.lang.Object implements FileIO, SupportsBulkOperations, SupportsPrefixOperations, CredentialSupplier
FileIO implementation backed by S3.Locations used must follow the conventions for S3 URIs (e.g. s3://bucket/path...). URIs with schemes s3a, s3n, https are also treated as s3 file paths. Using this FileIO with other schemes will result in
ValidationException
.- See Also:
- Serialized Form
-
-
Constructor Summary
Constructors Constructor Description S3FileIO()
No-arg constructor to load the FileIO dynamically.S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)
Constructor with custom s3 supplier and default AWS properties.S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3, AwsProperties awsProperties)
Constructor with custom s3 supplier and AWS properties.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
Close File IO to release underlying resources.void
deleteFile(java.lang.String path)
Delete the file at the given path.void
deleteFiles(java.lang.Iterable<java.lang.String> paths)
Deletes the given paths in a batched manner.void
deletePrefix(java.lang.String prefix)
This method provides a "best-effort" to delete all objects under the given prefix.java.lang.String
getCredential()
Returns the credential stringvoid
initialize(java.util.Map<java.lang.String,java.lang.String> props)
Initialize File IO from catalog properties.java.lang.Iterable<FileInfo>
listPrefix(java.lang.String prefix)
Return an iterable of all files under a prefix.InputFile
newInputFile(java.lang.String path)
Get aInputFile
instance to read bytes from the file at the given path.InputFile
newInputFile(java.lang.String path, long length)
Get aInputFile
instance to read bytes from the file at the given path, with a known file length.OutputFile
newOutputFile(java.lang.String path)
Get aOutputFile
instance to write bytes to the file at the given path.java.util.Map<java.lang.String,java.lang.String>
properties()
Returns the property map used to configure this FileIO-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.iceberg.io.FileIO
deleteFile, deleteFile
-
-
-
-
Constructor Detail
-
S3FileIO
public S3FileIO()
No-arg constructor to load the FileIO dynamically.All fields are initialized by calling
initialize(Map)
later.
-
S3FileIO
public S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)
Constructor with custom s3 supplier and default AWS properties.Calling
initialize(Map)
will overwrite information set in this constructor.- Parameters:
s3
- s3 supplier
-
S3FileIO
public S3FileIO(SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3, AwsProperties awsProperties)
Constructor with custom s3 supplier and AWS properties.Calling
initialize(Map)
will overwrite information set in this constructor.- Parameters:
s3
- s3 supplierawsProperties
- aws properties
-
-
Method Detail
-
newInputFile
public InputFile newInputFile(java.lang.String path)
Description copied from interface:FileIO
Get aInputFile
instance to read bytes from the file at the given path.- Specified by:
newInputFile
in interfaceFileIO
-
newInputFile
public InputFile newInputFile(java.lang.String path, long length)
Description copied from interface:FileIO
Get aInputFile
instance to read bytes from the file at the given path, with a known file length.- Specified by:
newInputFile
in interfaceFileIO
-
newOutputFile
public OutputFile newOutputFile(java.lang.String path)
Description copied from interface:FileIO
Get aOutputFile
instance to write bytes to the file at the given path.- Specified by:
newOutputFile
in interfaceFileIO
-
deleteFile
public void deleteFile(java.lang.String path)
Description copied from interface:FileIO
Delete the file at the given path.- Specified by:
deleteFile
in interfaceFileIO
-
properties
public java.util.Map<java.lang.String,java.lang.String> properties()
Description copied from interface:FileIO
Returns the property map used to configure this FileIO- Specified by:
properties
in interfaceFileIO
-
deleteFiles
public void deleteFiles(java.lang.Iterable<java.lang.String> paths) throws BulkDeletionFailureException
Deletes the given paths in a batched manner.The paths are grouped by bucket, and deletion is triggered when we either reach the configured batch size or have a final remainder batch for each bucket.
- Specified by:
deleteFiles
in interfaceSupportsBulkOperations
- Parameters:
paths
- paths to delete- Throws:
BulkDeletionFailureException
- in case of failure to delete at least 1 file
-
listPrefix
public java.lang.Iterable<FileInfo> listPrefix(java.lang.String prefix)
Description copied from interface:SupportsPrefixOperations
Return an iterable of all files under a prefix.Hierarchical file systems (e.g. HDFS) may impose additional restrictions like the prefix must fully match a directory whereas key/value object stores may allow for arbitrary prefixes.
- Specified by:
listPrefix
in interfaceSupportsPrefixOperations
- Parameters:
prefix
- prefix to list- Returns:
- iterable of file information
-
deletePrefix
public void deletePrefix(java.lang.String prefix)
This method provides a "best-effort" to delete all objects under the given prefix.Bulk delete operations are used and no reattempt is made for deletes if they fail, but will log any individual objects that are not deleted as part of the bulk operation.
- Specified by:
deletePrefix
in interfaceSupportsPrefixOperations
- Parameters:
prefix
- prefix to delete
-
getCredential
public java.lang.String getCredential()
Description copied from interface:CredentialSupplier
Returns the credential string- Specified by:
getCredential
in interfaceCredentialSupplier
-
initialize
public void initialize(java.util.Map<java.lang.String,java.lang.String> props)
Description copied from interface:FileIO
Initialize File IO from catalog properties.- Specified by:
initialize
in interfaceFileIO
- Parameters:
props
- catalog properties
-
close
public void close()
Description copied from interface:FileIO
Close File IO to release underlying resources.Calling this method is only required when this FileIO instance is no longer expected to be used, and the resources it holds need to be explicitly released to avoid resource leaks.
-
-