public class S3FileIOProperties
extends java.lang.Object
implements java.io.Serializable
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
ACCELERATION_ENABLED
Determines if S3 client will use the Acceleration Mode, default to false.
|
static boolean |
ACCELERATION_ENABLED_DEFAULT |
static java.lang.String |
ACCESS_KEY_ID
Configure the static access key ID used to access S3FileIO.
|
static java.lang.String |
ACCESS_POINTS_PREFIX
Used by
S3FileIO , prefix used for bucket access point configuration. |
static java.lang.String |
ACL
Used to configure canned access control list (ACL) for S3 client to use during write.
|
static java.lang.String |
CHECKSUM_ENABLED
Enables eTag checks for S3 PUT and MULTIPART upload requests.
|
static boolean |
CHECKSUM_ENABLED_DEFAULT |
static java.lang.String |
CLIENT_FACTORY
This property is used to pass in the aws client factory implementation class for S3 FileIO.
|
static java.lang.String |
DELETE_BATCH_SIZE
Configure the batch size used when deleting multiple files from a given S3 bucket
|
static int |
DELETE_BATCH_SIZE_DEFAULT
Default batch size used when deleting files.
|
static int |
DELETE_BATCH_SIZE_MAX
Max possible batch size for deletion.
|
static java.lang.String |
DELETE_ENABLED
Determines if
S3FileIO deletes the object when io.delete() is called, default to true. |
static boolean |
DELETE_ENABLED_DEFAULT |
static java.lang.String |
DELETE_TAGS_PREFIX
Used by
S3FileIO to tag objects when deleting. |
static java.lang.String |
DELETE_THREADS
Number of threads to use for adding delete tags to S3 objects, default to
Runtime.availableProcessors() |
static java.lang.String |
DUALSTACK_ENABLED
Determines if S3 client will use the Dualstack Mode, default to false.
|
static boolean |
DUALSTACK_ENABLED_DEFAULT |
static java.lang.String |
ENDPOINT
Configure an alternative endpoint of the S3 service for S3FileIO to access.
|
static java.lang.String |
MULTIPART_SIZE
The size of a single part for multipart upload requests in bytes (default: 32MB).
|
static int |
MULTIPART_SIZE_DEFAULT |
static int |
MULTIPART_SIZE_MIN |
static java.lang.String |
MULTIPART_THRESHOLD_FACTOR
The threshold expressed as a factor times the multipart size at which to switch from uploading
using a single put object request to uploading using multipart upload (default: 1.5).
|
static double |
MULTIPART_THRESHOLD_FACTOR_DEFAULT |
static java.lang.String |
MULTIPART_UPLOAD_THREADS
Number of threads to use for uploading parts to S3 (shared pool across all output streams),
default to
Runtime.availableProcessors() |
static java.lang.String |
PATH_STYLE_ACCESS
If set
true , requests to S3FileIO will use Path-Style, otherwise, Virtual Hosted-Style
will be used. |
static boolean |
PATH_STYLE_ACCESS_DEFAULT |
static java.lang.String |
PRELOAD_CLIENT_ENABLED
This flag controls whether the S3 client will be initialized during the S3FileIO
initialization, instead of default lazy initialization upon use.
|
static boolean |
PRELOAD_CLIENT_ENABLED_DEFAULT |
static java.lang.String |
REMOTE_SIGNING_ENABLED |
static boolean |
REMOTE_SIGNING_ENABLED_DEFAULT |
static java.lang.String |
S3_ACCESS_GRANTS_ENABLED
This property is used to enable using the S3 Access Grants product to control authorization to
S3 data.
|
static boolean |
S3_ACCESS_GRANTS_ENABLED_DEFAULT |
static java.lang.String |
S3_ACCESS_GRANTS_FALLBACK_TO_IAM_ENABLED
The fallback-to-iam property allows users to customize whether or not they would like their
jobs fall back to the Job Execution IAM role in case they get an Access Denied from the S3
Access Grants call.
|
static boolean |
S3_ACCESS_GRANTS_FALLBACK_TO_IAM_ENABLED_DEFAULT |
static java.lang.String |
S3_TAG_ICEBERG_NAMESPACE
Tag name that will be used by
WRITE_TAGS_PREFIX when WRITE_NAMESPACE_TAG_ENABLED is enabled |
static java.lang.String |
S3_TAG_ICEBERG_TABLE
Tag name that will be used by
WRITE_TAGS_PREFIX when WRITE_TABLE_TAG_ENABLED
is enabled |
static java.lang.String |
SECRET_ACCESS_KEY
Configure the static secret access key used to access S3FileIO.
|
static java.lang.String |
SESSION_TOKEN
Configure the static session token used to access S3FileIO.
|
static java.lang.String |
SSE_KEY
If S3 encryption type is SSE-KMS, input is a KMS Key ID or ARN.
|
static java.lang.String |
SSE_MD5
If S3 encryption type is SSE-C, input is the base-64 MD5 digest of the secret key.
|
static java.lang.String |
SSE_TYPE
Type of S3 Server side encryption used, default to
SSE_TYPE_NONE . |
static java.lang.String |
SSE_TYPE_CUSTOM
S3 SSE-C encryption.
|
static java.lang.String |
SSE_TYPE_KMS
S3 SSE-KMS encryption.
|
static java.lang.String |
SSE_TYPE_NONE
No server side encryption.
|
static java.lang.String |
SSE_TYPE_S3
S3 SSE-S3 encryption.
|
static java.lang.String |
STAGING_DIRECTORY
Location to put staging files for upload to S3, default to temp directory set in
java.io.tmpdir.
|
static java.lang.String |
USE_ARN_REGION_ENABLED
Enable to make S3FileIO, to make cross-region call to the region specified in the ARN of an
access point.
|
static boolean |
USE_ARN_REGION_ENABLED_DEFAULT |
static java.lang.String |
WRITE_NAMESPACE_TAG_ENABLED
Used by
GlueCatalog to tag objects when writing. |
static boolean |
WRITE_NAMESPACE_TAG_ENABLED_DEFAULT |
static java.lang.String |
WRITE_STORAGE_CLASS
Used by
S3FileIO to tag objects' storage class when writing. |
static java.lang.String |
WRITE_TABLE_TAG_ENABLED
Used by
GlueCatalog to tag objects when writing. |
static boolean |
WRITE_TABLE_TAG_ENABLED_DEFAULT |
static java.lang.String |
WRITE_TAGS_PREFIX
Used by
S3FileIO to tag objects when writing. |
Constructor and Description |
---|
S3FileIOProperties() |
S3FileIOProperties(java.util.Map<java.lang.String,java.lang.String> properties) |
Modifier and Type | Method and Description |
---|---|
java.lang.String |
accessKeyId() |
software.amazon.awssdk.services.s3.model.ObjectCannedACL |
acl() |
<T extends software.amazon.awssdk.services.s3.S3ClientBuilder> |
applyCredentialConfigurations(AwsClientProperties awsClientProperties,
T builder) |
<T extends software.amazon.awssdk.services.s3.S3ClientBuilder> |
applyEndpointConfigurations(T builder)
Override the endpoint for an S3 client.
|
<T extends software.amazon.awssdk.services.s3.S3ClientBuilder> |
applyS3AccessGrantsConfigurations(T builder)
Add the S3 Access Grants Plugin for an S3 client.
|
<T extends software.amazon.awssdk.services.s3.S3ClientBuilder> |
applyServiceConfigurations(T builder)
Configure services settings for an S3 client.
|
<T extends software.amazon.awssdk.services.s3.S3ClientBuilder> |
applySignerConfiguration(T builder)
Configure a signer for an S3 client.
|
java.util.Map<java.lang.String,java.lang.String> |
bucketToAccessPointMapping() |
int |
deleteBatchSize() |
java.util.Set<software.amazon.awssdk.services.s3.model.Tag> |
deleteTags() |
int |
deleteThreads() |
java.lang.String |
endpoint() |
boolean |
isAccelerationEnabled() |
boolean |
isChecksumEnabled() |
boolean |
isDeleteEnabled() |
boolean |
isDualStackEnabled() |
boolean |
isPathStyleAccess() |
boolean |
isPreloadClientEnabled() |
boolean |
isRemoteSigningEnabled() |
boolean |
isS3AccessGrantsEnabled() |
boolean |
isS3AccessGrantsFallbackToIamEnabled() |
boolean |
isUseArnRegionEnabled() |
boolean |
isWriteNamespaceTagEnabled() |
int |
multiPartSize() |
double |
multipartThresholdFactor() |
int |
multipartUploadThreads() |
java.lang.String |
secretAccessKey() |
java.lang.String |
sessionToken() |
void |
setAcl(software.amazon.awssdk.services.s3.model.ObjectCannedACL acl) |
void |
setChecksumEnabled(boolean eTagCheckEnabled) |
void |
setDeleteBatchSize(int deleteBatchSize) |
void |
setDeleteEnabled(boolean deleteEnabled) |
void |
setDeleteThreads(int threads) |
void |
setMultiPartSize(int size) |
void |
setMultipartThresholdFactor(double factor) |
void |
setMultipartUploadThreads(int threads) |
void |
setPreloadClientEnabled(boolean preloadClientEnabled) |
void |
setS3AccessGrantsEnabled(boolean s3AccessGrantsEnabled) |
void |
setS3AccessGrantsFallbackToIamEnabled(boolean s3AccessGrantsFallbackToIamEnabled) |
void |
setSseKey(java.lang.String sseKey) |
void |
setSseMd5(java.lang.String sseMd5) |
void |
setSseType(java.lang.String sseType) |
void |
setStagingDirectory(java.lang.String directory) |
void |
setWriteNamespaceTagEnabled(boolean writeNamespaceTagEnabled) |
void |
setWriteTableTagEnabled(boolean s3WriteTableNameTagEnabled) |
java.lang.String |
sseKey() |
java.lang.String |
sseMd5() |
java.lang.String |
sseType() |
java.lang.String |
stagingDirectory() |
java.lang.String |
writeStorageClass() |
boolean |
writeTableTagEnabled() |
java.util.Set<software.amazon.awssdk.services.s3.model.Tag> |
writeTags() |
public static final java.lang.String CLIENT_FACTORY
S3FileIOAwsClientFactory
. For example, DefaultS3FileIOAwsClientFactory
implements S3FileIOAwsClientFactory
. If this property
wasn't set, will load one of AwsClientFactory
factory classes to
provide backward compatibility.public static final java.lang.String S3_ACCESS_GRANTS_ENABLED
public static final boolean S3_ACCESS_GRANTS_ENABLED_DEFAULT
public static final java.lang.String S3_ACCESS_GRANTS_FALLBACK_TO_IAM_ENABLED
For more details, see: https://github.com/aws/aws-s3-accessgrants-plugin-java-v2
public static final boolean S3_ACCESS_GRANTS_FALLBACK_TO_IAM_ENABLED_DEFAULT
public static final java.lang.String SSE_TYPE
SSE_TYPE_NONE
.
For more details: https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
public static final java.lang.String SSE_TYPE_NONE
public static final java.lang.String SSE_TYPE_KMS
For more details: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
public static final java.lang.String SSE_TYPE_S3
For more details: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
public static final java.lang.String SSE_TYPE_CUSTOM
For more details: https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
public static final java.lang.String SSE_KEY
public static final java.lang.String SSE_MD5
public static final java.lang.String MULTIPART_UPLOAD_THREADS
Runtime.availableProcessors()
public static final java.lang.String MULTIPART_SIZE
For more details, see https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
public static final int MULTIPART_SIZE_DEFAULT
public static final int MULTIPART_SIZE_MIN
public static final java.lang.String MULTIPART_THRESHOLD_FACTOR
public static final double MULTIPART_THRESHOLD_FACTOR_DEFAULT
public static final java.lang.String STAGING_DIRECTORY
public static final java.lang.String ACL
The input must be one of ObjectCannedACL
,
such as 'public-read-write' For more details:
https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html
public static final java.lang.String ENDPOINT
This could be used to use S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud.
public static final java.lang.String PATH_STYLE_ACCESS
true
, requests to S3FileIO will use Path-Style, otherwise, Virtual Hosted-Style
will be used.
For more details: https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html
public static final boolean PATH_STYLE_ACCESS_DEFAULT
public static final java.lang.String ACCESS_KEY_ID
When set, the default client factory will use the basic or session credentials provided
instead of reading the default credential chain to create S3 access credentials. If SESSION_TOKEN
is set, session credential is used, otherwise basic credential is used.
public static final java.lang.String SECRET_ACCESS_KEY
When set, the default client factory will use the basic or session credentials provided
instead of reading the default credential chain to create S3 access credentials. If SESSION_TOKEN
is set, session credential is used, otherwise basic credential is used.
public static final java.lang.String SESSION_TOKEN
When set, the default client factory will use the session credentials provided instead of reading the default credential chain to create S3 access credentials.
public static final java.lang.String USE_ARN_REGION_ENABLED
By default, attempting to use an access point in a different region will throw an exception. When enabled, this property allows using access points in other regions.
For more details see: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3Configuration.html#useArnRegionEnabled--
public static final boolean USE_ARN_REGION_ENABLED_DEFAULT
public static final java.lang.String CHECKSUM_ENABLED
public static final boolean CHECKSUM_ENABLED_DEFAULT
public static final java.lang.String REMOTE_SIGNING_ENABLED
public static final boolean REMOTE_SIGNING_ENABLED_DEFAULT
public static final java.lang.String DELETE_BATCH_SIZE
public static final int DELETE_BATCH_SIZE_DEFAULT
Refer to https://github.com/apache/hadoop/commit/56dee667707926f3796c7757be1a133a362f05c9 for more details on why this value was chosen.
public static final int DELETE_BATCH_SIZE_MAX
public static final java.lang.String WRITE_TAGS_PREFIX
S3FileIO
to tag objects when writing. To set, we can pass a catalog property.
For more details, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html
Example: s3.write.tags.my_key=my_val
public static final java.lang.String WRITE_TABLE_TAG_ENABLED
GlueCatalog
to tag objects when writing. To set, we can pass a catalog
property.
For more details, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html
Example: s3.write.table-tag-enabled=true
public static final boolean WRITE_TABLE_TAG_ENABLED_DEFAULT
public static final java.lang.String WRITE_STORAGE_CLASS
S3FileIO
to tag objects' storage class when writing. To set, we can pass a
catalog property. After set, x-amz-storage-class header will be set to this property
For more details, see https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/storage-class-intro.html
Example: s3.write.storage-class=INTELLIGENT_TIERING
public static final java.lang.String WRITE_NAMESPACE_TAG_ENABLED
GlueCatalog
to tag objects when writing. To set, we can pass a catalog
property.
For more details, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html
Example: s3.write.namespace-tag-enabled=true
public static final boolean WRITE_NAMESPACE_TAG_ENABLED_DEFAULT
public static final java.lang.String S3_TAG_ICEBERG_TABLE
WRITE_TAGS_PREFIX
when WRITE_TABLE_TAG_ENABLED
is enabled
Example: iceberg.table=tableName
public static final java.lang.String S3_TAG_ICEBERG_NAMESPACE
WRITE_TAGS_PREFIX
when WRITE_NAMESPACE_TAG_ENABLED
is enabled
Example: iceberg.namespace=namespaceName
public static final java.lang.String DELETE_TAGS_PREFIX
S3FileIO
to tag objects when deleting. When this config is set, objects are
tagged with the configured key-value pairs before deletion. This is considered a soft-delete,
because users are able to configure tag-based object lifecycle policy at bucket level to
transition objects to different tiers.
For more details, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html
Example: s3.delete.tags.my_key=my_val
public static final java.lang.String DELETE_THREADS
Runtime.availableProcessors()
public static final java.lang.String DELETE_ENABLED
S3FileIO
deletes the object when io.delete() is called, default to true.
Once disabled, users are expected to set tags through DELETE_TAGS_PREFIX
and manage
deleted files through S3 lifecycle policy.public static final boolean DELETE_ENABLED_DEFAULT
public static final java.lang.String ACCELERATION_ENABLED
For more details, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html
public static final boolean ACCELERATION_ENABLED_DEFAULT
public static final java.lang.String DUALSTACK_ENABLED
For more details, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/dual-stack-endpoints.html
public static final boolean DUALSTACK_ENABLED_DEFAULT
public static final java.lang.String ACCESS_POINTS_PREFIX
S3FileIO
, prefix used for bucket access point configuration. To set, we can
pass a catalog property.
For more details, see https://aws.amazon.com/s3/features/access-points/
Example: s3.access-points.my-bucket=access-point
public static final java.lang.String PRELOAD_CLIENT_ENABLED
public static final boolean PRELOAD_CLIENT_ENABLED_DEFAULT
public S3FileIOProperties()
public S3FileIOProperties(java.util.Map<java.lang.String,java.lang.String> properties)
public java.lang.String sseType()
public void setSseType(java.lang.String sseType)
public java.lang.String sseKey()
public void setSseKey(java.lang.String sseKey)
public int deleteBatchSize()
public void setDeleteBatchSize(int deleteBatchSize)
public java.lang.String sseMd5()
public void setSseMd5(java.lang.String sseMd5)
public int multipartUploadThreads()
public void setMultipartUploadThreads(int threads)
public int multiPartSize()
public void setMultiPartSize(int size)
public double multipartThresholdFactor()
public void setMultipartThresholdFactor(double factor)
public java.lang.String stagingDirectory()
public void setStagingDirectory(java.lang.String directory)
public software.amazon.awssdk.services.s3.model.ObjectCannedACL acl()
public void setAcl(software.amazon.awssdk.services.s3.model.ObjectCannedACL acl)
public boolean isPreloadClientEnabled()
public void setPreloadClientEnabled(boolean preloadClientEnabled)
public boolean isDualStackEnabled()
public boolean isPathStyleAccess()
public boolean isUseArnRegionEnabled()
public boolean isAccelerationEnabled()
public boolean isChecksumEnabled()
public boolean isRemoteSigningEnabled()
public java.lang.String endpoint()
public void setChecksumEnabled(boolean eTagCheckEnabled)
public java.util.Set<software.amazon.awssdk.services.s3.model.Tag> writeTags()
public boolean writeTableTagEnabled()
public void setWriteTableTagEnabled(boolean s3WriteTableNameTagEnabled)
public boolean isWriteNamespaceTagEnabled()
public void setWriteNamespaceTagEnabled(boolean writeNamespaceTagEnabled)
public java.util.Set<software.amazon.awssdk.services.s3.model.Tag> deleteTags()
public int deleteThreads()
public void setDeleteThreads(int threads)
public boolean isDeleteEnabled()
public void setDeleteEnabled(boolean deleteEnabled)
public java.util.Map<java.lang.String,java.lang.String> bucketToAccessPointMapping()
public java.lang.String accessKeyId()
public java.lang.String secretAccessKey()
public java.lang.String sessionToken()
public java.lang.String writeStorageClass()
public boolean isS3AccessGrantsEnabled()
public void setS3AccessGrantsEnabled(boolean s3AccessGrantsEnabled)
public boolean isS3AccessGrantsFallbackToIamEnabled()
public void setS3AccessGrantsFallbackToIamEnabled(boolean s3AccessGrantsFallbackToIamEnabled)
public <T extends software.amazon.awssdk.services.s3.S3ClientBuilder> void applyCredentialConfigurations(AwsClientProperties awsClientProperties, T builder)
public <T extends software.amazon.awssdk.services.s3.S3ClientBuilder> void applyServiceConfigurations(T builder)
Sample usage:
S3Client.builder().applyMutation(s3FileIOProperties::applyS3ServiceConfigurations)
public <T extends software.amazon.awssdk.services.s3.S3ClientBuilder> void applySignerConfiguration(T builder)
Sample usage:
S3Client.builder().applyMutation(s3FileIOProperties::applyS3SignerConfiguration)
public <T extends software.amazon.awssdk.services.s3.S3ClientBuilder> void applyEndpointConfigurations(T builder)
Sample usage:
S3Client.builder().applyMutation(s3FileIOProperties::applyEndpointConfigurations)
public <T extends software.amazon.awssdk.services.s3.S3ClientBuilder> void applyS3AccessGrantsConfigurations(T builder)
Sample usage:
S3Client.builder().applyMutation(s3FileIOProperties::applyS3AccessGrantsConfigurations)