org.apache.iceberg.aws

Class AwsProperties

java.lang.Object

org.apache.iceberg.aws.AwsProperties

All Implemented Interfaces:

java.io.Serializable

public class AwsProperties
extends java.lang.Object
implements java.io.Serializable

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CLIENT_ASSUME_ROLE_ARN Used by AssumeRoleAwsClientFactory.
static java.lang.String	CLIENT_ASSUME_ROLE_EXTERNAL_ID Used by AssumeRoleAwsClientFactory.
static java.lang.String	CLIENT_ASSUME_ROLE_REGION Used by AssumeRoleAwsClientFactory.
static java.lang.String	CLIENT_ASSUME_ROLE_TIMEOUT_SEC Used by AssumeRoleAwsClientFactory.
static int	CLIENT_ASSUME_ROLE_TIMEOUT_SEC_DEFAULT
static boolean	CLIENT_ENABLE_ETAG_CHECK_DEFAULT
static java.lang.String	CLIENT_FACTORY The implementation class of AwsClientFactory to customize AWS client configurations.
static java.lang.String	DYNAMODB_TABLE_NAME DynamoDB table name for DynamoDbCatalog
static java.lang.String	DYNAMODB_TABLE_NAME_DEFAULT
static java.lang.String	GLUE_CATALOG_ID The ID of the Glue Data Catalog where the tables reside.
static java.lang.String	GLUE_CATALOG_SKIP_ARCHIVE If Glue should skip archiving an old table version when creating a new version in a commit.
static boolean	GLUE_CATALOG_SKIP_ARCHIVE_DEFAULT
static java.lang.String	S3_CHECKSUM_ENABLED Enables eTag checks for S3 PUT and MULTIPART upload requests.
static java.lang.String	S3FILEIO_ACCESS_KEY_ID Configure the static access key ID used to access S3FileIO.
static java.lang.String	S3FILEIO_ACL Used to configure canned access control list (ACL) for S3 client to use during write.
static java.lang.String	S3FILEIO_ENDPOINT Configure an alternative endpoint of the S3 service for S3FileIO to access.
static java.lang.String	S3FILEIO_MULTIPART_SIZE The size of a single part for multipart upload requests in bytes (default: 32MB).
static int	S3FILEIO_MULTIPART_SIZE_DEFAULT
static int	S3FILEIO_MULTIPART_SIZE_MIN
static java.lang.String	S3FILEIO_MULTIPART_THRESHOLD_FACTOR The threshold expressed as a factor times the multipart size at which to switch from uploading using a single put object request to uploading using multipart upload (default: 1.5).
static double	S3FILEIO_MULTIPART_THRESHOLD_FACTOR_DEFAULT
static java.lang.String	S3FILEIO_MULTIPART_UPLOAD_THREADS Number of threads to use for uploading parts to S3 (shared pool across all output streams), default to Runtime.availableProcessors()
static java.lang.String	S3FILEIO_SECRET_ACCESS_KEY Configure the static secret access key used to access S3FileIO.
static java.lang.String	S3FILEIO_SESSION_TOKEN Configure the static session token used to access S3FileIO.
static java.lang.String	S3FILEIO_SSE_KEY If S3 encryption type is SSE-KMS, input is a KMS Key ID or ARN.
static java.lang.String	S3FILEIO_SSE_MD5 If S3 encryption type is SSE-C, input is the base-64 MD5 digest of the secret key.
static java.lang.String	S3FILEIO_SSE_TYPE Type of S3 Server side encryption used, default to S3FILEIO_SSE_TYPE_NONE.
static java.lang.String	S3FILEIO_SSE_TYPE_CUSTOM S3 SSE-C encryption.
static java.lang.String	S3FILEIO_SSE_TYPE_KMS S3 SSE-KMS encryption.
static java.lang.String	S3FILEIO_SSE_TYPE_NONE No server side encryption.
static java.lang.String	S3FILEIO_SSE_TYPE_S3 S3 SSE-S3 encryption.
static java.lang.String	S3FILEIO_STAGING_DIRECTORY Location to put staging files for upload to S3, default to temp directory set in java.io.tmpdir.

Constructor Summary

Constructors

Constructor and Description
AwsProperties()
AwsProperties(java.util.Map<java.lang.String,java.lang.String> properties)

Method Summary

Modifier and Type	Method and Description
java.lang.String	dynamoDbTableName()
java.lang.String	glueCatalogId()
boolean	glueCatalogSkipArchive()
boolean	isS3ChecksumEnabled()
software.amazon.awssdk.services.s3.model.ObjectCannedACL	s3FileIoAcl()
int	s3FileIoMultiPartSize()
double	s3FileIOMultipartThresholdFactor()
int	s3FileIoMultipartUploadThreads()
java.lang.String	s3FileIoSseKey()
java.lang.String	s3FileIoSseMd5()
java.lang.String	s3FileIoSseType()
java.lang.String	s3fileIoStagingDirectory()
void	setDynamoDbTableName(java.lang.String name)
void	setGlueCatalogId(java.lang.String id)
void	setGlueCatalogSkipArchive(boolean skipArchive)
void	setS3ChecksumEnabled(boolean eTagCheckEnabled)
void	setS3FileIoAcl(software.amazon.awssdk.services.s3.model.ObjectCannedACL acl)
void	setS3FileIoMultiPartSize(int size)
void	setS3FileIoMultipartThresholdFactor(double factor)
void	setS3FileIoMultipartUploadThreads(int threads)
void	setS3FileIoSseKey(java.lang.String sseKey)
void	setS3FileIoSseMd5(java.lang.String sseMd5)
void	setS3FileIoSseType(java.lang.String sseType)
void	setS3fileIoStagingDirectory(java.lang.String directory)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

S3FILEIO_SSE_TYPE

public static final java.lang.String S3FILEIO_SSE_TYPE

Type of S3 Server side encryption used, default to S3FILEIO_SSE_TYPE_NONE.

For more details: https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html

See Also:

Constant Field Values

S3FILEIO_SSE_TYPE_NONE

public static final java.lang.String S3FILEIO_SSE_TYPE_NONE

No server side encryption.

See Also:

Constant Field Values

S3FILEIO_SSE_TYPE_KMS

public static final java.lang.String S3FILEIO_SSE_TYPE_KMS

S3 SSE-KMS encryption.

For more details: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html

See Also:

Constant Field Values

S3FILEIO_SSE_TYPE_S3

public static final java.lang.String S3FILEIO_SSE_TYPE_S3

S3 SSE-S3 encryption.

For more details: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html

See Also:

Constant Field Values

S3FILEIO_SSE_TYPE_CUSTOM

public static final java.lang.String S3FILEIO_SSE_TYPE_CUSTOM

S3 SSE-C encryption.

For more details: https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html

See Also:

Constant Field Values

S3FILEIO_SSE_KEY

public static final java.lang.String S3FILEIO_SSE_KEY

If S3 encryption type is SSE-KMS, input is a KMS Key ID or ARN. In case this property is not set, default key "aws/s3" is used. If encryption type is SSE-C, input is a custom base-64 AES256 symmetric key.

See Also:

Constant Field Values

S3FILEIO_SSE_MD5

public static final java.lang.String S3FILEIO_SSE_MD5

If S3 encryption type is SSE-C, input is the base-64 MD5 digest of the secret key. This MD5 must be explicitly passed in by the caller to ensure key integrity.

See Also:

Constant Field Values

GLUE_CATALOG_ID

public static final java.lang.String GLUE_CATALOG_ID

The ID of the Glue Data Catalog where the tables reside. If none is provided, Glue automatically uses the caller's AWS account ID by default.

For more details, see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-databases.html

See Also:

Constant Field Values

GLUE_CATALOG_SKIP_ARCHIVE

public static final java.lang.String GLUE_CATALOG_SKIP_ARCHIVE

If Glue should skip archiving an old table version when creating a new version in a commit. By default Glue archives all old table versions after an UpdateTable call, but Glue has a default max number of archived table versions (can be increased). So for streaming use case with lots of commits, it is recommended to set this value to true.

See Also:

Constant Field Values

GLUE_CATALOG_SKIP_ARCHIVE_DEFAULT

public static final boolean GLUE_CATALOG_SKIP_ARCHIVE_DEFAULT

See Also:

Constant Field Values

S3FILEIO_MULTIPART_UPLOAD_THREADS

public static final java.lang.String S3FILEIO_MULTIPART_UPLOAD_THREADS

Number of threads to use for uploading parts to S3 (shared pool across all output streams), default to Runtime.availableProcessors()

See Also:

Constant Field Values

S3FILEIO_MULTIPART_SIZE

public static final java.lang.String S3FILEIO_MULTIPART_SIZE

The size of a single part for multipart upload requests in bytes (default: 32MB). based on S3 requirement, the part size must be at least 5MB. Too ensure performance of the reader and writer, the part size must be less than 2GB.

For more details, see https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

See Also:

Constant Field Values

S3FILEIO_MULTIPART_SIZE_DEFAULT

public static final int S3FILEIO_MULTIPART_SIZE_DEFAULT

See Also:

Constant Field Values

S3FILEIO_MULTIPART_SIZE_MIN

public static final int S3FILEIO_MULTIPART_SIZE_MIN

See Also:

Constant Field Values

S3FILEIO_MULTIPART_THRESHOLD_FACTOR

public static final java.lang.String S3FILEIO_MULTIPART_THRESHOLD_FACTOR

The threshold expressed as a factor times the multipart size at which to switch from uploading using a single put object request to uploading using multipart upload (default: 1.5).

See Also:

Constant Field Values

S3FILEIO_MULTIPART_THRESHOLD_FACTOR_DEFAULT

public static final double S3FILEIO_MULTIPART_THRESHOLD_FACTOR_DEFAULT

See Also:

Constant Field Values

S3FILEIO_STAGING_DIRECTORY

public static final java.lang.String S3FILEIO_STAGING_DIRECTORY

Location to put staging files for upload to S3, default to temp directory set in java.io.tmpdir.

See Also:

Constant Field Values

S3FILEIO_ACL

public static final java.lang.String S3FILEIO_ACL

Used to configure canned access control list (ACL) for S3 client to use during write. If not set, ACL will not be set for requests.

The input must be one of ObjectCannedACL, such as 'public-read-write' For more details: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html

See Also:

Constant Field Values

S3FILEIO_ENDPOINT

public static final java.lang.String S3FILEIO_ENDPOINT

Configure an alternative endpoint of the S3 service for S3FileIO to access.

This could be used to use S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud.

See Also:

Constant Field Values

S3FILEIO_ACCESS_KEY_ID

public static final java.lang.String S3FILEIO_ACCESS_KEY_ID

Configure the static access key ID used to access S3FileIO.

When set, the default client factory will use the basic or session credentials provided instead of reading the default credential chain to create S3 access credentials. If S3FILEIO_SESSION_TOKEN is set, session credential is used, otherwise basic credential is used.

See Also:

Constant Field Values

S3FILEIO_SECRET_ACCESS_KEY

public static final java.lang.String S3FILEIO_SECRET_ACCESS_KEY

Configure the static secret access key used to access S3FileIO.

When set, the default client factory will use the basic or session credentials provided instead of reading the default credential chain to create S3 access credentials. If S3FILEIO_SESSION_TOKEN is set, session credential is used, otherwise basic credential is used.

See Also:

Constant Field Values

S3FILEIO_SESSION_TOKEN

public static final java.lang.String S3FILEIO_SESSION_TOKEN

Configure the static session token used to access S3FileIO.

When set, the default client factory will use the session credentials provided instead of reading the default credential chain to create S3 access credentials.

See Also:

Constant Field Values

DYNAMODB_TABLE_NAME

public static final java.lang.String DYNAMODB_TABLE_NAME

DynamoDB table name for DynamoDbCatalog

See Also:

Constant Field Values

DYNAMODB_TABLE_NAME_DEFAULT

public static final java.lang.String DYNAMODB_TABLE_NAME_DEFAULT

See Also:

Constant Field Values

CLIENT_FACTORY

public static final java.lang.String CLIENT_FACTORY

The implementation class of AwsClientFactory to customize AWS client configurations. If set, all AWS clients will be initialized by the specified factory. If not set, AwsClientFactories.defaultFactory() is used as default factory.

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_ARN

public static final java.lang.String CLIENT_ASSUME_ROLE_ARN

Used by AssumeRoleAwsClientFactory. If set, all AWS clients will assume a role of the given ARN, instead of using the default credential chain.

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_TIMEOUT_SEC

public static final java.lang.String CLIENT_ASSUME_ROLE_TIMEOUT_SEC

Used by AssumeRoleAwsClientFactory. The timeout of the assume role session in seconds, default to 1 hour. At the end of the timeout, a new set of role session credentials will be fetched through a STS client.

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_TIMEOUT_SEC_DEFAULT

public static final int CLIENT_ASSUME_ROLE_TIMEOUT_SEC_DEFAULT

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_EXTERNAL_ID

public static final java.lang.String CLIENT_ASSUME_ROLE_EXTERNAL_ID

Used by AssumeRoleAwsClientFactory. Optional external ID used to assume an IAM role.

For more details, see https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_REGION

public static final java.lang.String CLIENT_ASSUME_ROLE_REGION

Used by AssumeRoleAwsClientFactory. If set, all AWS clients except STS client will use the given region instead of the default region chain.

The value must be one of Region, such as 'us-east-1'. For more details, see https://docs.aws.amazon.com/general/latest/gr/rande.html

See Also:

Constant Field Values

S3_CHECKSUM_ENABLED

public static final java.lang.String S3_CHECKSUM_ENABLED

Enables eTag checks for S3 PUT and MULTIPART upload requests.

See Also:

Constant Field Values

CLIENT_ENABLE_ETAG_CHECK_DEFAULT

public static final boolean CLIENT_ENABLE_ETAG_CHECK_DEFAULT

See Also:

Constant Field Values

Constructor Detail

AwsProperties

public AwsProperties()

AwsProperties

public AwsProperties(java.util.Map<java.lang.String,java.lang.String> properties)

Method Detail

s3FileIoSseType

public java.lang.String s3FileIoSseType()

setS3FileIoSseType

public void setS3FileIoSseType(java.lang.String sseType)

s3FileIoSseKey

public java.lang.String s3FileIoSseKey()

setS3FileIoSseKey

public void setS3FileIoSseKey(java.lang.String sseKey)

s3FileIoSseMd5

public java.lang.String s3FileIoSseMd5()

setS3FileIoSseMd5

public void setS3FileIoSseMd5(java.lang.String sseMd5)

glueCatalogId

public java.lang.String glueCatalogId()

setGlueCatalogId

public void setGlueCatalogId(java.lang.String id)

glueCatalogSkipArchive

public boolean glueCatalogSkipArchive()

setGlueCatalogSkipArchive

public void setGlueCatalogSkipArchive(boolean skipArchive)

s3FileIoMultipartUploadThreads

public int s3FileIoMultipartUploadThreads()

setS3FileIoMultipartUploadThreads

public void setS3FileIoMultipartUploadThreads(int threads)

s3FileIoMultiPartSize

public int s3FileIoMultiPartSize()

setS3FileIoMultiPartSize

public void setS3FileIoMultiPartSize(int size)

s3FileIOMultipartThresholdFactor

public double s3FileIOMultipartThresholdFactor()

setS3FileIoMultipartThresholdFactor

public void setS3FileIoMultipartThresholdFactor(double factor)

s3fileIoStagingDirectory

public java.lang.String s3fileIoStagingDirectory()

setS3fileIoStagingDirectory

public void setS3fileIoStagingDirectory(java.lang.String directory)

s3FileIoAcl

public software.amazon.awssdk.services.s3.model.ObjectCannedACL s3FileIoAcl()

setS3FileIoAcl

public void setS3FileIoAcl(software.amazon.awssdk.services.s3.model.ObjectCannedACL acl)

dynamoDbTableName

public java.lang.String dynamoDbTableName()

setDynamoDbTableName

public void setDynamoDbTableName(java.lang.String name)

isS3ChecksumEnabled

public boolean isS3ChecksumEnabled()

setS3ChecksumEnabled

public void setS3ChecksumEnabled(boolean eTagCheckEnabled)