org.apache.iceberg.aws

Class AwsProperties

java.lang.Object

org.apache.iceberg.aws.AwsProperties

All Implemented Interfaces:

java.io.Serializable

public class AwsProperties
extends java.lang.Object
implements java.io.Serializable

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CLIENT_ASSUME_ROLE_ARN Used by AssumeRoleAwsClientFactory.
static java.lang.String	CLIENT_ASSUME_ROLE_EXTERNAL_ID Used by AssumeRoleAwsClientFactory.
static java.lang.String	CLIENT_ASSUME_ROLE_REGION Used by AssumeRoleAwsClientFactory.
static java.lang.String	CLIENT_ASSUME_ROLE_SESSION_NAME Used by AssumeRoleAwsClientFactory.
static java.lang.String	CLIENT_ASSUME_ROLE_TAGS_PREFIX Used by AssumeRoleAwsClientFactory to pass a list of sessions.
static java.lang.String	CLIENT_ASSUME_ROLE_TIMEOUT_SEC Used by AssumeRoleAwsClientFactory.
static int	CLIENT_ASSUME_ROLE_TIMEOUT_SEC_DEFAULT
static java.lang.String	CLIENT_FACTORY The implementation class of AwsClientFactory to customize AWS client configurations.
static java.lang.String	DYNAMODB_ENDPOINT Configure an alternative endpoint of the DynamoDB service to access.
static java.lang.String	DYNAMODB_TABLE_NAME DynamoDB table name for DynamoDbCatalog
static java.lang.String	DYNAMODB_TABLE_NAME_DEFAULT
static java.lang.String	GLUE_ACCOUNT_ID The account ID used in a Glue resource ARN, e.g.
static java.lang.String	GLUE_CATALOG_ENDPOINT Configure an alternative endpoint of the Glue service for GlueCatalog to access.
static java.lang.String	GLUE_CATALOG_ID The ID of the Glue Data Catalog where the tables reside.
static java.lang.String	GLUE_CATALOG_SKIP_ARCHIVE If Glue should skip archiving an old table version when creating a new version in a commit.
static boolean	GLUE_CATALOG_SKIP_ARCHIVE_DEFAULT
static java.lang.String	GLUE_CATALOG_SKIP_NAME_VALIDATION If Glue should skip name validations It is recommended to stick to Glue best practice in https://docs.aws.amazon.com/athena/latest/ug/glue-best-practices.html to make sure operations are Hive compatible.
static boolean	GLUE_CATALOG_SKIP_NAME_VALIDATION_DEFAULT
static java.lang.String	GLUE_LAKEFORMATION_ENABLED If set, GlueCatalog will use Lake Formation for access control.
static boolean	GLUE_LAKEFORMATION_ENABLED_DEFAULT
static java.lang.String	LAKE_FORMATION_DB_NAME Used by LakeFormationAwsClientFactory.
static java.lang.String	LAKE_FORMATION_TABLE_NAME Used by LakeFormationAwsClientFactory.
static java.lang.String	REST_ACCESS_KEY_ID Configure the static access key ID used for SigV4 signing.
static java.lang.String	REST_SECRET_ACCESS_KEY Configure the static secret access key used for SigV4 signing.
static java.lang.String	REST_SESSION_TOKEN Configure the static session token used for SigV4.
static java.lang.String	REST_SIGNER_REGION Region to be used by the SigV4 protocol for signing requests.
static java.lang.String	REST_SIGNING_NAME The service name to be used by the SigV4 protocol for signing requests.
static java.lang.String	REST_SIGNING_NAME_DEFAULT The default service name (API Gateway and lambda) used during SigV4 signing.

Constructor Summary

Constructors

Constructor and Description
AwsProperties()
AwsProperties(java.util.Map<java.lang.String,java.lang.String> properties)

Method Summary

Modifier and Type	Method and Description
<T extends software.amazon.awssdk.services.dynamodb.DynamoDbClientBuilder> void	applyDynamoDbEndpointConfigurations(T builder) Override the endpoint for a dynamoDb client.
<T extends software.amazon.awssdk.services.glue.GlueClientBuilder> void	applyGlueEndpointConfigurations(T builder) Override the endpoint for a glue client.
java.lang.String	clientAssumeRoleArn()
java.lang.String	clientAssumeRoleExternalId()
java.lang.String	clientAssumeRoleRegion()
java.lang.String	clientAssumeRoleSessionName()
int	clientAssumeRoleTimeoutSec()
java.lang.String	dynamoDbTableName()
java.lang.String	glueCatalogId()
boolean	glueCatalogSkipArchive()
boolean	glueCatalogSkipNameValidation()
boolean	glueLakeFormationEnabled()
software.amazon.awssdk.auth.credentials.AwsCredentialsProvider	restCredentialsProvider()
java.lang.String	restSigningName()
software.amazon.awssdk.regions.Region	restSigningRegion()
void	setDynamoDbTableName(java.lang.String name)
void	setGlueCatalogId(java.lang.String id)
void	setGlueCatalogSkipArchive(boolean skipArchive)
void	setGlueCatalogSkipNameValidation(boolean glueCatalogSkipNameValidation)
void	setGlueLakeFormationEnabled(boolean glueLakeFormationEnabled)
java.util.Set<software.amazon.awssdk.services.sts.model.Tag>	stsClientAssumeRoleTags()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

GLUE_CATALOG_ID

public static final java.lang.String GLUE_CATALOG_ID

The ID of the Glue Data Catalog where the tables reside. If none is provided, Glue automatically uses the caller's AWS account ID by default.

For more details, see https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-databases.html

See Also:

Constant Field Values

GLUE_ACCOUNT_ID

public static final java.lang.String GLUE_ACCOUNT_ID

The account ID used in a Glue resource ARN, e.g. arn:aws:glue:us-east-1:1000000000000:table/db1/table1

See Also:

Constant Field Values

GLUE_CATALOG_SKIP_ARCHIVE

public static final java.lang.String GLUE_CATALOG_SKIP_ARCHIVE

If Glue should skip archiving an old table version when creating a new version in a commit. By default Glue archives all old table versions after an UpdateTable call, but Glue has a default max number of archived table versions (can be increased). So for streaming use case with lots of commits, it is recommended to set this value to true.

See Also:

Constant Field Values

GLUE_CATALOG_SKIP_ARCHIVE_DEFAULT

public static final boolean GLUE_CATALOG_SKIP_ARCHIVE_DEFAULT

See Also:

Constant Field Values

GLUE_CATALOG_SKIP_NAME_VALIDATION

public static final java.lang.String GLUE_CATALOG_SKIP_NAME_VALIDATION

If Glue should skip name validations It is recommended to stick to Glue best practice in https://docs.aws.amazon.com/athena/latest/ug/glue-best-practices.html to make sure operations are Hive compatible. This is only added for users that have existing conventions using non-standard characters. When database name and table name validation are skipped, there is no guarantee that downstream systems would all support the names.

See Also:

Constant Field Values

GLUE_CATALOG_SKIP_NAME_VALIDATION_DEFAULT

public static final boolean GLUE_CATALOG_SKIP_NAME_VALIDATION_DEFAULT

See Also:

Constant Field Values

GLUE_LAKEFORMATION_ENABLED

public static final java.lang.String GLUE_LAKEFORMATION_ENABLED

If set, GlueCatalog will use Lake Formation for access control. For more credential vending details, see: https://docs.aws.amazon.com/lake-formation/latest/dg/api-overview.html. If enabled, the AwsClientFactory implementation must be LakeFormationAwsClientFactory or any class that extends it.

See Also:

Constant Field Values

GLUE_LAKEFORMATION_ENABLED_DEFAULT

public static final boolean GLUE_LAKEFORMATION_ENABLED_DEFAULT

See Also:

Constant Field Values

GLUE_CATALOG_ENDPOINT

public static final java.lang.String GLUE_CATALOG_ENDPOINT

Configure an alternative endpoint of the Glue service for GlueCatalog to access.

This could be used to use GlueCatalog with any glue-compatible metastore service that has a different endpoint

See Also:

Constant Field Values

DYNAMODB_ENDPOINT

public static final java.lang.String DYNAMODB_ENDPOINT

Configure an alternative endpoint of the DynamoDB service to access.

See Also:

Constant Field Values

DYNAMODB_TABLE_NAME

public static final java.lang.String DYNAMODB_TABLE_NAME

DynamoDB table name for DynamoDbCatalog

See Also:

Constant Field Values

DYNAMODB_TABLE_NAME_DEFAULT

public static final java.lang.String DYNAMODB_TABLE_NAME_DEFAULT

See Also:

Constant Field Values

CLIENT_FACTORY

public static final java.lang.String CLIENT_FACTORY

The implementation class of AwsClientFactory to customize AWS client configurations. If set, all AWS clients will be initialized by the specified factory. If not set, AwsClientFactories.defaultFactory() is used as default factory.

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_ARN

public static final java.lang.String CLIENT_ASSUME_ROLE_ARN

Used by AssumeRoleAwsClientFactory. If set, all AWS clients will assume a role of the given ARN, instead of using the default credential chain.

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_TAGS_PREFIX

public static final java.lang.String CLIENT_ASSUME_ROLE_TAGS_PREFIX

Used by AssumeRoleAwsClientFactory to pass a list of sessions. Each session tag consists of a key name and an associated value.

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_TIMEOUT_SEC

public static final java.lang.String CLIENT_ASSUME_ROLE_TIMEOUT_SEC

Used by AssumeRoleAwsClientFactory. The timeout of the assume role session in seconds, default to 1 hour. At the end of the timeout, a new set of role session credentials will be fetched through a STS client.

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_TIMEOUT_SEC_DEFAULT

public static final int CLIENT_ASSUME_ROLE_TIMEOUT_SEC_DEFAULT

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_EXTERNAL_ID

public static final java.lang.String CLIENT_ASSUME_ROLE_EXTERNAL_ID

Used by AssumeRoleAwsClientFactory. Optional external ID used to assume an IAM role.

For more details, see https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_REGION

public static final java.lang.String CLIENT_ASSUME_ROLE_REGION

Used by AssumeRoleAwsClientFactory. If set, all AWS clients except STS client will use the given region instead of the default region chain.

The value must be one of Region, such as 'us-east-1'. For more details, see https://docs.aws.amazon.com/general/latest/gr/rande.html

See Also:

Constant Field Values

CLIENT_ASSUME_ROLE_SESSION_NAME

public static final java.lang.String CLIENT_ASSUME_ROLE_SESSION_NAME

Used by AssumeRoleAwsClientFactory. Optional session name used to assume an IAM role.

For more details, see https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_iam-condition-keys.html#ck_rolesessionname

See Also:

Constant Field Values

LAKE_FORMATION_TABLE_NAME

public static final java.lang.String LAKE_FORMATION_TABLE_NAME

Used by LakeFormationAwsClientFactory. The table name used as part of lake formation credentials request.

See Also:

Constant Field Values

LAKE_FORMATION_DB_NAME

public static final java.lang.String LAKE_FORMATION_DB_NAME

Used by LakeFormationAwsClientFactory. The database name used as part of lake formation credentials request.

See Also:

Constant Field Values

REST_SIGNER_REGION

public static final java.lang.String REST_SIGNER_REGION

Region to be used by the SigV4 protocol for signing requests.

See Also:

Constant Field Values

REST_SIGNING_NAME

public static final java.lang.String REST_SIGNING_NAME

The service name to be used by the SigV4 protocol for signing requests.

See Also:

Constant Field Values

REST_SIGNING_NAME_DEFAULT

public static final java.lang.String REST_SIGNING_NAME_DEFAULT

The default service name (API Gateway and lambda) used during SigV4 signing.

See Also:

Constant Field Values

REST_ACCESS_KEY_ID

public static final java.lang.String REST_ACCESS_KEY_ID

Configure the static access key ID used for SigV4 signing.

When set, the default client factory will use the basic or session credentials provided instead of reading the default credential chain to create S3 access credentials. If REST_SESSION_TOKEN is set, session credential is used, otherwise basic credential is used.

See Also:

Constant Field Values

REST_SECRET_ACCESS_KEY

public static final java.lang.String REST_SECRET_ACCESS_KEY

Configure the static secret access key used for SigV4 signing.

When set, the default client factory will use the basic or session credentials provided instead of reading the default credential chain to create S3 access credentials. If REST_SESSION_TOKEN is set, session credential is used, otherwise basic credential is used.

See Also:

Constant Field Values

REST_SESSION_TOKEN

public static final java.lang.String REST_SESSION_TOKEN

Configure the static session token used for SigV4.

When set, the default client factory will use the session credentials provided instead of reading the default credential chain to create access credentials.

See Also:

Constant Field Values

Constructor Detail

AwsProperties

public AwsProperties()

AwsProperties

public AwsProperties(java.util.Map<java.lang.String,java.lang.String> properties)

Method Detail

stsClientAssumeRoleTags

public java.util.Set<software.amazon.awssdk.services.sts.model.Tag> stsClientAssumeRoleTags()

clientAssumeRoleArn

public java.lang.String clientAssumeRoleArn()

clientAssumeRoleTimeoutSec

public int clientAssumeRoleTimeoutSec()

clientAssumeRoleExternalId

public java.lang.String clientAssumeRoleExternalId()

clientAssumeRoleRegion

public java.lang.String clientAssumeRoleRegion()

clientAssumeRoleSessionName

public java.lang.String clientAssumeRoleSessionName()

glueCatalogId

public java.lang.String glueCatalogId()

setGlueCatalogId

public void setGlueCatalogId(java.lang.String id)

glueCatalogSkipArchive

public boolean glueCatalogSkipArchive()

setGlueCatalogSkipArchive

public void setGlueCatalogSkipArchive(boolean skipArchive)

glueCatalogSkipNameValidation

public boolean glueCatalogSkipNameValidation()

setGlueCatalogSkipNameValidation

public void setGlueCatalogSkipNameValidation(boolean glueCatalogSkipNameValidation)

glueLakeFormationEnabled

public boolean glueLakeFormationEnabled()

setGlueLakeFormationEnabled

public void setGlueLakeFormationEnabled(boolean glueLakeFormationEnabled)

dynamoDbTableName

public java.lang.String dynamoDbTableName()

setDynamoDbTableName

public void setDynamoDbTableName(java.lang.String name)

applyGlueEndpointConfigurations

public <T extends software.amazon.awssdk.services.glue.GlueClientBuilder> void applyGlueEndpointConfigurations(T builder)

Override the endpoint for a glue client.

Sample usage:

     GlueClient.builder().applyMutation(awsProperties::applyS3EndpointConfigurations)
 

applyDynamoDbEndpointConfigurations

public <T extends software.amazon.awssdk.services.dynamodb.DynamoDbClientBuilder> void applyDynamoDbEndpointConfigurations(T builder)

Override the endpoint for a dynamoDb client.

Sample usage:

     DynamoDbClient.builder().applyMutation(awsProperties::applyDynamoDbEndpointConfigurations)
 

restSigningRegion

public software.amazon.awssdk.regions.Region restSigningRegion()

restSigningName

public java.lang.String restSigningName()

restCredentialsProvider

public software.amazon.awssdk.auth.credentials.AwsCredentialsProvider restCredentialsProvider()