org.apache.iceberg.hive

Class HiveTableOperations

java.lang.Object

org.apache.iceberg.BaseMetastoreTableOperations

org.apache.iceberg.hive.HiveTableOperations

All Implemented Interfaces:

TableOperations

public class HiveTableOperations
extends BaseMetastoreTableOperations

TODO we should be able to extract some more commonalities to BaseMetastoreTableOperations to avoid code duplication between this class and Metacat Tables.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.iceberg.BaseMetastoreTableOperations

BaseMetastoreTableOperations.CommitStatus

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	HIVE_TABLE_PROPERTY_MAX_SIZE
static long	HIVE_TABLE_PROPERTY_MAX_SIZE_DEFAULT
static java.lang.String	NO_LOCK_EXPECTED_KEY
static java.lang.String	NO_LOCK_EXPECTED_VALUE

Fields inherited from class org.apache.iceberg.BaseMetastoreTableOperations

ICEBERG_TABLE_TYPE_VALUE, METADATA_LOCATION_PROP, PREVIOUS_METADATA_LOCATION_PROP, TABLE_TYPE_PROP

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HiveTableOperations(org.apache.hadoop.conf.Configuration conf, ClientPool<org.apache.hadoop.hive.metastore.IMetaStoreClient,org.apache.thrift.TException> metaClients, FileIO fileIO, java.lang.String catalogName, java.lang.String database, java.lang.String table)

Method Summary

Modifier and Type	Method and Description
static void	cleanupMetadata(FileIO io, java.lang.String commitStatus, java.lang.String metadataLocation)
java.lang.String	database()
protected void	doCommit(TableMetadata base, TableMetadata metadata)
protected void	doRefresh()
default boolean	exposeInHmsProperties()
default java.util.Map<java.lang.String,java.lang.String>	hmsEnvContext(java.lang.String metadataLocation)
FileIO	io() Returns a FileIO to read and write table data and metadata files.
long	maxHiveTablePropertySize()
ClientPool<org.apache.hadoop.hive.metastore.IMetaStoreClient,org.apache.thrift.TException>	metaClients()
default org.apache.hadoop.hive.metastore.api.Table	newHmsTable(java.lang.String hmsTableOwner)
default void	persistTable(org.apache.hadoop.hive.metastore.api.Table hmsTable, boolean updateHiveTable, java.lang.String metadataLocation)
default void	setField(java.util.Map<java.lang.String,java.lang.String> parameters, java.lang.String key, java.lang.String value)
default void	setSchema(TableMetadata metadata, java.util.Map<java.lang.String,java.lang.String> parameters)
static org.apache.hadoop.hive.metastore.api.StorageDescriptor	storageDescriptor(TableMetadata metadata, boolean hiveEngineEnabled)
java.lang.String	table()
protected java.lang.String	tableName() The full name of the table used for logging purposes only.
org.apache.hadoop.hive.metastore.TableType	tableType()
static java.lang.String	translateToIcebergProp(java.lang.String hmsProp) Provides key translation where necessary between Iceberg and HMS props.
static void	validateTableIsIceberg(org.apache.hadoop.hive.metastore.api.Table table, java.lang.String fullName)

Methods inherited from class org.apache.iceberg.BaseMetastoreTableOperations

checkCommitStatus, commit, current, currentMetadataLocation, currentVersion, disableRefresh, locationProvider, metadataFileLocation, refresh, refreshFromMetadataLocation, refreshFromMetadataLocation, refreshFromMetadataLocation, refreshFromMetadataLocation, requestRefresh, temp, writeNewMetadata, writeNewMetadataIfRequired

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.TableOperations

encryption, newSnapshotId, requireStrictCleanup

Field Detail

HIVE_TABLE_PROPERTY_MAX_SIZE

public static final java.lang.String HIVE_TABLE_PROPERTY_MAX_SIZE

See Also:

Constant Field Values

HIVE_TABLE_PROPERTY_MAX_SIZE_DEFAULT

public static final long HIVE_TABLE_PROPERTY_MAX_SIZE_DEFAULT

See Also:

Constant Field Values

NO_LOCK_EXPECTED_KEY

public static final java.lang.String NO_LOCK_EXPECTED_KEY

See Also:

Constant Field Values

NO_LOCK_EXPECTED_VALUE

public static final java.lang.String NO_LOCK_EXPECTED_VALUE

See Also:

Constant Field Values

Constructor Detail

HiveTableOperations

protected HiveTableOperations(org.apache.hadoop.conf.Configuration conf,
                              ClientPool<org.apache.hadoop.hive.metastore.IMetaStoreClient,org.apache.thrift.TException> metaClients,
                              FileIO fileIO,
                              java.lang.String catalogName,
                              java.lang.String database,
                              java.lang.String table)

Method Detail

translateToIcebergProp

public static java.lang.String translateToIcebergProp(java.lang.String hmsProp)

Provides key translation where necessary between Iceberg and HMS props. This translation is needed because some properties control the same behaviour but are named differently in Iceberg and Hive. Therefore changes to these property pairs should be synchronized.

Example: Deleting data files upon DROP TABLE is enabled using gc.enabled=true in Iceberg and external.table.purge=true in Hive. Hive and Iceberg users are unaware of each other's control flags, therefore inconsistent behaviour can occur from e.g. a Hive user's point of view if external.table.purge=true is set on the HMS table but gc.enabled=false is set on the Iceberg table, resulting in no data file deletion.

Parameters:

hmsProp - The HMS property that should be translated to Iceberg property

Returns:

Iceberg property equivalent to the hmsProp. If no such translation exists, the original hmsProp is returned

tableName

protected java.lang.String tableName()

Description copied from class: BaseMetastoreTableOperations

The full name of the table used for logging purposes only. For example for HiveTableOperations it is catalogName + "." + database + "." + table.

Specified by:

tableName in class BaseMetastoreTableOperations

Returns:

The full name

io

public FileIO io()

Description copied from interface: TableOperations

Returns a FileIO to read and write table data and metadata files.

Specified by:

io in interface TableOperations

doRefresh

protected void doRefresh()

Overrides:

doRefresh in class BaseMetastoreTableOperations

doCommit

protected void doCommit(TableMetadata base,
                        TableMetadata metadata)

Overrides:

doCommit in class BaseMetastoreTableOperations

maxHiveTablePropertySize

public long maxHiveTablePropertySize()

database

public java.lang.String database()

table

public java.lang.String table()

tableType

public org.apache.hadoop.hive.metastore.TableType tableType()

metaClients

public ClientPool<org.apache.hadoop.hive.metastore.IMetaStoreClient,org.apache.thrift.TException> metaClients()

hmsEnvContext

public java.util.Map<java.lang.String,java.lang.String> hmsEnvContext(java.lang.String metadataLocation)

exposeInHmsProperties

public boolean exposeInHmsProperties()

setSchema

public void setSchema(TableMetadata metadata,
                      java.util.Map<java.lang.String,java.lang.String> parameters)

setField

public void setField(java.util.Map<java.lang.String,java.lang.String> parameters,
                     java.lang.String key,
                     java.lang.String value)

validateTableIsIceberg

public static void validateTableIsIceberg(org.apache.hadoop.hive.metastore.api.Table table,
                                          java.lang.String fullName)

persistTable

public void persistTable(org.apache.hadoop.hive.metastore.api.Table hmsTable,
                         boolean updateHiveTable,
                         java.lang.String metadataLocation)
                  throws org.apache.thrift.TException,
                         java.lang.InterruptedException

Throws:

org.apache.thrift.TException

java.lang.InterruptedException

storageDescriptor

public static org.apache.hadoop.hive.metastore.api.StorageDescriptor storageDescriptor(TableMetadata metadata,
                                                                                       boolean hiveEngineEnabled)

cleanupMetadata

public static void cleanupMetadata(FileIO io,
                                   java.lang.String commitStatus,
                                   java.lang.String metadataLocation)

newHmsTable

public org.apache.hadoop.hive.metastore.api.Table newHmsTable(java.lang.String hmsTableOwner)