Package org.apache.iceberg

Class BaseMetastoreTableOperations

java.lang.Object

org.apache.iceberg.BaseMetastoreTableOperations

All Implemented Interfaces:

TableOperations

Direct Known Subclasses:

EcsTableOperations, HiveTableOperations, NessieTableOperations

public abstract class BaseMetastoreTableOperations
extends java.lang.Object
implements TableOperations

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	BaseMetastoreTableOperations.CommitStatus

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	ICEBERG_TABLE_TYPE_VALUE
static java.lang.String	METADATA_LOCATION_PROP
static java.lang.String	PREVIOUS_METADATA_LOCATION_PROP
static java.lang.String	TABLE_TYPE_PROP

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BaseMetastoreTableOperations()

Method Summary

Modifier and Type	Method	Description
protected BaseMetastoreTableOperations.CommitStatus	checkCommitStatus(java.lang.String newMetadataLocation, TableMetadata config)	Attempt to load the table and see if any current or past metadata location matches the one we were attempting to set.
void	commit(TableMetadata base, TableMetadata metadata)	Replace the base table metadata with a new version.
TableMetadata	current()	Return the currently loaded table metadata, without checking for updates.
java.lang.String	currentMetadataLocation()
int	currentVersion()
protected void	disableRefresh()
protected void	doCommit(TableMetadata base, TableMetadata metadata)
protected void	doRefresh()
LocationProvider	locationProvider()	Returns a LocationProvider that supplies locations for new new data files.
java.lang.String	metadataFileLocation(java.lang.String filename)	Given the name of a metadata file, obtain the full path of that file using an appropriate base location of the implementation's choosing.
TableMetadata	refresh()	Return the current table metadata after checking for updates.
protected void	refreshFromMetadataLocation(java.lang.String newLocation)
protected void	refreshFromMetadataLocation(java.lang.String newLocation, int numRetries)
protected void	refreshFromMetadataLocation(java.lang.String newLocation, java.util.function.Predicate<java.lang.Exception> shouldRetry, int numRetries)
protected void	refreshFromMetadataLocation(java.lang.String newLocation, java.util.function.Predicate<java.lang.Exception> shouldRetry, int numRetries, java.util.function.Function<java.lang.String,TableMetadata> metadataLoader)
protected void	requestRefresh()
protected abstract java.lang.String	tableName()	The full name of the table used for logging purposes only.
TableOperations	temp(TableMetadata uncommittedMetadata)	Return a temporary TableOperations instance that uses configuration from uncommitted metadata.
protected java.lang.String	writeNewMetadata(TableMetadata metadata, int newVersion)
protected java.lang.String	writeNewMetadataIfRequired(boolean newTable, TableMetadata metadata)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.TableOperations

encryption, io, newSnapshotId, requireStrictCleanup

Field Detail

TABLE_TYPE_PROP

public static final java.lang.String TABLE_TYPE_PROP

See Also:

Constant Field Values

ICEBERG_TABLE_TYPE_VALUE

public static final java.lang.String ICEBERG_TABLE_TYPE_VALUE

See Also:

Constant Field Values

METADATA_LOCATION_PROP

public static final java.lang.String METADATA_LOCATION_PROP

See Also:

Constant Field Values

PREVIOUS_METADATA_LOCATION_PROP

public static final java.lang.String PREVIOUS_METADATA_LOCATION_PROP

See Also:

Constant Field Values

Constructor Detail

BaseMetastoreTableOperations

protected BaseMetastoreTableOperations()

Method Detail

tableName

protected abstract java.lang.String tableName()

The full name of the table used for logging purposes only. For example for HiveTableOperations it is catalogName + "." + database + "." + table.

Returns:

The full name

current

public TableMetadata current()

Description copied from interface: TableOperations

Return the currently loaded table metadata, without checking for updates.

Specified by:

current in interface TableOperations

Returns:

table metadata

currentMetadataLocation

public java.lang.String currentMetadataLocation()

currentVersion

public int currentVersion()

refresh

public TableMetadata refresh()

Description copied from interface: TableOperations

Return the current table metadata after checking for updates.

Specified by:

refresh in interface TableOperations

Returns:

table metadata

doRefresh

protected void doRefresh()

commit

public void commit(TableMetadata base,
                   TableMetadata metadata)

Description copied from interface: TableOperations

Replace the base table metadata with a new version.

This method should implement and document atomicity guarantees.

Implementations must check that the base metadata is current to avoid overwriting updates. Once the atomic commit operation succeeds, implementations must not perform any operations that may fail because failure in this method cannot be distinguished from commit failure.

Implementations must throw a CommitStateUnknownException in cases where it cannot be determined if the commit succeeded or failed. For example if a network partition causes the confirmation of the commit to be lost, the implementation should throw a CommitStateUnknownException. This is important because downstream users of this API need to know whether they can clean up the commit or not, if the state is unknown then it is not safe to remove any files. All other exceptions will be treated as if the commit has failed.

Specified by:

commit in interface TableOperations

Parameters:

base - table metadata on which changes were based

metadata - new table metadata with updates

doCommit

protected void doCommit(TableMetadata base,
                        TableMetadata metadata)

requestRefresh

protected void requestRefresh()

disableRefresh

protected void disableRefresh()

writeNewMetadataIfRequired

protected java.lang.String writeNewMetadataIfRequired(boolean newTable,
                                                      TableMetadata metadata)

writeNewMetadata

protected java.lang.String writeNewMetadata(TableMetadata metadata,
                                            int newVersion)

refreshFromMetadataLocation

protected void refreshFromMetadataLocation(java.lang.String newLocation)

refreshFromMetadataLocation

protected void refreshFromMetadataLocation(java.lang.String newLocation,
                                           int numRetries)

refreshFromMetadataLocation

protected void refreshFromMetadataLocation(java.lang.String newLocation,
                                           java.util.function.Predicate<java.lang.Exception> shouldRetry,
                                           int numRetries)

refreshFromMetadataLocation

protected void refreshFromMetadataLocation(java.lang.String newLocation,
                                           java.util.function.Predicate<java.lang.Exception> shouldRetry,
                                           int numRetries,
                                           java.util.function.Function<java.lang.String,TableMetadata> metadataLoader)

metadataFileLocation

public java.lang.String metadataFileLocation(java.lang.String filename)

Description copied from interface: TableOperations

Given the name of a metadata file, obtain the full path of that file using an appropriate base location of the implementation's choosing.

The file may not exist yet, in which case the path should be returned as if it were to be created by e.g. FileIO.newOutputFile(String).

Specified by:

metadataFileLocation in interface TableOperations

locationProvider

public LocationProvider locationProvider()

Description copied from interface: TableOperations

Returns a LocationProvider that supplies locations for new new data files.

Specified by:

locationProvider in interface TableOperations

Returns:

a location provider configured for the current table state

temp

public TableOperations temp(TableMetadata uncommittedMetadata)

Description copied from interface: TableOperations

Return a temporary TableOperations instance that uses configuration from uncommitted metadata.

This is called by transactions when uncommitted table metadata should be used; for example, to create a metadata file location based on metadata in the transaction that has not been committed.

Transactions will not call TableOperations.refresh() or TableOperations.commit(TableMetadata, TableMetadata).

Specified by:

temp in interface TableOperations

Parameters:

uncommittedMetadata - uncommitted table metadata

Returns:

a temporary table operations that behaves like the uncommitted metadata is current

checkCommitStatus

protected BaseMetastoreTableOperations.CommitStatus checkCommitStatus(java.lang.String newMetadataLocation,
                                                                      TableMetadata config)

Attempt to load the table and see if any current or past metadata location matches the one we were attempting to set. This is used as a last resort when we are dealing with exceptions that may indicate the commit has failed but are not proof that this is the case. Past locations must also be searched on the chance that a second committer was able to successfully commit on top of our commit.

Parameters:

newMetadataLocation - the path of the new commit file

config - metadata to use for configuration

Returns:

Commit Status of Success, Failure or Unknown