org.apache.iceberg.hadoop

Class HadoopCatalog

java.lang.Object

org.apache.iceberg.BaseMetastoreCatalog

org.apache.iceberg.hadoop.HadoopCatalog

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, Catalog, SupportsNamespaces

public class HadoopCatalog
extends BaseMetastoreCatalog
implements java.io.Closeable, SupportsNamespaces

HadoopCatalog provides a way to use table names like db.table to work with path-based tables under a common location. It uses a specified directory under a specified filesystem as the warehouse directory, and organizes multiple levels directories that mapped to the database, namespace and the table respectively. The HadoopCatalog takes a location as the warehouse directory. When creating a table such as $db.$tbl, it creates $db/$tbl directory under the warehouse directory, and put the table metadata into that directory. The HadoopCatalog now supports Catalog.createTable(org.apache.iceberg.catalog.TableIdentifier, org.apache.iceberg.Schema, org.apache.iceberg.PartitionSpec, java.lang.String, java.util.Map<java.lang.String, java.lang.String>), Catalog.dropTable(org.apache.iceberg.catalog.TableIdentifier), the Catalog.renameTable(org.apache.iceberg.catalog.TableIdentifier, org.apache.iceberg.catalog.TableIdentifier) is not supported yet. Note: The HadoopCatalog requires that the underlying file system supports atomic rename.

Constructor Summary

Constructors

Constructor and Description
HadoopCatalog(org.apache.hadoop.conf.Configuration conf) The constructor of the HadoopCatalog.
HadoopCatalog(org.apache.hadoop.conf.Configuration conf, java.lang.String warehouseLocation) The constructor of the HadoopCatalog.
HadoopCatalog(java.lang.String name, org.apache.hadoop.conf.Configuration conf, java.lang.String warehouseLocation) The constructor of the HadoopCatalog.

Method Summary

Modifier and Type	Method and Description
void	close()
void	createNamespace(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> meta) Create a namespace in the catalog.
Table	createTable(TableIdentifier identifier, Schema schema, PartitionSpec spec, java.lang.String location, java.util.Map<java.lang.String,java.lang.String> properties) Create a table.
protected java.lang.String	defaultWarehouseLocation(TableIdentifier tableIdentifier)
boolean	dropNamespace(Namespace namespace) Drop a namespace.
boolean	dropTable(TableIdentifier identifier, boolean purge) Drop a table; optionally delete data and metadata files.
protected boolean	isValidIdentifier(TableIdentifier identifier)
java.util.List<Namespace>	listNamespaces(Namespace namespace) List namespaces from the namespace.
java.util.List<TableIdentifier>	listTables(Namespace namespace) Return all the identifiers under this namespace.
java.util.Map<java.lang.String,java.lang.String>	loadNamespaceMetadata(Namespace namespace) Load metadata properties for a namespace.
protected java.lang.String	name()
protected TableOperations	newTableOps(TableIdentifier identifier)
boolean	removeProperties(Namespace namespace, java.util.Set<java.lang.String> properties) Remove a set of metadata from a namespace in the catalog.
void	renameTable(TableIdentifier from, TableIdentifier to) Rename a table.
boolean	setProperties(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> properties) Apply a set of metadata to a namespace in the catalog.

Methods inherited from class org.apache.iceberg.BaseMetastoreCatalog

dropTableData, fullTableName, loadTable, newCreateTableTransaction, newReplaceTableTransaction, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.iceberg.catalog.SupportsNamespaces

createNamespace, listNamespaces, namespaceExists

Methods inherited from interface org.apache.iceberg.catalog.Catalog

createTable, createTable, createTable, dropTable, newCreateTableTransaction, newCreateTableTransaction, newCreateTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, tableExists

Constructor Detail

HadoopCatalog

public HadoopCatalog(java.lang.String name,
                     org.apache.hadoop.conf.Configuration conf,
                     java.lang.String warehouseLocation)

The constructor of the HadoopCatalog. It uses the passed location as its warehouse directory.

Parameters:

name - The catalog name

conf - The Hadoop configuration

warehouseLocation - The location used as warehouse directory

HadoopCatalog

public HadoopCatalog(org.apache.hadoop.conf.Configuration conf,
                     java.lang.String warehouseLocation)

The constructor of the HadoopCatalog. It uses the passed location as its warehouse directory.

Parameters:

conf - The Hadoop configuration

warehouseLocation - The location used as warehouse directory

HadoopCatalog

public HadoopCatalog(org.apache.hadoop.conf.Configuration conf)

The constructor of the HadoopCatalog. It gets the value of fs.defaultFS property from the passed Hadoop configuration as its default file system, and use the default directory iceberg/warehouse as the warehouse directory.

Parameters:

conf - The Hadoop configuration

Method Detail

name

protected java.lang.String name()

Specified by:

name in class BaseMetastoreCatalog

listTables

public java.util.List<TableIdentifier> listTables(Namespace namespace)

Description copied from interface: Catalog

Return all the identifiers under this namespace.

Specified by:

listTables in interface Catalog

Parameters:

namespace - a namespace

Returns:

a list of identifiers for tables

createTable

public Table createTable(TableIdentifier identifier,
                         Schema schema,
                         PartitionSpec spec,
                         java.lang.String location,
                         java.util.Map<java.lang.String,java.lang.String> properties)

Description copied from interface: Catalog

Create a table.

Specified by:

createTable in interface Catalog

Overrides:

createTable in class BaseMetastoreCatalog

Parameters:

identifier - a table identifier

schema - a schema

spec - a partition spec

location - a location for the table; leave null if unspecified

properties - a string map of table properties

Returns:

a Table instance

isValidIdentifier

protected boolean isValidIdentifier(TableIdentifier identifier)

Overrides:

isValidIdentifier in class BaseMetastoreCatalog

newTableOps

protected TableOperations newTableOps(TableIdentifier identifier)

Specified by:

newTableOps in class BaseMetastoreCatalog

defaultWarehouseLocation

protected java.lang.String defaultWarehouseLocation(TableIdentifier tableIdentifier)

Specified by:

defaultWarehouseLocation in class BaseMetastoreCatalog

dropTable

public boolean dropTable(TableIdentifier identifier,
                         boolean purge)

Description copied from interface: Catalog

Drop a table; optionally delete data and metadata files.

If purge is set to true the implementation should delete all data and metadata files.

Specified by:

dropTable in interface Catalog

Parameters:

identifier - a table identifier

purge - if true, delete all data and metadata files in the table

Returns:

true if the table was dropped, false if the table did not exist

renameTable

public void renameTable(TableIdentifier from,
                        TableIdentifier to)

Description copied from interface: Catalog

Rename a table.

Specified by:

renameTable in interface Catalog

Parameters:

from - identifier of the table to rename

to - new table name

createNamespace

public void createNamespace(Namespace namespace,
                            java.util.Map<java.lang.String,java.lang.String> meta)

Description copied from interface: SupportsNamespaces

Create a namespace in the catalog.

Specified by:

createNamespace in interface SupportsNamespaces

Parameters:

namespace - a multi-part namespace

meta - a string Map of properties for the given namespace

listNamespaces

public java.util.List<Namespace> listNamespaces(Namespace namespace)

Description copied from interface: SupportsNamespaces

List namespaces from the namespace.

For example, if table a.b.t exists, use 'SELECT NAMESPACE IN a' this method must return Namepace.of("a","b") Namespace.

Specified by:

listNamespaces in interface SupportsNamespaces

Returns:

a List of namespace Namespace names

dropNamespace

public boolean dropNamespace(Namespace namespace)

Description copied from interface: SupportsNamespaces

Drop a namespace. If the namespace exists and was dropped, this will return true.

Specified by:

dropNamespace in interface SupportsNamespaces

Parameters:

namespace - a namespace. Namespace

Returns:

true if the namespace was dropped, false otherwise.

setProperties

public boolean setProperties(Namespace namespace,
                             java.util.Map<java.lang.String,java.lang.String> properties)

Description copied from interface: SupportsNamespaces

Apply a set of metadata to a namespace in the catalog.

Specified by:

setProperties in interface SupportsNamespaces

Parameters:

namespace - a namespace. Namespace

properties - a collection of metadata to apply to the namespace

removeProperties

public boolean removeProperties(Namespace namespace,
                                java.util.Set<java.lang.String> properties)

Description copied from interface: SupportsNamespaces

Remove a set of metadata from a namespace in the catalog.

Specified by:

removeProperties in interface SupportsNamespaces

Parameters:

namespace - a namespace. Namespace

properties - a collection of metadata to apply to the namespace

loadNamespaceMetadata

public java.util.Map<java.lang.String,java.lang.String> loadNamespaceMetadata(Namespace namespace)

Description copied from interface: SupportsNamespaces

Load metadata properties for a namespace.

Specified by:

loadNamespaceMetadata in interface SupportsNamespaces

Parameters:

namespace - a namespace. Namespace

Returns:

a string map of properties for the given namespace

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException