Package org.apache.iceberg.hadoop
Class HadoopCatalog
- java.lang.Object
-
- org.apache.iceberg.BaseMetastoreCatalog
-
- org.apache.iceberg.hadoop.HadoopCatalog
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,Catalog
,SupportsNamespaces
public class HadoopCatalog extends BaseMetastoreCatalog implements java.io.Closeable, SupportsNamespaces
HadoopCatalog provides a way to use table names like db.table to work with path-based tables under a common location. It uses a specified directory under a specified filesystem as the warehouse directory, and organizes multiple levels directories that mapped to the database, namespace and the table respectively. The HadoopCatalog takes a location as the warehouse directory. When creating a table such as $db.$tbl, it creates $db/$tbl directory under the warehouse directory, and put the table metadata into that directory. The HadoopCatalog now supportsCatalog.createTable(org.apache.iceberg.catalog.TableIdentifier, org.apache.iceberg.Schema, org.apache.iceberg.PartitionSpec, java.lang.String, java.util.Map<java.lang.String, java.lang.String>)
,Catalog.dropTable(org.apache.iceberg.catalog.TableIdentifier)
, theCatalog.renameTable(org.apache.iceberg.catalog.TableIdentifier, org.apache.iceberg.catalog.TableIdentifier)
is not supported yet. Note: The HadoopCatalog requires that the underlying file system supports atomic rename.
-
-
Constructor Summary
Constructors Constructor Description HadoopCatalog(java.lang.String name, org.apache.hadoop.conf.Configuration conf, java.lang.String warehouseLocation)
The constructor of the HadoopCatalog.HadoopCatalog(org.apache.hadoop.conf.Configuration conf)
The constructor of the HadoopCatalog.HadoopCatalog(org.apache.hadoop.conf.Configuration conf, java.lang.String warehouseLocation)
The constructor of the HadoopCatalog.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
void
createNamespace(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> meta)
Create a namespace in the catalog.Table
createTable(TableIdentifier identifier, Schema schema, PartitionSpec spec, java.lang.String location, java.util.Map<java.lang.String,java.lang.String> properties)
Create a table.protected java.lang.String
defaultWarehouseLocation(TableIdentifier tableIdentifier)
boolean
dropNamespace(Namespace namespace)
Drop a namespace.boolean
dropTable(TableIdentifier identifier, boolean purge)
Drop a table; optionally delete data and metadata files.protected boolean
isValidIdentifier(TableIdentifier identifier)
java.util.List<Namespace>
listNamespaces(Namespace namespace)
List namespaces from the namespace.java.util.List<TableIdentifier>
listTables(Namespace namespace)
Return all the identifiers under this namespace.java.util.Map<java.lang.String,java.lang.String>
loadNamespaceMetadata(Namespace namespace)
Load metadata properties for a namespace.protected java.lang.String
name()
protected TableOperations
newTableOps(TableIdentifier identifier)
boolean
removeProperties(Namespace namespace, java.util.Set<java.lang.String> properties)
Remove a set of metadata from a namespace in the catalog.void
renameTable(TableIdentifier from, TableIdentifier to)
Rename a table.boolean
setProperties(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> properties)
Apply a set of metadata to a namespace in the catalog.-
Methods inherited from class org.apache.iceberg.BaseMetastoreCatalog
dropTableData, fullTableName, loadTable, newCreateTableTransaction, newReplaceTableTransaction, toString
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.apache.iceberg.catalog.Catalog
createTable, createTable, createTable, dropTable, newCreateTableTransaction, newCreateTableTransaction, newCreateTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, newReplaceTableTransaction, tableExists
-
Methods inherited from interface org.apache.iceberg.catalog.SupportsNamespaces
createNamespace, listNamespaces, namespaceExists
-
-
-
-
Constructor Detail
-
HadoopCatalog
public HadoopCatalog(java.lang.String name, org.apache.hadoop.conf.Configuration conf, java.lang.String warehouseLocation)
The constructor of the HadoopCatalog. It uses the passed location as its warehouse directory.- Parameters:
name
- The catalog nameconf
- The Hadoop configurationwarehouseLocation
- The location used as warehouse directory
-
HadoopCatalog
public HadoopCatalog(org.apache.hadoop.conf.Configuration conf, java.lang.String warehouseLocation)
The constructor of the HadoopCatalog. It uses the passed location as its warehouse directory.- Parameters:
conf
- The Hadoop configurationwarehouseLocation
- The location used as warehouse directory
-
HadoopCatalog
public HadoopCatalog(org.apache.hadoop.conf.Configuration conf)
The constructor of the HadoopCatalog. It gets the value offs.defaultFS
property from the passed Hadoop configuration as its default file system, and use the default directoryiceberg/warehouse
as the warehouse directory.- Parameters:
conf
- The Hadoop configuration
-
-
Method Detail
-
name
protected java.lang.String name()
- Specified by:
name
in classBaseMetastoreCatalog
-
listTables
public java.util.List<TableIdentifier> listTables(Namespace namespace)
Description copied from interface:Catalog
Return all the identifiers under this namespace.- Specified by:
listTables
in interfaceCatalog
- Parameters:
namespace
- a namespace- Returns:
- a list of identifiers for tables
-
createTable
public Table createTable(TableIdentifier identifier, Schema schema, PartitionSpec spec, java.lang.String location, java.util.Map<java.lang.String,java.lang.String> properties)
Description copied from interface:Catalog
Create a table.- Specified by:
createTable
in interfaceCatalog
- Overrides:
createTable
in classBaseMetastoreCatalog
- Parameters:
identifier
- a table identifierschema
- a schemaspec
- a partition speclocation
- a location for the table; leave null if unspecifiedproperties
- a string map of table properties- Returns:
- a Table instance
-
isValidIdentifier
protected boolean isValidIdentifier(TableIdentifier identifier)
- Overrides:
isValidIdentifier
in classBaseMetastoreCatalog
-
newTableOps
protected TableOperations newTableOps(TableIdentifier identifier)
- Specified by:
newTableOps
in classBaseMetastoreCatalog
-
defaultWarehouseLocation
protected java.lang.String defaultWarehouseLocation(TableIdentifier tableIdentifier)
- Specified by:
defaultWarehouseLocation
in classBaseMetastoreCatalog
-
dropTable
public boolean dropTable(TableIdentifier identifier, boolean purge)
Description copied from interface:Catalog
Drop a table; optionally delete data and metadata files.If purge is set to true the implementation should delete all data and metadata files.
-
renameTable
public void renameTable(TableIdentifier from, TableIdentifier to)
Description copied from interface:Catalog
Rename a table.- Specified by:
renameTable
in interfaceCatalog
- Parameters:
from
- identifier of the table to renameto
- new table name
-
createNamespace
public void createNamespace(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> meta)
Description copied from interface:SupportsNamespaces
Create a namespace in the catalog.- Specified by:
createNamespace
in interfaceSupportsNamespaces
- Parameters:
namespace
- a multi-part namespacemeta
- a string Map of properties for the given namespace
-
listNamespaces
public java.util.List<Namespace> listNamespaces(Namespace namespace)
Description copied from interface:SupportsNamespaces
List namespaces from the namespace.For example, if table a.b.t exists, use 'SELECT NAMESPACE IN a' this method must return Namepace.of("a","b")
Namespace
.- Specified by:
listNamespaces
in interfaceSupportsNamespaces
- Returns:
- a List of namespace
Namespace
names
-
dropNamespace
public boolean dropNamespace(Namespace namespace)
Description copied from interface:SupportsNamespaces
Drop a namespace. If the namespace exists and was dropped, this will return true.- Specified by:
dropNamespace
in interfaceSupportsNamespaces
- Parameters:
namespace
- a namespace.Namespace
- Returns:
- true if the namespace was dropped, false otherwise.
-
setProperties
public boolean setProperties(Namespace namespace, java.util.Map<java.lang.String,java.lang.String> properties)
Description copied from interface:SupportsNamespaces
Apply a set of metadata to a namespace in the catalog.- Specified by:
setProperties
in interfaceSupportsNamespaces
- Parameters:
namespace
- a namespace.Namespace
properties
- a collection of metadata to apply to the namespace
-
removeProperties
public boolean removeProperties(Namespace namespace, java.util.Set<java.lang.String> properties)
Description copied from interface:SupportsNamespaces
Remove a set of metadata from a namespace in the catalog.- Specified by:
removeProperties
in interfaceSupportsNamespaces
- Parameters:
namespace
- a namespace.Namespace
properties
- a collection of metadata to apply to the namespace
-
loadNamespaceMetadata
public java.util.Map<java.lang.String,java.lang.String> loadNamespaceMetadata(Namespace namespace)
Description copied from interface:SupportsNamespaces
Load metadata properties for a namespace.- Specified by:
loadNamespaceMetadata
in interfaceSupportsNamespaces
- Parameters:
namespace
- a namespace.Namespace
- Returns:
- a string map of properties for the given namespace
-
close
public void close() throws java.io.IOException
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Throws:
java.io.IOException
-
-