Package org.apache.iceberg.spark
Class SparkCatalog
- java.lang.Object
-
- org.apache.iceberg.spark.SparkCatalog
-
- All Implemented Interfaces:
org.apache.spark.sql.connector.catalog.CatalogPlugin
,org.apache.spark.sql.connector.catalog.StagingTableCatalog
,org.apache.spark.sql.connector.catalog.SupportsNamespaces
,org.apache.spark.sql.connector.catalog.TableCatalog
public class SparkCatalog extends java.lang.Object implements org.apache.spark.sql.connector.catalog.StagingTableCatalog, org.apache.spark.sql.connector.catalog.SupportsNamespaces
A Spark TableCatalog implementation that wraps an IcebergCatalog
.This supports the following catalog configuration options:
- type - catalog type, "hive" or "hadoop"
- uri - the Hive Metastore URI (Hive catalog only)
- warehouse - the warehouse path (Hadoop catalog only)
- default-namespace - a namespace to use as the default
To use a custom catalog that is not a Hive or Hadoop catalog, extend this class and override
buildIcebergCatalog(String, CaseInsensitiveStringMap)
.
-
-
Constructor Summary
Constructors Constructor Description SparkCatalog()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
alterNamespace(java.lang.String[] namespace, org.apache.spark.sql.connector.catalog.NamespaceChange... changes)
SparkTable
alterTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.connector.catalog.TableChange... changes)
protected Catalog
buildIcebergCatalog(java.lang.String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
Build an IcebergCatalog
to be used by this Spark catalog adapter.protected TableIdentifier
buildIdentifier(org.apache.spark.sql.connector.catalog.Identifier identifier)
Build an IcebergTableIdentifier
for the given Spark identifier.void
createNamespace(java.lang.String[] namespace, java.util.Map<java.lang.String,java.lang.String> metadata)
SparkTable
createTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, java.util.Map<java.lang.String,java.lang.String> properties)
java.lang.String[]
defaultNamespace()
boolean
dropNamespace(java.lang.String[] namespace)
boolean
dropTable(org.apache.spark.sql.connector.catalog.Identifier ident)
void
initialize(java.lang.String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
void
invalidateTable(org.apache.spark.sql.connector.catalog.Identifier ident)
java.lang.String[][]
listNamespaces()
java.lang.String[][]
listNamespaces(java.lang.String[] namespace)
org.apache.spark.sql.connector.catalog.Identifier[]
listTables(java.lang.String[] namespace)
java.util.Map<java.lang.String,java.lang.String>
loadNamespaceMetadata(java.lang.String[] namespace)
SparkTable
loadTable(org.apache.spark.sql.connector.catalog.Identifier ident)
java.lang.String
name()
void
renameTable(org.apache.spark.sql.connector.catalog.Identifier from, org.apache.spark.sql.connector.catalog.Identifier to)
org.apache.spark.sql.connector.catalog.StagedTable
stageCreate(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, java.util.Map<java.lang.String,java.lang.String> properties)
org.apache.spark.sql.connector.catalog.StagedTable
stageCreateOrReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, java.util.Map<java.lang.String,java.lang.String> properties)
org.apache.spark.sql.connector.catalog.StagedTable
stageReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, java.util.Map<java.lang.String,java.lang.String> properties)
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
-
-
-
Method Detail
-
buildIcebergCatalog
protected Catalog buildIcebergCatalog(java.lang.String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
Build an IcebergCatalog
to be used by this Spark catalog adapter.- Parameters:
name
- Spark's catalog nameoptions
- Spark's catalog options- Returns:
- an Iceberg catalog
-
buildIdentifier
protected TableIdentifier buildIdentifier(org.apache.spark.sql.connector.catalog.Identifier identifier)
Build an IcebergTableIdentifier
for the given Spark identifier.- Parameters:
identifier
- Spark's identifier- Returns:
- an Iceberg identifier
-
loadTable
public SparkTable loadTable(org.apache.spark.sql.connector.catalog.Identifier ident) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
- Specified by:
loadTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchTableException
-
createTable
public SparkTable createTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, java.util.Map<java.lang.String,java.lang.String> properties) throws org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
- Specified by:
createTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
-
stageCreate
public org.apache.spark.sql.connector.catalog.StagedTable stageCreate(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, java.util.Map<java.lang.String,java.lang.String> properties) throws org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
- Specified by:
stageCreate
in interfaceorg.apache.spark.sql.connector.catalog.StagingTableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
-
stageReplace
public org.apache.spark.sql.connector.catalog.StagedTable stageReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, java.util.Map<java.lang.String,java.lang.String> properties) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
- Specified by:
stageReplace
in interfaceorg.apache.spark.sql.connector.catalog.StagingTableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchTableException
-
stageCreateOrReplace
public org.apache.spark.sql.connector.catalog.StagedTable stageCreateOrReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, java.util.Map<java.lang.String,java.lang.String> properties)
- Specified by:
stageCreateOrReplace
in interfaceorg.apache.spark.sql.connector.catalog.StagingTableCatalog
-
alterTable
public SparkTable alterTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.connector.catalog.TableChange... changes) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
- Specified by:
alterTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchTableException
-
dropTable
public boolean dropTable(org.apache.spark.sql.connector.catalog.Identifier ident)
- Specified by:
dropTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
-
renameTable
public void renameTable(org.apache.spark.sql.connector.catalog.Identifier from, org.apache.spark.sql.connector.catalog.Identifier to) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException, org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
- Specified by:
renameTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchTableException
org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
-
invalidateTable
public void invalidateTable(org.apache.spark.sql.connector.catalog.Identifier ident)
- Specified by:
invalidateTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
-
listTables
public org.apache.spark.sql.connector.catalog.Identifier[] listTables(java.lang.String[] namespace)
- Specified by:
listTables
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
-
defaultNamespace
public java.lang.String[] defaultNamespace()
- Specified by:
defaultNamespace
in interfaceorg.apache.spark.sql.connector.catalog.CatalogPlugin
-
listNamespaces
public java.lang.String[][] listNamespaces()
- Specified by:
listNamespaces
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
-
listNamespaces
public java.lang.String[][] listNamespaces(java.lang.String[] namespace) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
listNamespaces
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
loadNamespaceMetadata
public java.util.Map<java.lang.String,java.lang.String> loadNamespaceMetadata(java.lang.String[] namespace) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
loadNamespaceMetadata
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
createNamespace
public void createNamespace(java.lang.String[] namespace, java.util.Map<java.lang.String,java.lang.String> metadata) throws org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException
- Specified by:
createNamespace
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
- Throws:
org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException
-
alterNamespace
public void alterNamespace(java.lang.String[] namespace, org.apache.spark.sql.connector.catalog.NamespaceChange... changes) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
alterNamespace
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
dropNamespace
public boolean dropNamespace(java.lang.String[] namespace) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
dropNamespace
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
initialize
public final void initialize(java.lang.String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
- Specified by:
initialize
in interfaceorg.apache.spark.sql.connector.catalog.CatalogPlugin
-
name
public java.lang.String name()
- Specified by:
name
in interfaceorg.apache.spark.sql.connector.catalog.CatalogPlugin
-
-