Package org.apache.iceberg.spark
Class SparkSessionCatalog<T extends org.apache.spark.sql.connector.catalog.TableCatalog & org.apache.spark.sql.connector.catalog.SupportsNamespaces>
- java.lang.Object
-
- org.apache.iceberg.spark.SparkSessionCatalog<T>
-
- Type Parameters:
T
- CatalogPlugin class to avoid casting to TableCatalog and SupportsNamespaces.
- All Implemented Interfaces:
org.apache.spark.sql.connector.catalog.CatalogExtension
,org.apache.spark.sql.connector.catalog.CatalogPlugin
,org.apache.spark.sql.connector.catalog.StagingTableCatalog
,org.apache.spark.sql.connector.catalog.SupportsNamespaces
,org.apache.spark.sql.connector.catalog.TableCatalog
,ProcedureCatalog
public class SparkSessionCatalog<T extends org.apache.spark.sql.connector.catalog.TableCatalog & org.apache.spark.sql.connector.catalog.SupportsNamespaces> extends java.lang.Object implements org.apache.spark.sql.connector.catalog.CatalogExtension
A Spark catalog that can also load non-Iceberg tables.
-
-
Constructor Summary
Constructors Constructor Description SparkSessionCatalog()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
alterNamespace(java.lang.String[] namespace, org.apache.spark.sql.connector.catalog.NamespaceChange... changes)
org.apache.spark.sql.connector.catalog.Table
alterTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.connector.catalog.TableChange... changes)
protected org.apache.spark.sql.connector.catalog.TableCatalog
buildSparkCatalog(java.lang.String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
Build aSparkCatalog
to be used for Iceberg operations.void
createNamespace(java.lang.String[] namespace, java.util.Map<java.lang.String,java.lang.String> metadata)
org.apache.spark.sql.connector.catalog.Table
createTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, java.util.Map<java.lang.String,java.lang.String> properties)
java.lang.String[]
defaultNamespace()
boolean
dropNamespace(java.lang.String[] namespace)
boolean
dropTable(org.apache.spark.sql.connector.catalog.Identifier ident)
void
initialize(java.lang.String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
void
invalidateTable(org.apache.spark.sql.connector.catalog.Identifier ident)
java.lang.String[][]
listNamespaces()
java.lang.String[][]
listNamespaces(java.lang.String[] namespace)
org.apache.spark.sql.connector.catalog.Identifier[]
listTables(java.lang.String[] namespace)
java.util.Map<java.lang.String,java.lang.String>
loadNamespaceMetadata(java.lang.String[] namespace)
Procedure
loadProcedure(org.apache.spark.sql.connector.catalog.Identifier ident)
Load astored procedure
byidentifier
.org.apache.spark.sql.connector.catalog.Table
loadTable(org.apache.spark.sql.connector.catalog.Identifier ident)
java.lang.String
name()
void
renameTable(org.apache.spark.sql.connector.catalog.Identifier from, org.apache.spark.sql.connector.catalog.Identifier to)
void
setDelegateCatalog(org.apache.spark.sql.connector.catalog.CatalogPlugin sparkSessionCatalog)
org.apache.spark.sql.connector.catalog.StagedTable
stageCreate(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, java.util.Map<java.lang.String,java.lang.String> properties)
org.apache.spark.sql.connector.catalog.StagedTable
stageCreateOrReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, java.util.Map<java.lang.String,java.lang.String> properties)
org.apache.spark.sql.connector.catalog.StagedTable
stageReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, java.util.Map<java.lang.String,java.lang.String> properties)
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
-
-
-
Method Detail
-
buildSparkCatalog
protected org.apache.spark.sql.connector.catalog.TableCatalog buildSparkCatalog(java.lang.String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
Build aSparkCatalog
to be used for Iceberg operations.The default implementation creates a new SparkCatalog with the session catalog's name and options.
- Parameters:
name
- catalog nameoptions
- catalog options- Returns:
- a SparkCatalog to be used for Iceberg tables
-
defaultNamespace
public java.lang.String[] defaultNamespace()
- Specified by:
defaultNamespace
in interfaceorg.apache.spark.sql.connector.catalog.CatalogPlugin
-
listNamespaces
public java.lang.String[][] listNamespaces() throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
listNamespaces
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
listNamespaces
public java.lang.String[][] listNamespaces(java.lang.String[] namespace) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
listNamespaces
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
loadNamespaceMetadata
public java.util.Map<java.lang.String,java.lang.String> loadNamespaceMetadata(java.lang.String[] namespace) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
loadNamespaceMetadata
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
createNamespace
public void createNamespace(java.lang.String[] namespace, java.util.Map<java.lang.String,java.lang.String> metadata) throws org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException
- Specified by:
createNamespace
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
- Throws:
org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException
-
alterNamespace
public void alterNamespace(java.lang.String[] namespace, org.apache.spark.sql.connector.catalog.NamespaceChange... changes) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
alterNamespace
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
dropNamespace
public boolean dropNamespace(java.lang.String[] namespace) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
dropNamespace
in interfaceorg.apache.spark.sql.connector.catalog.SupportsNamespaces
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
listTables
public org.apache.spark.sql.connector.catalog.Identifier[] listTables(java.lang.String[] namespace) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
listTables
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
loadTable
public org.apache.spark.sql.connector.catalog.Table loadTable(org.apache.spark.sql.connector.catalog.Identifier ident) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
- Specified by:
loadTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchTableException
-
invalidateTable
public void invalidateTable(org.apache.spark.sql.connector.catalog.Identifier ident)
- Specified by:
invalidateTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
-
createTable
public org.apache.spark.sql.connector.catalog.Table createTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, java.util.Map<java.lang.String,java.lang.String> properties) throws org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException, org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
createTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
stageCreate
public org.apache.spark.sql.connector.catalog.StagedTable stageCreate(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, java.util.Map<java.lang.String,java.lang.String> properties) throws org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException, org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
stageCreate
in interfaceorg.apache.spark.sql.connector.catalog.StagingTableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
stageReplace
public org.apache.spark.sql.connector.catalog.StagedTable stageReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, java.util.Map<java.lang.String,java.lang.String> properties) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException, org.apache.spark.sql.catalyst.analysis.NoSuchTableException
- Specified by:
stageReplace
in interfaceorg.apache.spark.sql.connector.catalog.StagingTableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
org.apache.spark.sql.catalyst.analysis.NoSuchTableException
-
stageCreateOrReplace
public org.apache.spark.sql.connector.catalog.StagedTable stageCreateOrReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, java.util.Map<java.lang.String,java.lang.String> properties) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
- Specified by:
stageCreateOrReplace
in interfaceorg.apache.spark.sql.connector.catalog.StagingTableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
-
alterTable
public org.apache.spark.sql.connector.catalog.Table alterTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.connector.catalog.TableChange... changes) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
- Specified by:
alterTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchTableException
-
dropTable
public boolean dropTable(org.apache.spark.sql.connector.catalog.Identifier ident)
- Specified by:
dropTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
-
renameTable
public void renameTable(org.apache.spark.sql.connector.catalog.Identifier from, org.apache.spark.sql.connector.catalog.Identifier to) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException, org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
- Specified by:
renameTable
in interfaceorg.apache.spark.sql.connector.catalog.TableCatalog
- Throws:
org.apache.spark.sql.catalyst.analysis.NoSuchTableException
org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
-
initialize
public final void initialize(java.lang.String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
- Specified by:
initialize
in interfaceorg.apache.spark.sql.connector.catalog.CatalogPlugin
-
setDelegateCatalog
public void setDelegateCatalog(org.apache.spark.sql.connector.catalog.CatalogPlugin sparkSessionCatalog)
- Specified by:
setDelegateCatalog
in interfaceorg.apache.spark.sql.connector.catalog.CatalogExtension
-
name
public java.lang.String name()
- Specified by:
name
in interfaceorg.apache.spark.sql.connector.catalog.CatalogPlugin
-
loadProcedure
public Procedure loadProcedure(org.apache.spark.sql.connector.catalog.Identifier ident) throws NoSuchProcedureException
Description copied from interface:ProcedureCatalog
Load astored procedure
byidentifier
.- Specified by:
loadProcedure
in interfaceProcedureCatalog
- Parameters:
ident
- a stored procedure identifier- Returns:
- the stored procedure's metadata
- Throws:
NoSuchProcedureException
- if there is no matching stored procedure
-
-