Package org.apache.iceberg.spark

Class SparkCatalog

java.lang.Object
org.apache.iceberg.spark.SparkCatalog
All Implemented Interfaces:
HasIcebergCatalog, SupportsReplaceView, org.apache.spark.sql.connector.catalog.CatalogPlugin, org.apache.spark.sql.connector.catalog.FunctionCatalog, org.apache.spark.sql.connector.catalog.ProcedureCatalog, org.apache.spark.sql.connector.catalog.StagingTableCatalog, org.apache.spark.sql.connector.catalog.SupportsNamespaces, org.apache.spark.sql.connector.catalog.TableCatalog, org.apache.spark.sql.connector.catalog.ViewCatalog
public class SparkCatalog extends Object
A Spark TableCatalog implementation that wraps an Iceberg Catalog.

This supports the following catalog configuration options:

  • type - catalog type, "hive" or "hadoop" or "rest". To specify a non-hive or hadoop catalog, use the catalog-impl option.
  • uri - the Hive Metastore URI for Hive catalog or REST URI for REST catalog
  • warehouse - the warehouse path (Hadoop catalog only)
  • catalog-impl - a custom Catalog implementation to use
  • io-impl - a custom FileIO implementation to use
  • metrics-reporter-impl - a custom MetricsReporter implementation to use
  • default-namespace - a namespace to use as the default
  • cache-enabled - whether to enable catalog cache
  • cache.case-sensitive - whether the catalog cache should compare table identifiers in a case sensitive way
  • cache.expiration-interval-ms - interval in millis before expiring tables from catalog cache. Refer to CatalogProperties.CACHE_EXPIRATION_INTERVAL_MS for further details and significant values.
  • table-default.$tablePropertyKey - table property $tablePropertyKey default at catalog level
  • table-override.$tablePropertyKey - table property $tablePropertyKey enforced at catalog level

  • Field Summary

    Fields inherited from interface org.apache.spark.sql.connector.catalog.SupportsNamespaces

    PROP_COMMENT, PROP_LOCATION, PROP_OWNER

    Fields inherited from interface org.apache.spark.sql.connector.catalog.TableCatalog

    OPTION_PREFIX, PROP_COLLATION, PROP_COMMENT, PROP_EXTERNAL, PROP_IS_MANAGED_LOCATION, PROP_LOCATION, PROP_OWNER, PROP_PROVIDER

    Fields inherited from interface org.apache.spark.sql.connector.catalog.ViewCatalog

    PROP_COMMENT, PROP_CREATE_ENGINE_VERSION, PROP_ENGINE_VERSION, PROP_OWNER, RESERVED_PROPERTIES

  • Constructor Summary

    Constructors
    Constructor
    Description
    SparkCatalog()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    alterNamespace(String[] namespace, org.apache.spark.sql.connector.catalog.NamespaceChange... changes)
     
    org.apache.spark.sql.connector.catalog.Table
    alterTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.connector.catalog.TableChange... changes)
     
    org.apache.spark.sql.connector.catalog.View
    alterView(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.connector.catalog.ViewChange... changes)
     
    protected Catalog
    buildIcebergCatalog(String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
    Build an Iceberg Catalog to be used by this Spark catalog adapter.
    protected TableIdentifier
    buildIdentifier(org.apache.spark.sql.connector.catalog.Identifier identifier)
    Build an Iceberg TableIdentifier for the given Spark identifier.
    void
    createNamespace(String[] namespace, Map<String,String> metadata)
     
    org.apache.spark.sql.connector.catalog.Table
    createTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, Map<String,String> properties)
     
    org.apache.spark.sql.connector.catalog.View
    createView(org.apache.spark.sql.connector.catalog.ViewInfo viewInfo)
     
    String[]
    defaultNamespace()
     
    boolean
    dropNamespace(String[] namespace, boolean cascade)
     
    boolean
    dropTable(org.apache.spark.sql.connector.catalog.Identifier ident)
     
    boolean
    dropView(org.apache.spark.sql.connector.catalog.Identifier ident)
     
    Catalog
    icebergCatalog()
    Returns the underlying Catalog backing this Spark Catalog
    final void
    initialize(String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
     
    void
    invalidateTable(org.apache.spark.sql.connector.catalog.Identifier ident)
     
    boolean
    isExistingNamespace(String[] namespace)
     
    boolean
    isFunctionNamespace(String[] namespace)
     
    default org.apache.spark.sql.connector.catalog.Identifier[]
    listFunctions(String[] namespace)
     
    String[][]
    listNamespaces()
     
    String[][]
    listNamespaces(String[] namespace)
     
    org.apache.spark.sql.connector.catalog.Identifier[]
    listTables(String[] namespace)
     
    org.apache.spark.sql.connector.catalog.Identifier[]
    listViews(String... namespace)
     
    default org.apache.spark.sql.connector.catalog.functions.UnboundFunction
    loadFunction(org.apache.spark.sql.connector.catalog.Identifier ident)
     
    Map<String,String>
    loadNamespaceMetadata(String[] namespace)
     
    org.apache.spark.sql.connector.catalog.procedures.UnboundProcedure
    loadProcedure(org.apache.spark.sql.connector.catalog.Identifier ident)
     
    org.apache.spark.sql.connector.catalog.Table
    loadTable(org.apache.spark.sql.connector.catalog.Identifier ident)
     
    org.apache.spark.sql.connector.catalog.Table
    loadTable(org.apache.spark.sql.connector.catalog.Identifier ident, long timestamp)
     
    org.apache.spark.sql.connector.catalog.Table
    loadTable(org.apache.spark.sql.connector.catalog.Identifier ident, String version)
     
    org.apache.spark.sql.connector.catalog.View
    loadView(org.apache.spark.sql.connector.catalog.Identifier ident)
     
    String
    name()
     
    boolean
    purgeTable(org.apache.spark.sql.connector.catalog.Identifier ident)
     
    void
    renameTable(org.apache.spark.sql.connector.catalog.Identifier from, org.apache.spark.sql.connector.catalog.Identifier to)
     
    void
    renameView(org.apache.spark.sql.connector.catalog.Identifier fromIdentifier, org.apache.spark.sql.connector.catalog.Identifier toIdentifier)
     
    org.apache.spark.sql.connector.catalog.View
    replaceView(org.apache.spark.sql.connector.catalog.Identifier ident, String sql, String currentCatalog, String[] currentNamespace, org.apache.spark.sql.types.StructType schema, String[] queryColumnNames, String[] columnAliases, String[] columnComments, Map<String,String> properties)
    Replace a view in the catalog
    org.apache.spark.sql.connector.catalog.StagedTable
    stageCreate(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, Map<String,String> properties)
     
    org.apache.spark.sql.connector.catalog.StagedTable
    stageCreateOrReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, Map<String,String> properties)
     
    org.apache.spark.sql.connector.catalog.StagedTable
    stageReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, Map<String,String> properties)
     
    boolean
    useNullableQuerySchema()
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.sql.connector.catalog.FunctionCatalog

    functionExists

    Methods inherited from interface org.apache.spark.sql.connector.catalog.StagingTableCatalog

    stageCreate, stageCreateOrReplace, stageReplace, supportedCustomMetrics

    Methods inherited from interface org.apache.spark.sql.connector.catalog.SupportsNamespaces

    namespaceExists

    Methods inherited from interface org.apache.spark.sql.connector.catalog.TableCatalog

    capabilities, createTable, loadTable, tableExists

    Methods inherited from interface org.apache.spark.sql.connector.catalog.ViewCatalog

    invalidateView, replaceView, viewExists

  • Constructor Details

    • SparkCatalog

      public SparkCatalog()

  • Method Details

    • buildIcebergCatalog

      protected Catalog buildIcebergCatalog(String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
      Build an Iceberg Catalog to be used by this Spark catalog adapter.
      Parameters:
      name - Spark's catalog name
      options - Spark's catalog options
      Returns:
      an Iceberg catalog

    • buildIdentifier

      protected TableIdentifier buildIdentifier(org.apache.spark.sql.connector.catalog.Identifier identifier)
      Build an Iceberg TableIdentifier for the given Spark identifier.
      Parameters:
      identifier - Spark's identifier
      Returns:
      an Iceberg identifier

    • loadTable

      public org.apache.spark.sql.connector.catalog.Table loadTable(org.apache.spark.sql.connector.catalog.Identifier ident) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchTableException

    • loadTable

      public org.apache.spark.sql.connector.catalog.Table loadTable(org.apache.spark.sql.connector.catalog.Identifier ident, String version) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchTableException

    • loadTable

      public org.apache.spark.sql.connector.catalog.Table loadTable(org.apache.spark.sql.connector.catalog.Identifier ident, long timestamp) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchTableException

    • createTable

      public org.apache.spark.sql.connector.catalog.Table createTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, Map<String,String> properties) throws org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
      Throws:
      org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException

    • stageCreate

      public org.apache.spark.sql.connector.catalog.StagedTable stageCreate(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, Map<String,String> properties) throws org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
      Throws:
      org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException

    • stageReplace

      public org.apache.spark.sql.connector.catalog.StagedTable stageReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, Map<String,String> properties) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchTableException

    • stageCreateOrReplace

      public org.apache.spark.sql.connector.catalog.StagedTable stageCreateOrReplace(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] transforms, Map<String,String> properties)

    • alterTable

      public org.apache.spark.sql.connector.catalog.Table alterTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.connector.catalog.TableChange... changes) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchTableException

    • dropTable

      public boolean dropTable(org.apache.spark.sql.connector.catalog.Identifier ident)

    • purgeTable

      public boolean purgeTable(org.apache.spark.sql.connector.catalog.Identifier ident)

    • renameTable

      public void renameTable(org.apache.spark.sql.connector.catalog.Identifier from, org.apache.spark.sql.connector.catalog.Identifier to) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException, org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchTableException
      org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException

    • invalidateTable

      public void invalidateTable(org.apache.spark.sql.connector.catalog.Identifier ident)

    • listTables

      public org.apache.spark.sql.connector.catalog.Identifier[] listTables(String[] namespace)

    • defaultNamespace

      public String[] defaultNamespace()

    • listNamespaces

      public String[][] listNamespaces()

    • listNamespaces

      public String[][] listNamespaces(String[] namespace) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

    • loadNamespaceMetadata

      public Map<String,String> loadNamespaceMetadata(String[] namespace) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

    • createNamespace

      public void createNamespace(String[] namespace, Map<String,String> metadata) throws org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException

    • alterNamespace

      public void alterNamespace(String[] namespace, org.apache.spark.sql.connector.catalog.NamespaceChange... changes) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

    • dropNamespace

      public boolean dropNamespace(String[] namespace, boolean cascade) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

    • listViews

      public org.apache.spark.sql.connector.catalog.Identifier[] listViews(String... namespace)

    • loadView

      public org.apache.spark.sql.connector.catalog.View loadView(org.apache.spark.sql.connector.catalog.Identifier ident) throws org.apache.spark.sql.catalyst.analysis.NoSuchViewException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchViewException

    • createView

      public org.apache.spark.sql.connector.catalog.View createView(org.apache.spark.sql.connector.catalog.ViewInfo viewInfo) throws org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException, org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
      Throws:
      org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException
      org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

    • replaceView

      public org.apache.spark.sql.connector.catalog.View replaceView(org.apache.spark.sql.connector.catalog.Identifier ident, String sql, String currentCatalog, String[] currentNamespace, org.apache.spark.sql.types.StructType schema, String[] queryColumnNames, String[] columnAliases, String[] columnComments, Map<String,String> properties) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException, org.apache.spark.sql.catalyst.analysis.NoSuchViewException
      Description copied from interface: SupportsReplaceView
      Replace a view in the catalog
      Parameters:
      ident - a view identifier
      sql - the SQL text that defines the view
      currentCatalog - the current catalog
      currentNamespace - the current namespace
      schema - the view query output schema
      queryColumnNames - the query column names
      columnAliases - the column aliases
      columnComments - the column comments
      properties - the view properties
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException - If the identifier namespace does not exist (optional)
      org.apache.spark.sql.catalyst.analysis.NoSuchViewException - If the view doesn't exist or is a table

    • alterView

      public org.apache.spark.sql.connector.catalog.View alterView(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.connector.catalog.ViewChange... changes) throws org.apache.spark.sql.catalyst.analysis.NoSuchViewException, IllegalArgumentException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchViewException
      IllegalArgumentException

    • dropView

      public boolean dropView(org.apache.spark.sql.connector.catalog.Identifier ident)

    • renameView

      public void renameView(org.apache.spark.sql.connector.catalog.Identifier fromIdentifier, org.apache.spark.sql.connector.catalog.Identifier toIdentifier) throws org.apache.spark.sql.catalyst.analysis.NoSuchViewException, org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchViewException
      org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException

    • initialize

      public final void initialize(String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
      Specified by:
      initialize in interface org.apache.spark.sql.connector.catalog.CatalogPlugin

    • name

      public String name()

    • icebergCatalog

      public Catalog icebergCatalog()
      Description copied from interface: HasIcebergCatalog
      Returns the underlying Catalog backing this Spark Catalog

    • loadProcedure

      public org.apache.spark.sql.connector.catalog.procedures.UnboundProcedure loadProcedure(org.apache.spark.sql.connector.catalog.Identifier ident)
      Specified by:
      loadProcedure in interface org.apache.spark.sql.connector.catalog.ProcedureCatalog

    • isFunctionNamespace

      public boolean isFunctionNamespace(String[] namespace)

    • isExistingNamespace

      public boolean isExistingNamespace(String[] namespace)

    • useNullableQuerySchema

      public boolean useNullableQuerySchema()
      Specified by:
      useNullableQuerySchema in interface org.apache.spark.sql.connector.catalog.TableCatalog

    • listFunctions

      default org.apache.spark.sql.connector.catalog.Identifier[] listFunctions(String[] namespace) throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
      Specified by:
      listFunctions in interface org.apache.spark.sql.connector.catalog.FunctionCatalog
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

    • loadFunction

      default org.apache.spark.sql.connector.catalog.functions.UnboundFunction loadFunction(org.apache.spark.sql.connector.catalog.Identifier ident) throws org.apache.spark.sql.catalyst.analysis.NoSuchFunctionException
      Specified by:
      loadFunction in interface org.apache.spark.sql.connector.catalog.FunctionCatalog
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchFunctionException