org.apache.iceberg.spark.source

Class IcebergSource

java.lang.Object

org.apache.iceberg.spark.source.IcebergSource

All Implemented Interfaces:

org.apache.spark.sql.connector.catalog.SupportsCatalogOptions, org.apache.spark.sql.connector.catalog.TableProvider, org.apache.spark.sql.sources.DataSourceRegister

public class IcebergSource
extends java.lang.Object
implements org.apache.spark.sql.sources.DataSourceRegister, org.apache.spark.sql.connector.catalog.SupportsCatalogOptions

The IcebergSource loads/writes tables with format "iceberg". It can load paths and tables.

How paths/tables are loaded when using spark.read().format("iceberg").load(table)

table = "file:///path/to/table" -> loads a HadoopTable at given path table = "tablename" -> loads currentCatalog.currentNamespace.tablename table = "catalog.tablename" -> load "tablename" from the specified catalog. table = "namespace.tablename" -> load "namespace.tablename" from current catalog table = "catalog.namespace.tablename" -> "namespace.tablename" from the specified catalog. table = "namespace1.namespace2.tablename" -> load "namespace1.namespace2.tablename" from current catalog

The above list is in order of priority. For example: a matching catalog will take priority over any namespace resolution.

Constructor Summary

Constructors

Constructor and Description
IcebergSource()

Method Summary

Modifier and Type	Method and Description
java.lang.String	extractCatalog(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
org.apache.spark.sql.connector.catalog.Identifier	extractIdentifier(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
java.util.Optional<java.lang.String>	extractTimeTravelTimestamp(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
java.util.Optional<java.lang.String>	extractTimeTravelVersion(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
org.apache.spark.sql.connector.catalog.Table	getTable(org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitioning, java.util.Map<java.lang.String,java.lang.String> options)
org.apache.spark.sql.connector.expressions.Transform[]	inferPartitioning(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
org.apache.spark.sql.types.StructType	inferSchema(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
java.lang.String	shortName()
boolean	supportsExternalMetadata()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IcebergSource

public IcebergSource()

Method Detail

shortName

public java.lang.String shortName()

Specified by:

shortName in interface org.apache.spark.sql.sources.DataSourceRegister

inferSchema

public org.apache.spark.sql.types.StructType inferSchema(org.apache.spark.sql.util.CaseInsensitiveStringMap options)

Specified by:

inferSchema in interface org.apache.spark.sql.connector.catalog.TableProvider

inferPartitioning

public org.apache.spark.sql.connector.expressions.Transform[] inferPartitioning(org.apache.spark.sql.util.CaseInsensitiveStringMap options)

Specified by:

inferPartitioning in interface org.apache.spark.sql.connector.catalog.TableProvider

supportsExternalMetadata

public boolean supportsExternalMetadata()

Specified by:

supportsExternalMetadata in interface org.apache.spark.sql.connector.catalog.TableProvider

getTable

public org.apache.spark.sql.connector.catalog.Table getTable(org.apache.spark.sql.types.StructType schema,
                                                             org.apache.spark.sql.connector.expressions.Transform[] partitioning,
                                                             java.util.Map<java.lang.String,java.lang.String> options)

Specified by:

getTable in interface org.apache.spark.sql.connector.catalog.TableProvider

extractIdentifier

public org.apache.spark.sql.connector.catalog.Identifier extractIdentifier(org.apache.spark.sql.util.CaseInsensitiveStringMap options)

Specified by:

extractIdentifier in interface org.apache.spark.sql.connector.catalog.SupportsCatalogOptions

extractCatalog

public java.lang.String extractCatalog(org.apache.spark.sql.util.CaseInsensitiveStringMap options)

Specified by:

extractCatalog in interface org.apache.spark.sql.connector.catalog.SupportsCatalogOptions

extractTimeTravelVersion

public java.util.Optional<java.lang.String> extractTimeTravelVersion(org.apache.spark.sql.util.CaseInsensitiveStringMap options)

Specified by:

extractTimeTravelVersion in interface org.apache.spark.sql.connector.catalog.SupportsCatalogOptions

extractTimeTravelTimestamp

public java.util.Optional<java.lang.String> extractTimeTravelTimestamp(org.apache.spark.sql.util.CaseInsensitiveStringMap options)

Specified by:

extractTimeTravelTimestamp in interface org.apache.spark.sql.connector.catalog.SupportsCatalogOptions