Package org.apache.iceberg.mr.hive

Class HiveIcebergStorageHandler

  • All Implemented Interfaces:
    org.apache.hadoop.conf.Configurable, org.apache.hadoop.hive.ql.metadata.HiveStorageHandler, org.apache.hadoop.hive.ql.metadata.HiveStoragePredicateHandler
    public class HiveIcebergStorageHandler
extends java.lang.Object
implements org.apache.hadoop.hive.ql.metadata.HiveStoragePredicateHandler, org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

    • Nested Class Summary

      • Nested classes/interfaces inherited from interface org.apache.hadoop.hive.ql.metadata.HiveStoragePredicateHandler

        org.apache.hadoop.hive.ql.metadata.HiveStoragePredicateHandler.DecomposedPredicate

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      static java.lang.String catalogName​(org.apache.hadoop.conf.Configuration config, java.lang.String name)
      Returns the catalog name serialized to the configuration.
      void configureInputJobCredentials​(org.apache.hadoop.hive.ql.plan.TableDesc tableDesc, java.util.Map<java.lang.String,​java.lang.String> secrets)  
      void configureInputJobProperties​(org.apache.hadoop.hive.ql.plan.TableDesc tableDesc, java.util.Map<java.lang.String,​java.lang.String> map)  
      void configureJobConf​(org.apache.hadoop.hive.ql.plan.TableDesc tableDesc, org.apache.hadoop.mapred.JobConf jobConf)  
      void configureOutputJobProperties​(org.apache.hadoop.hive.ql.plan.TableDesc tableDesc, java.util.Map<java.lang.String,​java.lang.String> map)  
      void configureTableJobProperties​(org.apache.hadoop.hive.ql.plan.TableDesc tableDesc, java.util.Map<java.lang.String,​java.lang.String> map)  
      org.apache.hadoop.hive.ql.metadata.HiveStoragePredicateHandler.DecomposedPredicate decomposePredicate​(org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.hive.serde2.Deserializer deserializer, org.apache.hadoop.hive.ql.plan.ExprNodeDesc exprNodeDesc)  
      org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProvider getAuthorizationProvider()  
      org.apache.hadoop.conf.Configuration getConf()  
      java.lang.Class<? extends org.apache.hadoop.mapred.InputFormat> getInputFormatClass()  
      org.apache.hadoop.hive.metastore.HiveMetaHook getMetaHook()  
      java.lang.Class<? extends org.apache.hadoop.mapred.OutputFormat> getOutputFormatClass()  
      java.lang.Class<? extends org.apache.hadoop.hive.serde2.AbstractSerDe> getSerDeClass()  
      static java.util.Collection<java.lang.String> outputTables​(org.apache.hadoop.conf.Configuration config)
      Returns the names of the output tables stored in the configuration.
      static Schema schema​(org.apache.hadoop.conf.Configuration config)
      Returns the Table Schema serialized to the configuration.
      void setConf​(org.apache.hadoop.conf.Configuration conf)  
      static Table table​(org.apache.hadoop.conf.Configuration config, java.lang.String name)
      Returns the Table serialized to the configuration based on the table name.
      java.lang.String toString()  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Constructor Detail

      • HiveIcebergStorageHandler

        public HiveIcebergStorageHandler()

    • Method Detail

      • getInputFormatClass

        public java.lang.Class<? extends org.apache.hadoop.mapred.InputFormat> getInputFormatClass()
        Specified by:
        getInputFormatClass in interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

      • getOutputFormatClass

        public java.lang.Class<? extends org.apache.hadoop.mapred.OutputFormat> getOutputFormatClass()
        Specified by:
        getOutputFormatClass in interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

      • getSerDeClass

        public java.lang.Class<? extends org.apache.hadoop.hive.serde2.AbstractSerDe> getSerDeClass()
        Specified by:
        getSerDeClass in interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

      • getMetaHook

        public org.apache.hadoop.hive.metastore.HiveMetaHook getMetaHook()
        Specified by:
        getMetaHook in interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

      • getAuthorizationProvider

        public org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProvider getAuthorizationProvider()
        Specified by:
        getAuthorizationProvider in interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

      • configureInputJobProperties

        public void configureInputJobProperties​(org.apache.hadoop.hive.ql.plan.TableDesc tableDesc,
                                        java.util.Map<java.lang.String,​java.lang.String> map)
        Specified by:
        configureInputJobProperties in interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

      • configureOutputJobProperties

        public void configureOutputJobProperties​(org.apache.hadoop.hive.ql.plan.TableDesc tableDesc,
                                         java.util.Map<java.lang.String,​java.lang.String> map)
        Specified by:
        configureOutputJobProperties in interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

      • configureTableJobProperties

        public void configureTableJobProperties​(org.apache.hadoop.hive.ql.plan.TableDesc tableDesc,
                                        java.util.Map<java.lang.String,​java.lang.String> map)
        Specified by:
        configureTableJobProperties in interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

      • configureInputJobCredentials

        public void configureInputJobCredentials​(org.apache.hadoop.hive.ql.plan.TableDesc tableDesc,
                                         java.util.Map<java.lang.String,​java.lang.String> secrets)

      • configureJobConf

        public void configureJobConf​(org.apache.hadoop.hive.ql.plan.TableDesc tableDesc,
                             org.apache.hadoop.mapred.JobConf jobConf)
        Specified by:
        configureJobConf in interface org.apache.hadoop.hive.ql.metadata.HiveStorageHandler

      • getConf

        public org.apache.hadoop.conf.Configuration getConf()
        Specified by:
        getConf in interface org.apache.hadoop.conf.Configurable

      • setConf

        public void setConf​(org.apache.hadoop.conf.Configuration conf)
        Specified by:
        setConf in interface org.apache.hadoop.conf.Configurable

      • toString

        public java.lang.String toString()
        Overrides:
        toString in class java.lang.Object

      • decomposePredicate

        public org.apache.hadoop.hive.ql.metadata.HiveStoragePredicateHandler.DecomposedPredicate decomposePredicate​(org.apache.hadoop.mapred.JobConf jobConf,
                                                                                                             org.apache.hadoop.hive.serde2.Deserializer deserializer,
                                                                                                             org.apache.hadoop.hive.ql.plan.ExprNodeDesc exprNodeDesc)
        Specified by:
        decomposePredicate in interface org.apache.hadoop.hive.ql.metadata.HiveStoragePredicateHandler
        Parameters:
        jobConf - Job configuration for InputFormat to access
        deserializer - Deserializer
        exprNodeDesc - Filter expression extracted by Hive
        Returns:
        Entire filter to take advantage of Hive's pruning as well as Iceberg's pruning.

      • table

        public static Table table​(org.apache.hadoop.conf.Configuration config,
                          java.lang.String name)
        Returns the Table serialized to the configuration based on the table name.
        Parameters:
        config - The configuration used to get the data from
        name - The name of the table we need as returned by TableDesc.getTableName()
        Returns:
        The Table

      • outputTables

        public static java.util.Collection<java.lang.String> outputTables​(org.apache.hadoop.conf.Configuration config)
        Returns the names of the output tables stored in the configuration.
        Parameters:
        config - The configuration used to get the data from
        Returns:
        The collection of the table names as returned by TableDesc.getTableName()

      • catalogName

        public static java.lang.String catalogName​(org.apache.hadoop.conf.Configuration config,
                                           java.lang.String name)
        Returns the catalog name serialized to the configuration.
        Parameters:
        config - The configuration used to get the data from
        name - The name of the table we neeed as returned by TableDesc.getTableName()
        Returns:
        catalog name

      • schema

        public static Schema schema​(org.apache.hadoop.conf.Configuration config)
        Returns the Table Schema serialized to the configuration.
        Parameters:
        config - The configuration used to get the data from
        Returns:
        The Table Schema object