Class HadoopTables

  • All Implemented Interfaces:
    org.apache.hadoop.conf.Configurable, Tables

    public class HadoopTables
    extends java.lang.Object
    implements Tables, org.apache.hadoop.conf.Configurable
    Implementation of Iceberg tables that uses the Hadoop FileSystem to store metadata and manifests.
    • Field Summary

      Fields 
      Modifier and Type Field Description
      static java.lang.String LOCK_PROPERTY_PREFIX  
    • Constructor Summary

      Constructors 
      Constructor Description
      HadoopTables()  
      HadoopTables​(org.apache.hadoop.conf.Configuration conf)  
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      Catalog.TableBuilder buildTable​(java.lang.String location, Schema schema)  
      Table create​(Schema schema, PartitionSpec spec, SortOrder order, java.util.Map<java.lang.String,​java.lang.String> properties, java.lang.String location)
      Create a table using the FileSystem implementation resolve from location.
      boolean dropTable​(java.lang.String location)
      Drop a table and delete all data and metadata files.
      boolean dropTable​(java.lang.String location, boolean purge)
      Drop a table; optionally delete data and metadata files.
      boolean exists​(java.lang.String location)  
      org.apache.hadoop.conf.Configuration getConf()  
      Table load​(java.lang.String location)
      Loads the table location from a FileSystem path location.
      Transaction newCreateTableTransaction​(java.lang.String location, Schema schema, PartitionSpec spec, java.util.Map<java.lang.String,​java.lang.String> properties)
      Start a transaction to create a table.
      Transaction newReplaceTableTransaction​(java.lang.String location, Schema schema, PartitionSpec spec, java.util.Map<java.lang.String,​java.lang.String> properties, boolean orCreate)
      Start a transaction to replace a table.
      void setConf​(org.apache.hadoop.conf.Configuration conf)  
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • LOCK_PROPERTY_PREFIX

        public static final java.lang.String LOCK_PROPERTY_PREFIX
        See Also:
        Constant Field Values
    • Constructor Detail

      • HadoopTables

        public HadoopTables()
      • HadoopTables

        public HadoopTables​(org.apache.hadoop.conf.Configuration conf)
    • Method Detail

      • load

        public Table load​(java.lang.String location)
        Loads the table location from a FileSystem path location.
        Specified by:
        load in interface Tables
        Parameters:
        location - a path URI (e.g. hdfs:///warehouse/my_table/)
        Returns:
        table implementation
      • exists

        public boolean exists​(java.lang.String location)
        Specified by:
        exists in interface Tables
      • create

        public Table create​(Schema schema,
                            PartitionSpec spec,
                            SortOrder order,
                            java.util.Map<java.lang.String,​java.lang.String> properties,
                            java.lang.String location)
        Create a table using the FileSystem implementation resolve from location.
        Specified by:
        create in interface Tables
        Parameters:
        schema - iceberg schema used to create the table
        spec - partitioning spec, if null the table will be unpartitioned
        properties - a string map of table properties, initialized to empty if null
        location - a path URI (e.g. hdfs:///warehouse/my_table)
        Returns:
        newly created table implementation
      • dropTable

        public boolean dropTable​(java.lang.String location)
        Drop a table and delete all data and metadata files.
        Parameters:
        location - a path URI (e.g. hdfs:///warehouse/my_table)
        Returns:
        true if the table was dropped, false if it did not exist
      • dropTable

        public boolean dropTable​(java.lang.String location,
                                 boolean purge)
        Drop a table; optionally delete data and metadata files.

        If purge is set to true the implementation should delete all data and metadata files.

        Parameters:
        location - a path URI (e.g. hdfs:///warehouse/my_table)
        purge - if true, delete all data and metadata files in the table
        Returns:
        true if the table was dropped, false if it did not exist
      • newCreateTableTransaction

        public Transaction newCreateTableTransaction​(java.lang.String location,
                                                     Schema schema,
                                                     PartitionSpec spec,
                                                     java.util.Map<java.lang.String,​java.lang.String> properties)
        Start a transaction to create a table.
        Parameters:
        location - a location for the table
        schema - a schema
        spec - a partition spec
        properties - a string map of table properties
        Returns:
        a Transaction to create the table
        Throws:
        AlreadyExistsException - if the table already exists
      • newReplaceTableTransaction

        public Transaction newReplaceTableTransaction​(java.lang.String location,
                                                      Schema schema,
                                                      PartitionSpec spec,
                                                      java.util.Map<java.lang.String,​java.lang.String> properties,
                                                      boolean orCreate)
        Start a transaction to replace a table.
        Parameters:
        location - a location for the table
        schema - a schema
        spec - a partition spec
        properties - a string map of table properties
        orCreate - whether to create the table if not exists
        Returns:
        a Transaction to replace the table
        Throws:
        NoSuchTableException - if the table doesn't exist and orCreate is false
      • setConf

        public void setConf​(org.apache.hadoop.conf.Configuration conf)
        Specified by:
        setConf in interface org.apache.hadoop.conf.Configurable
      • getConf

        public org.apache.hadoop.conf.Configuration getConf()
        Specified by:
        getConf in interface org.apache.hadoop.conf.Configurable