Package org.apache.iceberg.hadoop

Interface HadoopConfigurable

All Superinterfaces:
Configurable<org.apache.hadoop.conf.Configuration>
All Known Implementing Classes:
HadoopFileIO, HadoopInputFile, ResolvingFileIO
public interface HadoopConfigurable extends Configurable<org.apache.hadoop.conf.Configuration>
An interface that extends Configurable to offer better serialization support for customizable Iceberg objects such as FileIO.

If an object is serialized and needs to use Hadoop configuration, it is recommended for the object to implement this interface so that a serializable supplier of configuration can be provided instead of an actual Hadoop configuration which is not serializable.

  • Method Summary

    Modifier and Type
    Method
    Description
    default org.apache.hadoop.conf.Configuration
    getConf()
    Return the configuration used by this object.
    void
    serializeConfWith(Function<org.apache.hadoop.conf.Configuration,SerializableSupplier<org.apache.hadoop.conf.Configuration>> confSerializer)
    Take a function that serializes Hadoop configuration into a supplier.
    default void
    setConf(org.apache.hadoop.conf.Configuration conf)
    Set the configuration to be used by this object.

  • Method Details

    • serializeConfWith

      void serializeConfWith(Function<org.apache.hadoop.conf.Configuration,SerializableSupplier<org.apache.hadoop.conf.Configuration>> confSerializer)
      Take a function that serializes Hadoop configuration into a supplier. An implementation is supposed to pass in its current Hadoop configuration into this function, and the result can be safely serialized for future use.
      Parameters:
      confSerializer - A function that takes Hadoop configuration and returns a serializable supplier of it.

    • setConf

      default void setConf(org.apache.hadoop.conf.Configuration conf)
      Set the configuration to be used by this object.
      Specified by:
      setConf in interface Configurable<org.apache.hadoop.conf.Configuration>
      Parameters:
      conf - configuration to be used

    • getConf

      default org.apache.hadoop.conf.Configuration getConf()
      Return the configuration used by this object.
      Returns:
      Configuration