org.apache.iceberg.hadoop

Class HadoopFileIO

java.lang.Object

org.apache.iceberg.hadoop.HadoopFileIO

All Implemented Interfaces:

java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable, org.apache.hadoop.conf.Configurable, HadoopConfigurable, FileIO

public class HadoopFileIO
extends java.lang.Object
implements FileIO, HadoopConfigurable

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
HadoopFileIO() Constructor used for dynamic FileIO loading.
HadoopFileIO(org.apache.hadoop.conf.Configuration hadoopConf)
HadoopFileIO(SerializableSupplier<org.apache.hadoop.conf.Configuration> hadoopConf)

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.conf.Configuration	conf()
void	deleteFile(java.lang.String path) Delete the file at the given path.
org.apache.hadoop.conf.Configuration	getConf()
InputFile	newInputFile(java.lang.String path) Get a InputFile instance to read bytes from the file at the given path.
OutputFile	newOutputFile(java.lang.String path) Get a OutputFile instance to write bytes to the file at the given path.
void	serializeConfWith(java.util.function.Function<org.apache.hadoop.conf.Configuration,SerializableSupplier<org.apache.hadoop.conf.Configuration>> confSerializer) Take a function that serializes Hadoop configuration into a supplier.
void	setConf(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.io.FileIO

close, deleteFile, deleteFile, initialize

Constructor Detail

HadoopFileIO

public HadoopFileIO()

Constructor used for dynamic FileIO loading.

Hadoop configuration must be set through setConf(Configuration)

HadoopFileIO

public HadoopFileIO(org.apache.hadoop.conf.Configuration hadoopConf)

HadoopFileIO

public HadoopFileIO(SerializableSupplier<org.apache.hadoop.conf.Configuration> hadoopConf)

Method Detail

conf

public org.apache.hadoop.conf.Configuration conf()

newInputFile

public InputFile newInputFile(java.lang.String path)

Description copied from interface: FileIO

Get a InputFile instance to read bytes from the file at the given path.

Specified by:

newInputFile in interface FileIO

newOutputFile

public OutputFile newOutputFile(java.lang.String path)

Description copied from interface: FileIO

Get a OutputFile instance to write bytes to the file at the given path.

Specified by:

newOutputFile in interface FileIO

deleteFile

public void deleteFile(java.lang.String path)

Description copied from interface: FileIO

Delete the file at the given path.

Specified by:

deleteFile in interface FileIO

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

serializeConfWith

public void serializeConfWith(java.util.function.Function<org.apache.hadoop.conf.Configuration,SerializableSupplier<org.apache.hadoop.conf.Configuration>> confSerializer)

Description copied from interface: HadoopConfigurable

Take a function that serializes Hadoop configuration into a supplier. An implementation is supposed to pass in its current Hadoop configuration into this function, and the result can be safely serialized for future use.

Specified by:

serializeConfWith in interface HadoopConfigurable

Parameters:

confSerializer - A function that takes Hadoop configuration and returns a serializable supplier of it.