Package org.apache.iceberg.io

Class BufferedFileAppender<D>

java.lang.Object

org.apache.iceberg.io.BufferedFileAppender<D>

Type Parameters:

D - the row type

All Implemented Interfaces:

Closeable, AutoCloseable, FileAppender<D>

public class BufferedFileAppender<D>
extends Object
implements FileAppender<D>

A FileAppender that buffers the first N rows, then creates a delegate appender via a factory.

The factory receives the buffered rows and is responsible for creating the real appender. Row replay is handled internally. All subsequent add(D) calls delegate directly to the real appender.

If fewer than bufferSize rows are written before close, the factory is called with whatever rows were buffered. If no rows were written, the factory is not called and no file is created on disk. In this case, metrics() returns new Metrics(0L) and length() returns 0L.

Constructor Summary

Constructors

Constructor	Description
BufferedFileAppender(int bufferRowCount, Function<List<D>,FileAppender<D>> appenderFactory)
BufferedFileAppender(int bufferRowCount, Function<List<D>,FileAppender<D>> appenderFactory, UnaryOperator<D> copyFunc)

Method Summary

Modifier and Type	Method	Description
void	add(D datum)
void	close()
long	length()	Returns the length of this file.
Metrics	metrics()	Returns Metrics for this file.
List<Long>	splitOffsets()	Returns a list of recommended split locations, if applicable, null otherwise.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.io.FileAppender

addAll, addAll

Constructor Details

BufferedFileAppender

public BufferedFileAppender(int bufferRowCount,
 Function<List<D>,FileAppender<D>> appenderFactory)

Parameters:

bufferRowCount - number of rows to buffer before creating the delegate appender

appenderFactory - given the buffered rows, creates the delegate appender

BufferedFileAppender

public BufferedFileAppender(int bufferRowCount,
 Function<List<D>,FileAppender<D>> appenderFactory,
 UnaryOperator<D> copyFunc)

Parameters:

bufferRowCount - number of rows to buffer before creating the delegate appender

appenderFactory - given the buffered rows, creates the delegate appender

copyFunc - copies a row before buffering (needed when row objects are reused, e.g. Spark InternalRow)

Method Details

add

public void add(D datum)

Specified by:

add in interface FileAppender<D>

metrics

public Metrics metrics()

Description copied from interface: FileAppender

Returns Metrics for this file. Only valid after the file is closed.

Specified by:

metrics in interface FileAppender<D>

length

public long length()

Description copied from interface: FileAppender

Returns the length of this file.

Specified by:

length in interface FileAppender<D>

splitOffsets

public List<Long> splitOffsets()

Description copied from interface: FileAppender

Returns a list of recommended split locations, if applicable, null otherwise.

When available, this information is used for planning scan tasks whose boundaries are determined by these offsets. The returned list must be sorted in ascending order. Only valid after the file is closed.

Specified by:

splitOffsets in interface FileAppender<D>

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException