org.apache.iceberg.spark.source

Class SparkPartitionedWriter

java.lang.Object

org.apache.iceberg.io.BaseTaskWriter<T>

org.apache.iceberg.io.PartitionedWriter<org.apache.spark.sql.catalyst.InternalRow>

org.apache.iceberg.spark.source.SparkPartitionedWriter

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, TaskWriter<org.apache.spark.sql.catalyst.InternalRow>

public class SparkPartitionedWriter
extends PartitionedWriter<org.apache.spark.sql.catalyst.InternalRow>

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.iceberg.io.BaseTaskWriter

BaseTaskWriter.RollingFileWriter

Constructor Summary

Constructors

Constructor and Description
SparkPartitionedWriter(PartitionSpec spec, FileFormat format, FileAppenderFactory<org.apache.spark.sql.catalyst.InternalRow> appenderFactory, OutputFileFactory fileFactory, FileIO io, long targetFileSize, Schema schema, org.apache.spark.sql.types.StructType sparkSchema)

Method Summary

Modifier and Type	Method and Description
protected PartitionKey	partition(org.apache.spark.sql.catalyst.InternalRow row) Create a PartitionKey from the values in row.

Methods inherited from class org.apache.iceberg.io.PartitionedWriter

close, write

Methods inherited from class org.apache.iceberg.io.BaseTaskWriter

abort, complete

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkPartitionedWriter

public SparkPartitionedWriter(PartitionSpec spec,
                              FileFormat format,
                              FileAppenderFactory<org.apache.spark.sql.catalyst.InternalRow> appenderFactory,
                              OutputFileFactory fileFactory,
                              FileIO io,
                              long targetFileSize,
                              Schema schema,
                              org.apache.spark.sql.types.StructType sparkSchema)

Method Detail

partition

protected PartitionKey partition(org.apache.spark.sql.catalyst.InternalRow row)

Description copied from class: PartitionedWriter

Create a PartitionKey from the values in row.

Any PartitionKey returned by this method can be reused by the implementation.

Specified by:

partition in class PartitionedWriter<org.apache.spark.sql.catalyst.InternalRow>

Parameters:

row - a data row