Package org.apache.iceberg.spark.source

Class SparkPartitionedFanoutWriter

java.lang.Object

org.apache.iceberg.io.BaseTaskWriter<org.apache.spark.sql.catalyst.InternalRow>

org.apache.iceberg.io.PartitionedFanoutWriter<org.apache.spark.sql.catalyst.InternalRow>

org.apache.iceberg.spark.source.SparkPartitionedFanoutWriter

All Implemented Interfaces:

Closeable, AutoCloseable, TaskWriter<org.apache.spark.sql.catalyst.InternalRow>

public class SparkPartitionedFanoutWriter
extends PartitionedFanoutWriter<org.apache.spark.sql.catalyst.InternalRow>

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.iceberg.io.BaseTaskWriter

BaseTaskWriter.BaseEqualityDeltaWriter, BaseTaskWriter.RollingEqDeleteWriter, BaseTaskWriter.RollingFileWriter

Constructor Summary

Constructors

Constructor	Description
SparkPartitionedFanoutWriter(PartitionSpec spec, FileFormat format, FileAppenderFactory<org.apache.spark.sql.catalyst.InternalRow> appenderFactory, OutputFileFactory fileFactory, FileIO io, long targetFileSize, Schema schema, org.apache.spark.sql.types.StructType sparkSchema)

Method Summary

Modifier and Type	Method	Description
protected PartitionKey	partition(org.apache.spark.sql.catalyst.InternalRow row)	Create a PartitionKey from the values in row.

Methods inherited from class org.apache.iceberg.io.PartitionedFanoutWriter

close, write

Methods inherited from class org.apache.iceberg.io.BaseTaskWriter

abort, complete, setFailure, spec

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.io.TaskWriter

dataFiles

Constructor Details

SparkPartitionedFanoutWriter

public SparkPartitionedFanoutWriter(PartitionSpec spec,
 FileFormat format,
 FileAppenderFactory<org.apache.spark.sql.catalyst.InternalRow> appenderFactory,
 OutputFileFactory fileFactory,
 FileIO io,
 long targetFileSize,
 Schema schema,
 org.apache.spark.sql.types.StructType sparkSchema)

Method Details

partition

protected PartitionKey partition(org.apache.spark.sql.catalyst.InternalRow row)

Description copied from class: PartitionedFanoutWriter

Create a PartitionKey from the values in row.

Any PartitionKey returned by this method can be reused by the implementation.

Specified by:

partition in class PartitionedFanoutWriter<org.apache.spark.sql.catalyst.InternalRow>

Parameters:

row - a data row