Package org.apache.iceberg.spark.source

Class SparkPartitionedWriter

java.lang.Object

org.apache.iceberg.io.BaseTaskWriter<org.apache.spark.sql.catalyst.InternalRow>

org.apache.iceberg.io.PartitionedWriter<org.apache.spark.sql.catalyst.InternalRow>

org.apache.iceberg.spark.source.SparkPartitionedWriter

All Implemented Interfaces:

Closeable, AutoCloseable, TaskWriter<org.apache.spark.sql.catalyst.InternalRow>

public class SparkPartitionedWriter
extends PartitionedWriter<org.apache.spark.sql.catalyst.InternalRow>

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.iceberg.io.BaseTaskWriter

BaseTaskWriter.BaseEqualityDeltaWriter, BaseTaskWriter.RollingEqDeleteWriter, BaseTaskWriter.RollingFileWriter

Constructor Summary

Constructors

Constructor	Description
SparkPartitionedWriter(PartitionSpec spec, FileFormat format, FileAppenderFactory<org.apache.spark.sql.catalyst.InternalRow> appenderFactory, OutputFileFactory fileFactory, FileIO io, long targetFileSize, Schema schema, org.apache.spark.sql.types.StructType sparkSchema)

Method Summary

Modifier and Type	Method	Description
protected PartitionKey	partition(org.apache.spark.sql.catalyst.InternalRow row)	Create a PartitionKey from the values in row.

Methods inherited from class org.apache.iceberg.io.PartitionedWriter

close, write

Methods inherited from class org.apache.iceberg.io.BaseTaskWriter

abort, complete, setFailure, spec

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.io.TaskWriter

dataFiles

Constructor Details

SparkPartitionedWriter

public SparkPartitionedWriter(PartitionSpec spec,
 FileFormat format,
 FileAppenderFactory<org.apache.spark.sql.catalyst.InternalRow> appenderFactory,
 OutputFileFactory fileFactory,
 FileIO io,
 long targetFileSize,
 Schema schema,
 org.apache.spark.sql.types.StructType sparkSchema)

Method Details

partition

protected PartitionKey partition(org.apache.spark.sql.catalyst.InternalRow row)

Description copied from class: PartitionedWriter

Create a PartitionKey from the values in row.

Any PartitionKey returned by this method can be reused by the implementation.

Specified by:

partition in class PartitionedWriter<org.apache.spark.sql.catalyst.InternalRow>

Parameters:

row - a data row