Package org.apache.iceberg.spark.source

Class SerializableTableWithSize

java.lang.Object

org.apache.iceberg.SerializableTable

org.apache.iceberg.spark.source.SerializableTableWithSize

All Implemented Interfaces:

Serializable, AutoCloseable, HasTableOperations, Table, org.apache.spark.util.KnownSizeEstimation

public class SerializableTableWithSize
extends SerializableTable
implements org.apache.spark.util.KnownSizeEstimation, AutoCloseable

This class provides a serializable table with a known size estimate. Spark calls its SizeEstimator class when broadcasting variables and this can be an expensive operation, so providing a known size estimate allows that operation to be skipped.

This class also implements AutoCloseable to avoid leaking resources upon broadcasting. Broadcast variables are destroyed and cleaned up on the driver and executors once they are garbage collected on the driver. The implementation ensures only resources used by copies of the main table are released.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	SerializableTableWithSize.SerializableMetadataTableWithSize

Nested classes/interfaces inherited from class org.apache.iceberg.SerializableTable

SerializableTable.SerializableMetadataTable

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	SerializableTableWithSize(Table table)

Method Summary

Modifier and Type	Method	Description
void	close()
static Table	copyOf(Table table)
long	estimatedSize()

Methods inherited from class org.apache.iceberg.SerializableTable

currentSnapshot, encryption, expireSnapshots, formatVersion, history, io, location, locationProvider, manageSnapshots, metadataFileLocation, name, newAppend, newBatchScan, newDelete, newIncrementalAppendScan, newIncrementalChangelogScan, newOverwrite, newReplacePartitions, newRewrite, newRowDelta, newScan, newTable, newTransaction, operations, partitionStatisticsFiles, properties, refresh, refs, replaceSortOrder, rewriteManifests, schema, schemas, snapshot, snapshots, sortOrder, sortOrders, spec, specs, statisticsFiles, updateLocation, updatePartitionStatistics, updateProperties, updateSchema, updateSpec, updateStatistics, uuid

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.Table

newFastAppend, snapshot

Constructor Details

SerializableTableWithSize

protected SerializableTableWithSize(Table table)

Method Details

estimatedSize

public long estimatedSize()

Specified by:

estimatedSize in interface org.apache.spark.util.KnownSizeEstimation

copyOf

public static Table copyOf(Table table)

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception