Class SerializableTableWithSize

  • All Implemented Interfaces:, java.lang.AutoCloseable, HasTableOperations, Table, org.apache.spark.util.KnownSizeEstimation

    public class SerializableTableWithSize
    extends SerializableTable
    implements org.apache.spark.util.KnownSizeEstimation, java.lang.AutoCloseable
    This class provides a serializable table with a known size estimate. Spark calls its SizeEstimator class when broadcasting variables and this can be an expensive operation, so providing a known size estimate allows that operation to be skipped.

    This class also implements AutoCloseable to avoid leaking resources upon broadcasting. Broadcast variables are destroyed and cleaned up on the driver and executors once they are garbage collected on the driver. The implementation ensures only resources used by copies of the main table are released.

    See Also:
    Serialized Form
    • Constructor Detail

      • SerializableTableWithSize

        protected SerializableTableWithSize​(Table table)
    • Method Detail

      • estimatedSize

        public long estimatedSize()
        Specified by:
        estimatedSize in interface org.apache.spark.util.KnownSizeEstimation
      • copyOf

        public static Table copyOf​(Table table)
      • close

        public void close()
                   throws java.lang.Exception
        Specified by:
        close in interface java.lang.AutoCloseable