org.apache.iceberg.mr.hive

Class HiveIcebergOutputCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapred.OutputCommitter

org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter

public class HiveIcebergOutputCommitter
extends org.apache.hadoop.mapred.OutputCommitter

An Iceberg table committer for adding data files to the Iceberg tables. Currently independent of the Hive ACID transactions.

Constructor Summary

Constructors

Constructor and Description
HiveIcebergOutputCommitter()

Method Summary

Modifier and Type	Method and Description
void	abortJob(org.apache.hadoop.mapred.JobContext jobContext, int status) Removes the generated data files, if there is a commit file already generated for them.
void	abortTask(org.apache.hadoop.mapred.TaskAttemptContext context) Removes files generated by this task.
void	commitJob(org.apache.hadoop.mapred.JobContext jobContext) Reads the commit files stored in the temp directory and collects the generated committed data files.
void	commitTask(org.apache.hadoop.mapred.TaskAttemptContext context) Collects the generated data files and creates a commit file storing the data file list.
boolean	needsTaskCommit(org.apache.hadoop.mapred.TaskAttemptContext context)
void	setupJob(org.apache.hadoop.mapred.JobContext jobContext)
void	setupTask(org.apache.hadoop.mapred.TaskAttemptContext taskAttemptContext)

Methods inherited from class org.apache.hadoop.mapred.OutputCommitter

abortJob, abortTask, cleanupJob, cleanupJob, commitJob, commitTask, isCommitJobRepeatable, isCommitJobRepeatable, isRecoverySupported, isRecoverySupported, isRecoverySupported, needsTaskCommit, recoverTask, recoverTask, setupJob, setupTask

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HiveIcebergOutputCommitter

public HiveIcebergOutputCommitter()

Method Detail

setupJob

public void setupJob(org.apache.hadoop.mapred.JobContext jobContext)

Specified by:

setupJob in class org.apache.hadoop.mapred.OutputCommitter

setupTask

public void setupTask(org.apache.hadoop.mapred.TaskAttemptContext taskAttemptContext)

Specified by:

setupTask in class org.apache.hadoop.mapred.OutputCommitter

needsTaskCommit

public boolean needsTaskCommit(org.apache.hadoop.mapred.TaskAttemptContext context)

Specified by:

needsTaskCommit in class org.apache.hadoop.mapred.OutputCommitter

commitTask

public void commitTask(org.apache.hadoop.mapred.TaskAttemptContext context)
                throws java.io.IOException

Collects the generated data files and creates a commit file storing the data file list.

Specified by:

commitTask in class org.apache.hadoop.mapred.OutputCommitter

Parameters:

context - The job context

Throws:

java.io.IOException - Thrown if there is an error writing the commit file

abortTask

public void abortTask(org.apache.hadoop.mapred.TaskAttemptContext context)
               throws java.io.IOException

Removes files generated by this task.

Specified by:

abortTask in class org.apache.hadoop.mapred.OutputCommitter

Parameters:

context - The task context

Throws:

java.io.IOException - Thrown if there is an error closing the writer

commitJob

public void commitJob(org.apache.hadoop.mapred.JobContext jobContext)
               throws java.io.IOException

Reads the commit files stored in the temp directory and collects the generated committed data files. Appends the data files to the table. At the end removes the temporary directory.

Overrides:

commitJob in class org.apache.hadoop.mapred.OutputCommitter

Parameters:

jobContext - The job context

Throws:

java.io.IOException - if there is a failure deleting the files

abortJob

public void abortJob(org.apache.hadoop.mapred.JobContext jobContext,
                     int status)
              throws java.io.IOException

Removes the generated data files, if there is a commit file already generated for them. The cleanup at the end removes the temporary directory as well.

Overrides:

abortJob in class org.apache.hadoop.mapred.OutputCommitter

Parameters:

jobContext - The job context

status - The status of the job

Throws:

java.io.IOException - if there is a failure deleting the files