Package org.apache.iceberg.flink.maintenance.operator

Class OrphanFilesDetector

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction<String,String,String,String>

org.apache.iceberg.flink.maintenance.operator.OrphanFilesDetector

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction

@Internal
public class OrphanFilesDetector
extends org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction<String,String,String,String>

A specialized co-process function that performs an anti-join between two streams of file URIs.

Emits every file that exists in the file system but is not referenced in the table metadata, which are considered orphan files. It also handles URI normalization using provided scheme and authority equivalence mappings.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction

org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction.Context, org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction.OnTimerContext

Constructor Summary

Constructors

Constructor	Description
OrphanFilesDetector(DeleteOrphanFiles.PrefixMismatchMode prefixMismatchMode, Map<String,String> equalSchemes, Map<String,String> equalAuthorities)

Method Summary

Modifier and Type	Method	Description
void	onTimer(long timestamp, org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction<String,String,String,String>.org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction.OnTimerContext ctx, org.apache.flink.util.Collector<String> out)
void	open(org.apache.flink.api.common.functions.OpenContext openContext)
void	processElement1(String value, org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction<String,String,String,String>.org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction.Context context, org.apache.flink.util.Collector<String> collector)
void	processElement2(String value, org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction<String,String,String,String>.org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction.Context context, org.apache.flink.util.Collector<String> collector)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

OrphanFilesDetector

public OrphanFilesDetector(DeleteOrphanFiles.PrefixMismatchMode prefixMismatchMode,
 Map<String,String> equalSchemes,
 Map<String,String> equalAuthorities)

Method Details

open

public void open(org.apache.flink.api.common.functions.OpenContext openContext)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

processElement1

public void processElement1(String value,
 org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction<String,String,String,String>.org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction.Context context,
 org.apache.flink.util.Collector<String> collector)
                     throws Exception

Specified by:

processElement1 in class org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction<String,String,String,String>

Throws:

Exception

processElement2

public void processElement2(String value,
 org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction<String,String,String,String>.org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction.Context context,
 org.apache.flink.util.Collector<String> collector)
                     throws Exception

Specified by:

processElement2 in class org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction<String,String,String,String>

Throws:

Exception

onTimer

public void onTimer(long timestamp,
 org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction<String,String,String,String>.org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction.OnTimerContext ctx,
 org.apache.flink.util.Collector<String> out)
             throws Exception

Overrides:

onTimer in class org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction<String,String,String,String>

Throws:

Exception