org.wikimedia.gobblin.publisher

Class TimePartitionedFlagDataPublisher

java.lang.Object

org.apache.gobblin.publisher.DataPublisher

org.wikimedia.gobblin.publisher.TimePartitionedFlagDataPublisher

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.gobblin.capability.CapabilityAware

public class TimePartitionedFlagDataPublisher
extends org.apache.gobblin.publisher.DataPublisher

IMPORTANT NOTES: - Gobblin published folders are expected to be in the form {PUBLISHERDIR}/{TABLE_NAME}/{PARTITION} This means that the "writer.file.path.type" property is expected to be "tablename" - Gobblin writter partition-scheme is expected to follow time-order when sorted alphabetically Publisher output flag path: {PUBLISHERDIR}/{TABLE_NAME}/{PARTITION}/{FLAG} For topics having crossed time-partitions boundary across all their kafka-partitions. The partitions to be flagged are processed by topic. The flag is written on every time-partition folder having been written to by writers, and for which every kafka partition have also written to a later partition.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_PUBLISHER_TIME_PARTITION_FLAG
static String	PUBLISHER_PUBLISHED_FLAGS_KEY
static String	PUBLISHER_TIME_PARTITION_FLAG_KEY

Fields inherited from class org.apache.gobblin.publisher.DataPublisher

REUSABLE, state

Constructor Summary

Constructors

Constructor and Description
TimePartitionedFlagDataPublisher(org.apache.gobblin.configuration.State state)

Method Summary

Modifier and Type	Method and Description
void	close()
static org.apache.hadoop.fs.FileSystem	getPublisherFileSystem(org.apache.gobblin.configuration.State state)
void	initialize() Deprecated.
void	publishData(Collection<? extends org.apache.gobblin.configuration.WorkUnitState> states)
void	publishMetadata(Collection<? extends org.apache.gobblin.configuration.WorkUnitState> states)

Methods inherited from class org.apache.gobblin.publisher.DataPublisher

canBeSkipped, getInstance, getState, isThreadSafe, markCommit, publish, shouldPublishMetadataFirst, supportsCapability

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PUBLISHER_PUBLISHED_FLAGS_KEY

public static final String PUBLISHER_PUBLISHED_FLAGS_KEY

See Also:

Constant Field Values

PUBLISHER_TIME_PARTITION_FLAG_KEY

public static final String PUBLISHER_TIME_PARTITION_FLAG_KEY

See Also:

Constant Field Values

DEFAULT_PUBLISHER_TIME_PARTITION_FLAG

public static final String DEFAULT_PUBLISHER_TIME_PARTITION_FLAG

See Also:

Constant Field Values

Constructor Detail

TimePartitionedFlagDataPublisher

public TimePartitionedFlagDataPublisher(org.apache.gobblin.configuration.State state)
                                 throws IOException

Throws:

IOException

Method Detail

getPublisherFileSystem

public static org.apache.hadoop.fs.FileSystem getPublisherFileSystem(org.apache.gobblin.configuration.State state)
                                                              throws IOException

Throws:

IOException

initialize

@Deprecated
public void initialize()

Deprecated.

Specified by:

initialize in class org.apache.gobblin.publisher.DataPublisher

close

public void close()
           throws IOException

Throws:

IOException

publishData

public void publishData(Collection<? extends org.apache.gobblin.configuration.WorkUnitState> states)
                 throws IOException

Specified by:

publishData in class org.apache.gobblin.publisher.DataPublisher

Throws:

IOException

publishMetadata

public void publishMetadata(Collection<? extends org.apache.gobblin.configuration.WorkUnitState> states)
                     throws IOException

Specified by:

publishMetadata in class org.apache.gobblin.publisher.DataPublisher

Throws:

IOException