org.wikimedia.gobblin.copy

Class TimePartitionedDataPublisher

java.lang.Object

org.apache.gobblin.publisher.DataPublisher

org.apache.gobblin.publisher.SingleTaskDataPublisher

org.wikimedia.gobblin.copy.BaseDataPublisher

org.wikimedia.gobblin.copy.TimePartitionedDataPublisher

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.gobblin.capability.CapabilityAware

Direct Known Subclasses:

TimePartitionedDataPublisherWithFlag

public class TimePartitionedDataPublisher
extends BaseDataPublisher

For time partition jobs, writer output directory is $GOBBLIN_WORK_DIR/task-output/{extractId}/{tableName}/{partitionPath}, where partition path is the time bucket, e.g., 2015/04/08/15. Publisher output directory is $GOBBLIN_WORK_DIR/job-output/{tableName}/{partitionPath}

This is an updated copy of TimePartitionedDataPublisher in gobblin-core module. This file should be deleted in favor of the upstream version when possible. Updates are:

• Override addWriterOutputToNewDir(org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path, org.apache.gobblin.configuration.WorkUnitState, int, org.apache.gobblin.util.ParallelRunner) to recursively iterate moving files from subfolders, allowing to have detailed folders in states PUBLISH_DIRS -- https://github.com/apache/gobblin/pull/3409

Field Summary

Fields inherited from class org.wikimedia.gobblin.copy.BaseDataPublisher

closer, lineageInfo, metadataMergers, metaDataWriterFileSystemByBranches, numBranches, parallelRunnerCloser, parallelRunners, parallelRunnerThreads, permissions, publisherFileSystemByBranches, publisherFinalDirOwnerGroupsByBranches, publisherOutputDirs, retrierConfig, shouldRetry, writerFileSystemByBranches

Fields inherited from class org.apache.gobblin.publisher.DataPublisher

REUSABLE, state

Constructor Summary

Constructors

Constructor and Description
TimePartitionedDataPublisher(org.apache.gobblin.configuration.State state)

Method Summary

Modifier and Type	Method and Description
protected void	addWriterOutputToExistingDir(org.apache.hadoop.fs.Path writerOutput, org.apache.hadoop.fs.Path publisherOutput, org.apache.gobblin.configuration.WorkUnitState workUnitState, int branchId, org.apache.gobblin.util.ParallelRunner parallelRunner) This method needs to be overridden for TimePartitionedDataPublisher, since the output folder structure contains timestamp, we have to move the files recursively.
protected void	addWriterOutputToNewDir(org.apache.hadoop.fs.Path writerOutput, org.apache.hadoop.fs.Path publisherOutput, org.apache.gobblin.configuration.WorkUnitState workUnitState, int branchId, org.apache.gobblin.util.ParallelRunner parallelRunner) This method needs to be overridden for TimePartitionedDataPublisher, since the output folder structure contains timestamp, we need to move the files recursively.

Methods inherited from class org.wikimedia.gobblin.copy.BaseDataPublisher

addSingleTaskWriterOutputToExistingDir, close, createDestinationDescriptor, getPublisherOutputDir, initialize, movePath, publishData, publishData, publishData, publishMetadata, publishMetadata, publishMultiTaskData, recordPublisherOutputDirs, shouldPublishMetadataFirst

Methods inherited from class org.apache.gobblin.publisher.SingleTaskDataPublisher

getInstance, publish

Methods inherited from class org.apache.gobblin.publisher.DataPublisher

canBeSkipped, getState, isThreadSafe, markCommit, publish, supportsCapability

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TimePartitionedDataPublisher

public TimePartitionedDataPublisher(org.apache.gobblin.configuration.State state)
                             throws IOException

Throws:

IOException

Method Detail

addWriterOutputToNewDir

protected void addWriterOutputToNewDir(org.apache.hadoop.fs.Path writerOutput,
                                       org.apache.hadoop.fs.Path publisherOutput,
                                       org.apache.gobblin.configuration.WorkUnitState workUnitState,
                                       int branchId,
                                       org.apache.gobblin.util.ParallelRunner parallelRunner)
                                throws IOException

This method needs to be overridden for TimePartitionedDataPublisher, since the output folder structure contains timestamp, we need to move the files recursively. For example, move {writerOutput}/2015/04/08/15/output.avro to {publisherOutput}/2015/04/08/15/output.avro

Overrides:

addWriterOutputToNewDir in class BaseDataPublisher

Throws:

IOException

addWriterOutputToExistingDir

protected void addWriterOutputToExistingDir(org.apache.hadoop.fs.Path writerOutput,
                                            org.apache.hadoop.fs.Path publisherOutput,
                                            org.apache.gobblin.configuration.WorkUnitState workUnitState,
                                            int branchId,
                                            org.apache.gobblin.util.ParallelRunner parallelRunner)
                                     throws IOException

This method needs to be overridden for TimePartitionedDataPublisher, since the output folder structure contains timestamp, we have to move the files recursively. For example, move {writerOutput}/2015/04/08/15/output.avro to {publisherOutput}/2015/04/08/15/output.avro

Overrides:

addWriterOutputToExistingDir in class BaseDataPublisher

Throws:

IOException