org.wikimedia.gobblin.writer.partitioner

Class TimestampedRecordTimeBasedWriterPartitioner<P>

java.lang.Object

org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner<TimestampedRecord<P>>

org.wikimedia.gobblin.writer.partitioner.TimestampedRecordTimeBasedWriterPartitioner<P>

All Implemented Interfaces:

org.apache.gobblin.writer.partitioner.WriterPartitioner<TimestampedRecord<P>>

Direct Known Subclasses:

TimestampedRecordOrJsonStringTimeBasedWriterPartitioner

public class TimestampedRecordTimeBasedWriterPartitioner<P>
extends org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner<TimestampedRecord<P>>

A TimeBasedWriterPartitioner for byte[] containing json. The json field that contains the timestamp can be specified using TIMESTAMP_COLUMN_KEY. The format of the json field that contains the timestamp can be specified using TIMESTAMP_FORMAT_KEY. If a record doesn't contain the specified field, or if no field is specified, the current timestamp will be used.

Field Summary

Fields inherited from class org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner

DEFAULT_WRITER_PARTITION_GRANULARITY, DEFAULT_WRITER_PARTITION_TIMEUNIT, DEFAULT_WRITER_PARTITION_TIMEZONE, PARTITIONED_PATH, PREFIX, SUFFIX, timeUnit, WRITER_PARTITION_GRANULARITY, WRITER_PARTITION_PATTERN, WRITER_PARTITION_PREFIX, WRITER_PARTITION_SUFFIX, WRITER_PARTITION_TIMEUNIT, WRITER_PARTITION_TIMEZONE

Constructor Summary

Constructors

Constructor and Description
TimestampedRecordTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state)
TimestampedRecordTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state, int numBranches, int branchId)

Method Summary

Modifier and Type	Method and Description
protected long	getNonKafkaTimestamp(P payload)
long	getRecordTimestamp(TimestampedRecord<P> rec)

Methods inherited from class org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner

getTimeUnit, partitionForRecord, partitionSchema

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TimestampedRecordTimeBasedWriterPartitioner

public TimestampedRecordTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state)

TimestampedRecordTimeBasedWriterPartitioner

public TimestampedRecordTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state,
                                                   int numBranches,
                                                   int branchId)

Method Detail

getRecordTimestamp

public long getRecordTimestamp(TimestampedRecord<P> rec)

Specified by:

getRecordTimestamp in class org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner<TimestampedRecord<P>>

getNonKafkaTimestamp

protected long getNonKafkaTimestamp(P payload)