org.wikimedia.gobblin.writer.partitioner

Class TimestampedRecordOrJsonStringTimeBasedWriterPartitioner

java.lang.Object

org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner<TimestampedRecord<P>>

org.wikimedia.gobblin.writer.partitioner.TimestampedRecordTimeBasedWriterPartitioner<String>

org.wikimedia.gobblin.writer.partitioner.TimestampedRecordOrJsonStringTimeBasedWriterPartitioner

All Implemented Interfaces:

org.apache.gobblin.writer.partitioner.WriterPartitioner<TimestampedRecord<String>>

public class TimestampedRecordOrJsonStringTimeBasedWriterPartitioner
extends TimestampedRecordTimeBasedWriterPartitioner<String>

A TimeBasedWriterPartitioner for Timestamped byte[] records whose payload contains json. This class is a superclass of TimestampedRecordTimeBasedWriterPartitioner overriding the TimestampedRecordTimeBasedWriterPartitioner.getNonKafkaTimestamp(P) method only. This class uses JsonStringTimestampExtractor to parse the Json and extract the timestamp. It takes advantage of JsonStringTimeBasedWriterPartitioner configuration keys for timestamp column and timestamp format, as well as static classes to extract their values. Note: If a record doesn't contain the specified field, or if no field is specified, the current timestamp will be used.

Field Summary

Fields inherited from class org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner

DEFAULT_WRITER_PARTITION_GRANULARITY, DEFAULT_WRITER_PARTITION_TIMEUNIT, DEFAULT_WRITER_PARTITION_TIMEZONE, PARTITIONED_PATH, PREFIX, SUFFIX, timeUnit, WRITER_PARTITION_GRANULARITY, WRITER_PARTITION_PATTERN, WRITER_PARTITION_PREFIX, WRITER_PARTITION_SUFFIX, WRITER_PARTITION_TIMEUNIT, WRITER_PARTITION_TIMEZONE

Constructor Summary

Constructors

Constructor and Description
TimestampedRecordOrJsonStringTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state)
TimestampedRecordOrJsonStringTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state, int numBranches, int branchId)

Method Summary

Modifier and Type	Method and Description
protected long	getNonKafkaTimestamp(String payload)

Methods inherited from class org.wikimedia.gobblin.writer.partitioner.TimestampedRecordTimeBasedWriterPartitioner

getRecordTimestamp

Methods inherited from class org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner

getTimeUnit, partitionForRecord, partitionSchema

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TimestampedRecordOrJsonStringTimeBasedWriterPartitioner

public TimestampedRecordOrJsonStringTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state)

TimestampedRecordOrJsonStringTimeBasedWriterPartitioner

public TimestampedRecordOrJsonStringTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state,
                                                               int numBranches,
                                                               int branchId)

Method Detail

getNonKafkaTimestamp

protected long getNonKafkaTimestamp(String payload)

Overrides:

getNonKafkaTimestamp in class TimestampedRecordTimeBasedWriterPartitioner<String>