org.wikimedia.gobblin.writer.partitioner

Class JsonStringTimeBasedWriterPartitioner

java.lang.Object

org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner<String>

org.wikimedia.gobblin.writer.partitioner.JsonStringTimeBasedWriterPartitioner

All Implemented Interfaces:

org.apache.gobblin.writer.partitioner.WriterPartitioner<String>

public class JsonStringTimeBasedWriterPartitioner
extends org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner<String>

A TimeBasedWriterPartitioner for byte[] containing json. The json field that contains the timestamp can be specified using TIMESTAMP_COLUMN_KEY. The format of the json field that contains the timestamp can be specified using TIMESTAMP_FORMAT_KEY. If a record doesn't contain the specified field, or if no field is specified, the current timestamp will be used.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_TIMESTAMP_COLUMN
static String	DEFAULT_TIMESTAMP_FORMAT
static String	TIMESTAMP_COLUMNS_KEY
static String	TIMESTAMP_FORMAT_KEY

Fields inherited from class org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner

DEFAULT_WRITER_PARTITION_GRANULARITY, DEFAULT_WRITER_PARTITION_TIMEUNIT, DEFAULT_WRITER_PARTITION_TIMEZONE, PARTITIONED_PATH, PREFIX, SUFFIX, timeUnit, WRITER_PARTITION_GRANULARITY, WRITER_PARTITION_PATTERN, WRITER_PARTITION_PREFIX, WRITER_PARTITION_SUFFIX, WRITER_PARTITION_TIMEUNIT, WRITER_PARTITION_TIMEZONE

Constructor Summary

Constructors

Constructor and Description
JsonStringTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state)
JsonStringTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state, int numBranches, int branchId)

Method Summary

Modifier and Type	Method and Description
long	getRecordTimestamp(String rec)
static List<String>	getWriterPartitionerTimestampColumns(org.apache.gobblin.configuration.State state, int numBranches, int branchId) Utility function facilitating getting a timestamp-column.
static String	getWriterPartitionerTimestampFormat(org.apache.gobblin.configuration.State state, int numBranches, int branchId) Utility function facilitating getting a timestamp-format.

Methods inherited from class org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner

getTimeUnit, partitionForRecord, partitionSchema

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TIMESTAMP_COLUMNS_KEY

public static final String TIMESTAMP_COLUMNS_KEY

See Also:

Constant Field Values

DEFAULT_TIMESTAMP_COLUMN

public static final String DEFAULT_TIMESTAMP_COLUMN

See Also:

Constant Field Values

TIMESTAMP_FORMAT_KEY

public static final String TIMESTAMP_FORMAT_KEY

See Also:

Constant Field Values

DEFAULT_TIMESTAMP_FORMAT

public static final String DEFAULT_TIMESTAMP_FORMAT

See Also:

Constant Field Values

Constructor Detail

JsonStringTimeBasedWriterPartitioner

public JsonStringTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state)

JsonStringTimeBasedWriterPartitioner

public JsonStringTimeBasedWriterPartitioner(org.apache.gobblin.configuration.State state,
                                            int numBranches,
                                            int branchId)

Method Detail

getRecordTimestamp

public long getRecordTimestamp(String rec)

Specified by:

getRecordTimestamp in class org.apache.gobblin.writer.partitioner.TimeBasedWriterPartitioner<String>

getWriterPartitionerTimestampColumns

public static List<String> getWriterPartitionerTimestampColumns(org.apache.gobblin.configuration.State state,
                                                                int numBranches,
                                                                int branchId)

Utility function facilitating getting a timestamp-column.

getWriterPartitionerTimestampFormat

public static String getWriterPartitionerTimestampFormat(org.apache.gobblin.configuration.State state,
                                                         int numBranches,
                                                         int branchId)

Utility function facilitating getting a timestamp-format.