org.wikimedia.gobblin.kafka

Class WikimediaKafkaSource<S,D>

java.lang.Object

org.apache.gobblin.source.extractor.extract.AbstractSource<S,D>

org.apache.gobblin.source.extractor.extract.EventBasedSource<S,D>

org.wikimedia.gobblin.copy.KafkaSource<S,D>

org.wikimedia.gobblin.kafka.WikimediaKafkaSource<S,D>

All Implemented Interfaces:

org.apache.gobblin.source.Source<S,D>

Direct Known Subclasses:

Kafka1TimestampedRecordSource

public abstract class WikimediaKafkaSource<S,D>
extends KafkaSource<S,D>

Field Summary

Fields inherited from class org.wikimedia.gobblin.copy.KafkaSource

ALL_TOPICS, AVG_RECORD_MILLIS, BOOTSTRAP_WITH_OFFSET, DEFAULT_BOOTSTRAP_WITH_OFFSET, DEFAULT_GOBBLIN_KAFKA_CONSUMER_CLIENT_FACTORY_CLASS, DEFAULT_GOBBLIN_KAFKA_SHOULD_ENABLE_DATASET_STATESTORE, DEFAULT_MAX_POSSIBLE_OBSERVED_LATENCY_IN_HOURS, DEFAULT_NAMESPACE_NAME, DEFAULT_OBSERVED_LATENCY_MEASUREMENT_ENABLED, DEFAULT_OBSERVED_LATENCY_PRECISION, DEFAULT_RESET_ON_OFFSET_OUT_OF_RANGE, DEFAULT_TABLE_TYPE, EARLIEST_OFFSET, GOBBLIN_KAFKA_CONSUMER_CLIENT_FACTORY_CLASS, GOBBLIN_KAFKA_EXTRACT_ALLOW_TABLE_TYPE_NAMESPACE_CUSTOMIZATION, GOBBLIN_KAFKA_SHOULD_ENABLE_DATASET_STATESTORE, KAFKA_OFFSET_LOOKBACK, kafkaConsumerClient, kafkaConsumerClientPool, LATEST_OFFSET, LEADER_HOSTANDPORT, LEADER_ID, lineageInfo, MAX_POSSIBLE_OBSERVED_LATENCY_IN_HOURS, NEAREST_OFFSET, NUM_TOPIC_PARTITIONS, OBSERVED_LATENCY_MEASUREMENT_ENABLED, OBSERVED_LATENCY_PRECISION, OFFSET_FETCH_EPOCH_TIME, OFFSET_FETCH_TIMER, OFFSET_LOOKBACK, PARTITION_ID, PREVIOUS_HIGH_WATERMARK, PREVIOUS_LATEST_OFFSET, PREVIOUS_LOW_WATERMARK, PREVIOUS_OFFSET_FETCH_EPOCH_TIME, PREVIOUS_START_FETCH_EPOCH_TIME, PREVIOUS_STOP_FETCH_EPOCH_TIME, RECORD_CREATION_TIMESTAMP_FIELD, RECORD_CREATION_TIMESTAMP_UNIT, RECORD_LEVEL_SLA_MINUTES_KEY, RESET_ON_OFFSET_OUT_OF_RANGE, START_FETCH_EPOCH_TIME, STOP_FETCH_EPOCH_TIME, TOPIC_BLACKLIST, TOPIC_NAME, TOPIC_WHITELIST, TOPICS_MOVE_TO_LATEST_OFFSET

Constructor Summary

Constructors

Constructor and Description
WikimediaKafkaSource()

Method Summary

Modifier and Type	Method and Description
protected List<org.apache.gobblin.source.extractor.extract.kafka.KafkaTopic>	getFilteredTopics(org.apache.gobblin.configuration.SourceState state) Finds Kafka topics to ingest.
protected Map<String,String>	settingsListToMap(Collection<String> settingsFilters)

Methods inherited from class org.wikimedia.gobblin.copy.KafkaSource

getWorkunits, isTopicQualified, populateClientPool, shutdown

Methods inherited from class org.apache.gobblin.source.extractor.extract.AbstractSource

createExtract, getPreviousWorkUnitsForRetry, getPreviousWorkUnitStatesForRetry, isEarlyStopped

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.gobblin.source.Source

getExtractor

Constructor Detail

WikimediaKafkaSource

public WikimediaKafkaSource()

Method Detail

getFilteredTopics

protected List<org.apache.gobblin.source.extractor.extract.kafka.KafkaTopic> getFilteredTopics(org.apache.gobblin.configuration.SourceState state)

Finds Kafka topics to ingest. Defaults to a literal include list. In absence of that, uses Wikimedia's EventStreamConfig service to get a list of topics based on: - a configured service URI - a list of stream names to get topics from - a list of settings filters in the form key1:value1,key2:value2

Overrides:

getFilteredTopics in class KafkaSource<S,D>

Parameters:

state - Work unit state

Returns:

List of configured topics

settingsListToMap

protected Map<String,String> settingsListToMap(Collection<String> settingsFilters)