K
- Message key typeV
- Message value typepublic class Kafka1ConsumerClient<K,V>
extends org.apache.gobblin.kafka.client.AbstractBaseKafkaConsumerClient
GobblinKafkaConsumerClient
that uses kafka 1.1 consumer client. Use Kafka1ConsumerClient.Factory.create(Config)
to create
new Kafka1.1ConsumerClients. The Config
used to create clients must have required key "source.kafka.value.deserializer"
This is an updated copy of Kafka1ConsumerClient
in gobblin-kafka-1 module. This file should be deleted in favor of the upstream version when possible.
Updates are:
Kafka1ConsumerClient.Kafka1ConsumerRecord
-- https://github.com/apache/gobblin/pull/3244Modifier and Type | Class and Description |
---|---|
static class |
Kafka1ConsumerClient.Factory
A factory class to instantiate
Kafka1ConsumerClient |
static class |
Kafka1ConsumerClient.Kafka1ConsumerRecord<K,V>
A record returned by
Kafka1ConsumerClient |
Modifier and Type | Field and Description |
---|---|
static String |
DEFAULT_KEY_DESERIALIZER |
static String |
GOBBLIN_CONFIG_KEY_DESERIALIZER_CLASS_KEY |
static String |
GOBBLIN_CONFIG_VALUE_DESERIALIZER_CLASS_KEY |
brokers, config, CONFIG_ENABLE_SCHEMA_CHECK, CONFIG_KAFKA_FETCH_REQUEST_MIN_BYTES, CONFIG_KAFKA_FETCH_TIMEOUT_VALUE, CONFIG_KAFKA_FETCH_TIMEOUT_VALUE_DEFAULT, CONFIG_KAFKA_SOCKET_TIMEOUT_VALUE, CONFIG_KAFKA_SOCKET_TIMEOUT_VALUE_DEFAULT, CONFIG_NAMESPACE, CONFIG_PREFIX, CONSUMER_CONFIG, ENABLE_SCHEMA_CHECK_DEFAULT, fetchMinBytes, fetchTimeoutMillis, schemaCheckEnabled, schemaRegistry, socketTimeoutMillis
Constructor and Description |
---|
Kafka1ConsumerClient(com.typesafe.config.Config config,
org.apache.kafka.clients.consumer.Consumer<K,V> consumer) |
Modifier and Type | Method and Description |
---|---|
void |
close() |
void |
commitOffsetsAsync(Map<org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition,Long> partitionOffsets)
Commit offsets to Kafka asynchronously
|
void |
commitOffsetsSync(Map<org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition,Long> partitionOffsets)
Commit offsets to Kafka synchronously
|
long |
committed(org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition partition)
returns the last committed offset for a KafkaPartition
|
Iterator<org.apache.gobblin.kafka.client.KafkaConsumerRecord> |
consume() |
Iterator<org.apache.gobblin.kafka.client.KafkaConsumerRecord> |
consume(org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition partition,
long nextOffset,
long maxOffset) |
long |
getEarliestOffset(org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition partition) |
long |
getLatestOffset(org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition partition) |
Map<String,com.codahale.metrics.Metric> |
getMetrics() |
List<org.apache.gobblin.source.extractor.extract.kafka.KafkaTopic> |
getTopics() |
void |
subscribe(String topic)
Subscribe to a kafka topic
TODO Add multi topic support
|
void |
subscribe(String topic,
org.apache.gobblin.kafka.client.GobblinConsumerRebalanceListener listener)
Subscribe to a kafka topic with a {#GobblinConsumerRebalanceListener}
TODO Add multi topic support
|
canonicalMetricName, getFilteredTopics
public static final String DEFAULT_KEY_DESERIALIZER
public static final String GOBBLIN_CONFIG_KEY_DESERIALIZER_CLASS_KEY
public static final String GOBBLIN_CONFIG_VALUE_DESERIALIZER_CLASS_KEY
public List<org.apache.gobblin.source.extractor.extract.kafka.KafkaTopic> getTopics()
getTopics
in class org.apache.gobblin.kafka.client.AbstractBaseKafkaConsumerClient
public long getEarliestOffset(org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition partition)
public long getLatestOffset(org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition partition) throws org.apache.gobblin.source.extractor.extract.kafka.KafkaOffsetRetrievalFailureException
org.apache.gobblin.source.extractor.extract.kafka.KafkaOffsetRetrievalFailureException
public Iterator<org.apache.gobblin.kafka.client.KafkaConsumerRecord> consume(org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition partition, long nextOffset, long maxOffset)
public Iterator<org.apache.gobblin.kafka.client.KafkaConsumerRecord> consume()
public void subscribe(String topic)
topic
- public void subscribe(String topic, org.apache.gobblin.kafka.client.GobblinConsumerRebalanceListener listener)
topic
- public void commitOffsetsAsync(Map<org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition,Long> partitionOffsets)
public void commitOffsetsSync(Map<org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition,Long> partitionOffsets)
public long committed(org.apache.gobblin.source.extractor.extract.kafka.KafkaPartition partition)
partition
- public void close() throws IOException
IOException
Copyright © 2021. All rights reserved.