You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by Prateek Gupta <pr...@myntra.com> on 2018/09/23 08:30:18 UTC

Kafka Extractor

Hi Team,

I have a query regarding the KafkaDeserializerExtractor. May I've missed
something, but there's no out-of-the-box API provided to read the records
from Kafka *one-by-one*.

I could only see *readRecordImpl* (KafkaExtractor) which inherently uses a
*messageIterator* (access modifier is private). Our use is that we wanted
to leverage the WorkUnits genrated by Gobblin and do some of our own custom
processing / calculation on the records being read.

Any advise or guidance would be greatly appreciated!!!

Thanks,
Prateek Gupta

Re: Kafka Extractor

Posted by Tamas Nemeth <ta...@prezi.com.INVALID>.
Hey Prateek,

Why don't you use a converter for that?
For example we do geoip enrichment for the records we read from Kafka and
it is implemented as a converter ->
https://github.com/prezi/gobblin/blob/geoip_lookup_enricher/gobblin-modules/gobblin-geoip/src/main/java/org/apache/gobblin/converter/avro/AvroGeoipEnricherConverter.java

Tamas

On Sun, Sep 23, 2018 at 10:30 AM Prateek Gupta <pr...@myntra.com>
wrote:

> Hi Team,
>
> I have a query regarding the KafkaDeserializerExtractor. May I've missed
> something, but there's no out-of-the-box API provided to read the records
> from Kafka *one-by-one*.
>
> I could only see *readRecordImpl* (KafkaExtractor) which inherently uses a
> *messageIterator* (access modifier is private). Our use is that we wanted
> to leverage the WorkUnits genrated by Gobblin and do some of our own custom
> processing / calculation on the records being read.
>
> Any advise or guidance would be greatly appreciated!!!
>
> Thanks,
> Prateek Gupta
>

Re: Kafka Extractor

Posted by Tamas Nemeth <ta...@prezi.com>.
Hey Prateek,

Why don't you use a converter for that?
For example we do geoip enrichment for the records we read from Kafka and
it is implemented as a converter ->
https://github.com/prezi/gobblin/blob/geoip_lookup_enricher/gobblin-modules/gobblin-geoip/src/main/java/org/apache/gobblin/converter/avro/AvroGeoipEnricherConverter.java

Tamas

On Sun, Sep 23, 2018 at 10:30 AM Prateek Gupta <pr...@myntra.com>
wrote:

> Hi Team,
>
> I have a query regarding the KafkaDeserializerExtractor. May I've missed
> something, but there's no out-of-the-box API provided to read the records
> from Kafka *one-by-one*.
>
> I could only see *readRecordImpl* (KafkaExtractor) which inherently uses a
> *messageIterator* (access modifier is private). Our use is that we wanted
> to leverage the WorkUnits genrated by Gobblin and do some of our own custom
> processing / calculation on the records being read.
>
> Any advise or guidance would be greatly appreciated!!!
>
> Thanks,
> Prateek Gupta
>