Kafka with Flume

["Baris Akgun (Garanti Teknoloji)" <Ba...@garanti.com.tr>]

Hi,

We are using flume with kafka source. Flume gets json messages from kafka topic with given morphline file. If any message that has not well formed with morphline file json structure is added to kafka topic, flume (consumer) always tries to consume that wrong message. When we looked the kafka topic offset, we noticed that flume can not update offset value if any not well formed message comes. Is there any paremeter to pass wrong message and update the kafka offset in flume confugration file?

Thanks.

Barış Akgün

Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.

This message and attachments are confidential and intended solely for the individual(s) stated in this message. If you received this message although you are not the addressee, you are responsible to keep the message confidential. The sender has no responsibility for the accuracy or correctness of the information in the message and its attachments. Our company shall have no liability for any changes or late receiving, loss of integrity and confidentiality, viruses and any damages caused in anyway to your computer system.

Re: Kafka with Flume

[Jeff Holoman <jh...@cloudera.com>]

This is probably better posted on the flume dev or users list (
dev@flume.apache.org and user@flume.apache.org). I suspect you'll get a
better response there (or even the Cloudera community forums as there is
likely some Kite SDK experience there)

I think what you are saying is that you have a morphline (using Flume
Morphline interceptor) that parses JSON events, but that you can't
guarantee your input data will match your morphline extraction, so you are
getting some unhandled exception during processing, and this seems to be
failing the batch, which means the offsets won't get committed.

If you could post to the flume lists and include any stack traces, along
with your Flume configuration, I'm sure that would be helpful. I don't have
a ton of experience using morphlines but I'm happy to take a look.

In the meantime, you might check out this link, as I suspect it wold be
relevant:

http://kitesdk.org/docs/0.17.1/morphlines/morphlines-reference-guide.html#tryRules

Cheers,

Jeff

On Mon, Sep 19, 2016 at 4:52 AM, Baris Akgun (Garanti Teknoloji) <
BarisAkgu@garanti.com.tr> wrote:

> Hi,
>
> We are using flume with kafka source. Flume gets json messages from kafka
> topic with given morphline file. If any message that has not well formed
> with morphline file json structure is added to kafka topic, flume
> (consumer) always tries to consume that wrong message. When we looked the
> kafka topic offset, we noticed that flume can not update offset value if
> any not well formed message comes. Is there any paremeter to pass wrong
> message and update the kafka offset in flume confugration file?
>
> Thanks.
>
> Barış Akgün
>
> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir
> ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis
> olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune
> uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde
> yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da
> sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin
> ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan,
> butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve
> bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.
>
> This message and attachments are confidential and intended solely for the
> individual(s) stated in this message. If you received this message although
> you are not the addressee, you are responsible to keep the message
> confidential. The sender has no responsibility for the accuracy or
> correctness of the information in the message and its attachments. Our
> company shall have no liability for any changes or late receiving, loss of
> integrity and confidentiality, viruses and any damages caused in anyway to
> your computer system.
>



--