You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Ali Karray <c...@karray.net> on 2016/04/06 12:10:48 UTC
putKafka processor with Avro Format
Hi,
I'm exploring integration possibility of nifi processors with kafka.
I use PutKafka to puch avro generated flow from ExecuteSQL.
I verify that the topic created in Kafka is alimented with data using
kafka-console-consumer bash command.
Then I try to use getKafka processor to pull all the Data stored in
the kafka topic and store all that data in a local File using the
PutFile processor. I realize that the getKafka processor don't pull
the topic content.
I verify if all is ok with kafka-avro-console-consumer (confluent
command), I get this exception :
org.apache.kafka.common.errors.SerializationException: Error
deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException:
Unknown magic byte!
I tested also getKafka from a basic topic format and it doesn't work for me.
So I have some questions :
How can I deal with avro format using getKafka anf putKafka processor ?
Any idea to resolve getKafka processor problem to pull content from a
kafka Topic ?
Thank you in advance
A. Karray
Re: putKafka processor with Avro Format
Posted by Ali Karray <c...@karray.net>.
Awesome product.
Thank you
Ali
On Wed, Apr 6, 2016 at 4:27 PM, Joe Witt <jo...@gmail.com> wrote:
> Ali - I can confirm that all formats are supported for Put/Get Kafka
> processors. The interaction with Kafka as far as the messages are
> concerned is just 'bytes'. The reason it isn't working well is due to
> the bug I'm mentioning and that is a NiFi bug (not Kafka). It is
> fixed and we'll have a release for you asap but if you want to build
> that branch then you're in great shape right away.
>
> Thanks
>
> On Wed, Apr 6, 2016 at 10:24 AM, Ali Karray <c...@karray.net> wrote:
>> I see the patch in Jira.
>> You confirm that avro format is supported for putKafka and getKafka processor ?
>>
>> Thanks
>> Ali
>>
>> On Wed, Apr 6, 2016 at 4:14 PM, Joe Witt <jo...@gmail.com> wrote:
>>> Ali
>>>
>>> It is very possible you're being bitten by a bug we have with
>>> PutKafka. It is fixed in this branch which we're in the release
>>> motions for right now.
>>>
>>> https://github.com/apache/nifi/tree/support/nifi-0.6.x
>>>
>>> Thanks
>>> Joe
>>>
>>> On Wed, Apr 6, 2016 at 10:07 AM, Ali Karray <c...@karray.net> wrote:
>>>> I'm using the last version 0.6 of Nifi and the 0.9 kafka version
>>>> When I use plain csv files, all is working.
>>>>
>>>> Thanks,
>>>> Ali
>>>>
>>>>
>>>> On Wed, Apr 6, 2016 at 3:23 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> What version of Apache NiFi are you using?
>>>>>
>>>>> Also, lets take Avro and kafka-avro-console-consumer out of the mix for a minute, what happens if you used PutKafka to send some plain text to Kafka, and then tried to consumer from that topic with GetKafka?
>>>>> You mentioned this didn't work, but what happened? was there an error message? did it just receive 0 messages from Kafka?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Bryan
>>>>>
>>>>> On Wed, Apr 6, 2016 at 6:10 AM, Ali Karray <c...@karray.net> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm exploring integration possibility of nifi processors with kafka.
>>>>>> I use PutKafka to puch avro generated flow from ExecuteSQL.
>>>>>> I verify that the topic created in Kafka is alimented with data using
>>>>>> kafka-console-consumer bash command.
>>>>>> Then I try to use getKafka processor to pull all the Data stored in
>>>>>> the kafka topic and store all that data in a local File using the
>>>>>> PutFile processor. I realize that the getKafka processor don't pull
>>>>>> the topic content.
>>>>>>
>>>>>> I verify if all is ok with kafka-avro-console-consumer (confluent
>>>>>> command), I get this exception :
>>>>>> org.apache.kafka.common.errors.SerializationException: Error
>>>>>> deserializing Avro message for id -1
>>>>>> Caused by: org.apache.kafka.common.errors.SerializationException:
>>>>>> Unknown magic byte!
>>>>>>
>>>>>> I tested also getKafka from a basic topic format and it doesn't work for me.
>>>>>>
>>>>>> So I have some questions :
>>>>>> How can I deal with avro format using getKafka anf putKafka processor ?
>>>>>> Any idea to resolve getKafka processor problem to pull content from a
>>>>>> kafka Topic ?
>>>>>>
>>>>>>
>>>>>> Thank you in advance
>>>>>>
>>>>>> A. Karray
>>>>>
>>>>>
Re: putKafka processor with Avro Format
Posted by Joe Witt <jo...@gmail.com>.
Ali - I can confirm that all formats are supported for Put/Get Kafka
processors. The interaction with Kafka as far as the messages are
concerned is just 'bytes'. The reason it isn't working well is due to
the bug I'm mentioning and that is a NiFi bug (not Kafka). It is
fixed and we'll have a release for you asap but if you want to build
that branch then you're in great shape right away.
Thanks
On Wed, Apr 6, 2016 at 10:24 AM, Ali Karray <c...@karray.net> wrote:
> I see the patch in Jira.
> You confirm that avro format is supported for putKafka and getKafka processor ?
>
> Thanks
> Ali
>
> On Wed, Apr 6, 2016 at 4:14 PM, Joe Witt <jo...@gmail.com> wrote:
>> Ali
>>
>> It is very possible you're being bitten by a bug we have with
>> PutKafka. It is fixed in this branch which we're in the release
>> motions for right now.
>>
>> https://github.com/apache/nifi/tree/support/nifi-0.6.x
>>
>> Thanks
>> Joe
>>
>> On Wed, Apr 6, 2016 at 10:07 AM, Ali Karray <c...@karray.net> wrote:
>>> I'm using the last version 0.6 of Nifi and the 0.9 kafka version
>>> When I use plain csv files, all is working.
>>>
>>> Thanks,
>>> Ali
>>>
>>>
>>> On Wed, Apr 6, 2016 at 3:23 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> What version of Apache NiFi are you using?
>>>>
>>>> Also, lets take Avro and kafka-avro-console-consumer out of the mix for a minute, what happens if you used PutKafka to send some plain text to Kafka, and then tried to consumer from that topic with GetKafka?
>>>> You mentioned this didn't work, but what happened? was there an error message? did it just receive 0 messages from Kafka?
>>>>
>>>> Thanks,
>>>>
>>>> Bryan
>>>>
>>>> On Wed, Apr 6, 2016 at 6:10 AM, Ali Karray <c...@karray.net> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I'm exploring integration possibility of nifi processors with kafka.
>>>>> I use PutKafka to puch avro generated flow from ExecuteSQL.
>>>>> I verify that the topic created in Kafka is alimented with data using
>>>>> kafka-console-consumer bash command.
>>>>> Then I try to use getKafka processor to pull all the Data stored in
>>>>> the kafka topic and store all that data in a local File using the
>>>>> PutFile processor. I realize that the getKafka processor don't pull
>>>>> the topic content.
>>>>>
>>>>> I verify if all is ok with kafka-avro-console-consumer (confluent
>>>>> command), I get this exception :
>>>>> org.apache.kafka.common.errors.SerializationException: Error
>>>>> deserializing Avro message for id -1
>>>>> Caused by: org.apache.kafka.common.errors.SerializationException:
>>>>> Unknown magic byte!
>>>>>
>>>>> I tested also getKafka from a basic topic format and it doesn't work for me.
>>>>>
>>>>> So I have some questions :
>>>>> How can I deal with avro format using getKafka anf putKafka processor ?
>>>>> Any idea to resolve getKafka processor problem to pull content from a
>>>>> kafka Topic ?
>>>>>
>>>>>
>>>>> Thank you in advance
>>>>>
>>>>> A. Karray
>>>>
>>>>
Re: putKafka processor with Avro Format
Posted by Ali Karray <c...@karray.net>.
I see the patch in Jira.
You confirm that avro format is supported for putKafka and getKafka processor ?
Thanks
Ali
On Wed, Apr 6, 2016 at 4:14 PM, Joe Witt <jo...@gmail.com> wrote:
> Ali
>
> It is very possible you're being bitten by a bug we have with
> PutKafka. It is fixed in this branch which we're in the release
> motions for right now.
>
> https://github.com/apache/nifi/tree/support/nifi-0.6.x
>
> Thanks
> Joe
>
> On Wed, Apr 6, 2016 at 10:07 AM, Ali Karray <c...@karray.net> wrote:
>> I'm using the last version 0.6 of Nifi and the 0.9 kafka version
>> When I use plain csv files, all is working.
>>
>> Thanks,
>> Ali
>>
>>
>> On Wed, Apr 6, 2016 at 3:23 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> What version of Apache NiFi are you using?
>>>
>>> Also, lets take Avro and kafka-avro-console-consumer out of the mix for a minute, what happens if you used PutKafka to send some plain text to Kafka, and then tried to consumer from that topic with GetKafka?
>>> You mentioned this didn't work, but what happened? was there an error message? did it just receive 0 messages from Kafka?
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>> On Wed, Apr 6, 2016 at 6:10 AM, Ali Karray <c...@karray.net> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm exploring integration possibility of nifi processors with kafka.
>>>> I use PutKafka to puch avro generated flow from ExecuteSQL.
>>>> I verify that the topic created in Kafka is alimented with data using
>>>> kafka-console-consumer bash command.
>>>> Then I try to use getKafka processor to pull all the Data stored in
>>>> the kafka topic and store all that data in a local File using the
>>>> PutFile processor. I realize that the getKafka processor don't pull
>>>> the topic content.
>>>>
>>>> I verify if all is ok with kafka-avro-console-consumer (confluent
>>>> command), I get this exception :
>>>> org.apache.kafka.common.errors.SerializationException: Error
>>>> deserializing Avro message for id -1
>>>> Caused by: org.apache.kafka.common.errors.SerializationException:
>>>> Unknown magic byte!
>>>>
>>>> I tested also getKafka from a basic topic format and it doesn't work for me.
>>>>
>>>> So I have some questions :
>>>> How can I deal with avro format using getKafka anf putKafka processor ?
>>>> Any idea to resolve getKafka processor problem to pull content from a
>>>> kafka Topic ?
>>>>
>>>>
>>>> Thank you in advance
>>>>
>>>> A. Karray
>>>
>>>
Re: putKafka processor with Avro Format
Posted by Joe Witt <jo...@gmail.com>.
Ali
It is very possible you're being bitten by a bug we have with
PutKafka. It is fixed in this branch which we're in the release
motions for right now.
https://github.com/apache/nifi/tree/support/nifi-0.6.x
Thanks
Joe
On Wed, Apr 6, 2016 at 10:07 AM, Ali Karray <c...@karray.net> wrote:
> I'm using the last version 0.6 of Nifi and the 0.9 kafka version
> When I use plain csv files, all is working.
>
> Thanks,
> Ali
>
>
> On Wed, Apr 6, 2016 at 3:23 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> Hello,
>>
>> What version of Apache NiFi are you using?
>>
>> Also, lets take Avro and kafka-avro-console-consumer out of the mix for a minute, what happens if you used PutKafka to send some plain text to Kafka, and then tried to consumer from that topic with GetKafka?
>> You mentioned this didn't work, but what happened? was there an error message? did it just receive 0 messages from Kafka?
>>
>> Thanks,
>>
>> Bryan
>>
>> On Wed, Apr 6, 2016 at 6:10 AM, Ali Karray <c...@karray.net> wrote:
>>>
>>> Hi,
>>>
>>> I'm exploring integration possibility of nifi processors with kafka.
>>> I use PutKafka to puch avro generated flow from ExecuteSQL.
>>> I verify that the topic created in Kafka is alimented with data using
>>> kafka-console-consumer bash command.
>>> Then I try to use getKafka processor to pull all the Data stored in
>>> the kafka topic and store all that data in a local File using the
>>> PutFile processor. I realize that the getKafka processor don't pull
>>> the topic content.
>>>
>>> I verify if all is ok with kafka-avro-console-consumer (confluent
>>> command), I get this exception :
>>> org.apache.kafka.common.errors.SerializationException: Error
>>> deserializing Avro message for id -1
>>> Caused by: org.apache.kafka.common.errors.SerializationException:
>>> Unknown magic byte!
>>>
>>> I tested also getKafka from a basic topic format and it doesn't work for me.
>>>
>>> So I have some questions :
>>> How can I deal with avro format using getKafka anf putKafka processor ?
>>> Any idea to resolve getKafka processor problem to pull content from a
>>> kafka Topic ?
>>>
>>>
>>> Thank you in advance
>>>
>>> A. Karray
>>
>>
Re: putKafka processor with Avro Format
Posted by Ali Karray <c...@karray.net>.
I'm using the last version 0.6 of Nifi and the 0.9 kafka version
When I use plain csv files, all is working.
Thanks,
Ali
On Wed, Apr 6, 2016 at 3:23 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> Hello,
>
> What version of Apache NiFi are you using?
>
> Also, lets take Avro and kafka-avro-console-consumer out of the mix for a minute, what happens if you used PutKafka to send some plain text to Kafka, and then tried to consumer from that topic with GetKafka?
> You mentioned this didn't work, but what happened? was there an error message? did it just receive 0 messages from Kafka?
>
> Thanks,
>
> Bryan
>
> On Wed, Apr 6, 2016 at 6:10 AM, Ali Karray <c...@karray.net> wrote:
>>
>> Hi,
>>
>> I'm exploring integration possibility of nifi processors with kafka.
>> I use PutKafka to puch avro generated flow from ExecuteSQL.
>> I verify that the topic created in Kafka is alimented with data using
>> kafka-console-consumer bash command.
>> Then I try to use getKafka processor to pull all the Data stored in
>> the kafka topic and store all that data in a local File using the
>> PutFile processor. I realize that the getKafka processor don't pull
>> the topic content.
>>
>> I verify if all is ok with kafka-avro-console-consumer (confluent
>> command), I get this exception :
>> org.apache.kafka.common.errors.SerializationException: Error
>> deserializing Avro message for id -1
>> Caused by: org.apache.kafka.common.errors.SerializationException:
>> Unknown magic byte!
>>
>> I tested also getKafka from a basic topic format and it doesn't work for me.
>>
>> So I have some questions :
>> How can I deal with avro format using getKafka anf putKafka processor ?
>> Any idea to resolve getKafka processor problem to pull content from a
>> kafka Topic ?
>>
>>
>> Thank you in advance
>>
>> A. Karray
>
>
Re: putKafka processor with Avro Format
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
What version of Apache NiFi are you using?
Also, lets take Avro and kafka-avro-console-consumer out of the mix for a
minute, what happens if you used PutKafka to send some plain text to Kafka,
and then tried to consumer from that topic with GetKafka?
You mentioned this didn't work, but what happened? was there an error
message? did it just receive 0 messages from Kafka?
Thanks,
Bryan
On Wed, Apr 6, 2016 at 6:10 AM, Ali Karray <c...@karray.net> wrote:
> Hi,
>
> I'm exploring integration possibility of nifi processors with kafka.
> I use PutKafka to puch avro generated flow from ExecuteSQL.
> I verify that the topic created in Kafka is alimented with data using
> kafka-console-consumer bash command.
> Then I try to use getKafka processor to pull all the Data stored in
> the kafka topic and store all that data in a local File using the
> PutFile processor. I realize that the getKafka processor don't pull
> the topic content.
>
> I verify if all is ok with kafka-avro-console-consumer (confluent
> command), I get this exception :
> org.apache.kafka.common.errors.SerializationException: Error
> deserializing Avro message for id -1
> Caused by: org.apache.kafka.common.errors.SerializationException:
> Unknown magic byte!
>
> I tested also getKafka from a basic topic format and it doesn't work for
> me.
>
> So I have some questions :
> How can I deal with avro format using getKafka anf putKafka processor ?
> Any idea to resolve getKafka processor problem to pull content from a
> kafka Topic ?
>
>
> Thank you in advance
>
> A. Karray
>