You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by amir bahmanyari <am...@yahoo.com> on 2016/07/26 20:13:25 UTC
Avoid reading duplicates from KafkaIO!!!
Hi Colleagues,I run the same Beam Flink app code without reading from KafkaIO, and I get accurate results (perfect matched records).When run exact same code but read the records from KafkaIO, the number of matched records exceeds the accurate expected number at the end.This tells me that I MAY BE receiving duplicates from KafkaIO causing my matched counter to increase superficially.Could this be a possibility with KafkaIO function & how it interacts with Kafka server?If so, how can I prevent KafkaIO from duplicate readings?ThanksAmir-
Re: Avoid reading duplicates from KafkaIO!!!
Posted by amir bahmanyari <am...@yahoo.com>.
Ok. The state persistence concurrency issue Imentioned below in Flink Cluster is finally resolved.I tried everything with Redis. There is a concurrency/locking issue which I couldnt figure it out.Replaced Redis with simple Java 1.7 ConcurrentHashMaps only for key "state" values...magic!! J
And you get a bonus as well: It gets REALLY Fast...Thanks+regardsAmir- From: Raghu Angadi <ra...@google.com>
To: amir bahmanyari <am...@yahoo.com>
Cc: "user@beam.incubator.apache.org" <us...@beam.incubator.apache.org>
Sent: Wednesday, July 27, 2016 10:16 AM
Subject: Re: Avoid reading duplicates from KafkaIO!!!
Thanks for reporting your findings back.
I don't know many details of your app, but I strongly suggest you do as many aggregations as possible in Dataflow itself rather than using an external storage. This will be lot more scalable and more importantly much simpler to manage and make changes in future.
Raghu.
On Wed, Jul 27, 2016 at 10:06 AM, amir bahmanyari <am...@yahoo.com> wrote:
Hi Raghu,Kafka is ok sending one record at a time. I proved it.The issue is maintaining concurrency in Redis maps thats being accessed+modified by parallel Pipelines at runtime.And the variability of the apparent duplication is due to unpredictable number of the parallel pipelines accessing a shared Redis map during each separate run.Thats my challenge at the moment.Thanks for your help.Amir
From: Raghu Angadi <ra...@google.com>
To: user@beam.incubator.apache.org; amir bahmanyari <am...@yahoo.com>
Sent: Tuesday, July 26, 2016 2:34 PM
Subject: Re: Avoid reading duplicates from KafkaIO!!!
On Tue, Jul 26, 2016 at 2:29 PM, amir bahmanyari <am...@yahoo.com> wrote:
I know this is not Kafka forum ,but could Kafka be sending redundant records?
no, equally unlikely. You could verify if you pushed duplicates to Kafka by reading directly from kafka console consumer.
e.g. : bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic my_topi --from-beginning | grep ....
Re: Avoid reading duplicates from KafkaIO!!!
Posted by Raghu Angadi <ra...@google.com>.
Thanks for reporting your findings back.
I don't know many details of your app, but I strongly suggest you do as
many aggregations as possible in Dataflow itself rather than using an
external storage. This will be lot more scalable and more importantly much
simpler to manage and make changes in future.
Raghu.
On Wed, Jul 27, 2016 at 10:06 AM, amir bahmanyari <am...@yahoo.com>
wrote:
> Hi Raghu,
> Kafka is ok sending one record at a time. I proved it.
> The issue is maintaining concurrency in Redis maps thats being
> accessed+modified by parallel Pipelines at runtime.
> And the variability of the apparent duplication is due to unpredictable
> number of the parallel pipelines accessing a shared Redis map during each
> separate run.
> Thats my challenge at the moment.
> Thanks for your help.
> Amir
>
>
> ------------------------------
> *From:* Raghu Angadi <ra...@google.com>
> *To:* user@beam.incubator.apache.org; amir bahmanyari <am...@yahoo.com>
>
> *Sent:* Tuesday, July 26, 2016 2:34 PM
> *Subject:* Re: Avoid reading duplicates from KafkaIO!!!
>
>
> On Tue, Jul 26, 2016 at 2:29 PM, amir bahmanyari <am...@yahoo.com>
> wrote:
>
> I know this is not Kafka forum ,but could Kafka be sending redundant
> records?
>
>
> no, equally unlikely. You could verify if you pushed duplicates to Kafka
> by reading directly from kafka console consumer.
>
> e.g. : bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic
> my_topi --from-beginning | grep ....
>
>
>
Re: Avoid reading duplicates from KafkaIO!!!
Posted by Lukasz Cwik <lc...@google.com>.
Redis supports check and set operations, and also has some functions for
primitives like increment which could help solve your issue.
On Wed, Jul 27, 2016 at 1:06 PM, amir bahmanyari <am...@yahoo.com>
wrote:
> Hi Raghu,
> Kafka is ok sending one record at a time. I proved it.
> The issue is maintaining concurrency in Redis maps thats being
> accessed+modified by parallel Pipelines at runtime.
> And the variability of the apparent duplication is due to unpredictable
> number of the parallel pipelines accessing a shared Redis map during each
> separate run.
> Thats my challenge at the moment.
> Thanks for your help.
> Amir
>
>
> ------------------------------
> *From:* Raghu Angadi <ra...@google.com>
> *To:* user@beam.incubator.apache.org; amir bahmanyari <am...@yahoo.com>
>
> *Sent:* Tuesday, July 26, 2016 2:34 PM
> *Subject:* Re: Avoid reading duplicates from KafkaIO!!!
>
>
> On Tue, Jul 26, 2016 at 2:29 PM, amir bahmanyari <am...@yahoo.com>
> wrote:
>
> I know this is not Kafka forum ,but could Kafka be sending redundant
> records?
>
>
> no, equally unlikely. You could verify if you pushed duplicates to Kafka
> by reading directly from kafka console consumer.
>
> e.g. : bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic
> my_topi --from-beginning | grep ....
>
>
>
Re: Avoid reading duplicates from KafkaIO!!!
Posted by amir bahmanyari <am...@yahoo.com>.
Hi Raghu,Kafka is ok sending one record at a time. I proved it.The issue is maintaining concurrency in Redis maps thats being accessed+modified by parallel Pipelines at runtime.And the variability of the apparent duplication is due to unpredictable number of the parallel pipelines accessing a shared Redis map during each separate run.Thats my challenge at the moment.Thanks for your help.Amir
From: Raghu Angadi <ra...@google.com>
To: user@beam.incubator.apache.org; amir bahmanyari <am...@yahoo.com>
Sent: Tuesday, July 26, 2016 2:34 PM
Subject: Re: Avoid reading duplicates from KafkaIO!!!
On Tue, Jul 26, 2016 at 2:29 PM, amir bahmanyari <am...@yahoo.com> wrote:
I know this is not Kafka forum ,but could Kafka be sending redundant records?
no, equally unlikely. You could verify if you pushed duplicates to Kafka by reading directly from kafka console consumer.
e.g. : bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic my_topi --from-beginning | grep ....
Re: Avoid reading duplicates from KafkaIO!!!
Posted by Raghu Angadi <ra...@google.com>.
On Tue, Jul 26, 2016 at 2:29 PM, amir bahmanyari <am...@yahoo.com>
wrote:
> I know this is not Kafka forum ,but could Kafka be sending redundant
> records?
>
no, equally unlikely. You could verify if you pushed duplicates to Kafka by
reading directly from kafka console consumer.
e.g. : bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic
my_topi --from-beginning | grep ....
Re: Avoid reading duplicates from KafkaIO!!!
Posted by amir bahmanyari <am...@yahoo.com>.
Thanks Raghu..I know this is not Kafka forum ,but could Kafka be sending redundant records?I need to set a counter outside of my code in Redis and check it.Thanks
From: Raghu Angadi <ra...@google.com>
To: user@beam.incubator.apache.org; amir bahmanyari <am...@yahoo.com>
Sent: Tuesday, July 26, 2016 1:28 PM
Subject: Re: Avoid reading duplicates from KafkaIO!!!
On Tue, Jul 26, 2016 at 1:13 PM, amir bahmanyari <am...@yahoo.com> wrote:
Could this be a possibility with KafkaIO function & how it interacts with Kafka server?
No. Not expected.It should be a bug if observe duplicates. did you have autoscaling enabled (not that I expect a bug here)? Did the number of workers change?
If so, how can I prevent KafkaIO from duplicate readings?
Re: Avoid reading duplicates from KafkaIO!!!
Posted by Raghu Angadi <ra...@google.com>.
On Tue, Jul 26, 2016 at 1:13 PM, amir bahmanyari <am...@yahoo.com>
wrote:
> Could this be a possibility with KafkaIO function & how it interacts with
> Kafka server?
>
No. Not expected.
It should be a bug if observe duplicates. did you have autoscaling enabled
(not that I expect a bug here)? Did the number of workers change?
> If so, how can I prevent KafkaIO from duplicate readings?
>