You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Teena Kappen // BPRISE <te...@bprise.com> on 2018/01/18 08:53:28 UTC
Multiple Elasticsearch sinks not working in Flink
Hi,
I am running flink 1.4 in single node. My job has two Kafka consumers reading from separate topics. After fetching the data, the job writes it to two separate Elasticsearch sinks. So the process is like this
KafkaTopic1 -> Kafkaconsumer1 -> create output record -> Elasticsearchsink1
KafkaTopic2 -> Kafkaconsumer2 -> create output record -> Elasticsearchsink2
Both the streams and their processing are completely unrelated. The first sink works as expected and it writes the output for all input records. The second sink writes to Elasticsearch only once and after that it stops writing to Elasticsearch even if there is more data that gets fed into Kafka. Sometimes, it does not even write once. We tested this in two other jobs and the same issue is there in all of them.
I have attached a sample code I had created to illustrate the issue. We are using Elasticsearch version 5.6.4 and hence the dependency used is 'flink-connector-elasticsearch5_2.11'.
Regards,
Teena
Re: Multiple Elasticsearch sinks not working in Flink
Posted by Fabian Hueske <fh...@gmail.com>.
Great, thanks for the feedback!
Best, Fabian
2018-02-03 9:37 GMT+01:00 Teena Kappen // BPRISE <te...@bprise.com>:
> Hi Fabian,
>
>
>
> We tried the fix that was merged and the sinks are working correctly now.
> Thank you for resolving the issue.
>
>
>
> Regards,
>
> Teena
>
>
>
> *From:* Teena Kappen // BPRISE
> *Sent:* 01 February 2018 19:12
> *To:* Fabian Hueske <fh...@gmail.com>; Stephan Ewen <se...@apache.org>
> *Cc:* Timo Walther <tw...@apache.org>; user@flink.apache.org
> *Subject:* RE: Multiple Elasticsearch sinks not working in Flink
>
>
>
> @Fabian: I will run the code with the Git repo source and let you know the
> results.
>
>
>
> @Stephan: Sorry I missed the email from you somehow. I understand from the
> JIRA link that you already have the answer for this. Yet I tried using two
> separate config map objects in my code and that resolved the issue. Both
> the sinks wrote into Elasticsearch as expected.
>
>
>
> Thank you for taking this up. I will report back on the test results soon.
>
>
>
> Regards,
>
> Teena
>
>
>
> *From:* Fabian Hueske [mailto:fhueske@gmail.com <fh...@gmail.com>]
> *Sent:* 31 January 2018 19:41
> *To:* Stephan Ewen <se...@apache.org>
> *Cc:* Teena Kappen // BPRISE <te...@bprise.com>; Timo Walther <
> twalthr@apache.org>; user@flink.apache.org
> *Subject:* Re: Multiple Elasticsearch sinks not working in Flink
>
>
>
> Hi Teena,
>
> a potential fix for the issue has been merged: https://issues.apache.org/
> jira/browse/FLINK-8489
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C28211d80416345c8fecf08d568b49a35%7C9a9bc2522d894aabb48c9884315adfa9%7C0%7C0%7C636530047252331606&sdata=4VLZXSwvjt9dJtEQi8VvyhtrZ51JMDFJse5NRs5bKcQ%3D&reserved=0>
>
> It would be great if you could check if that fixes the problem and report
> back.
>
> Thank you,
>
> Fabian
>
>
>
> 2018-01-23 20:04 GMT+01:00 Stephan Ewen <se...@apache.org>:
>
> As mentioned in the issue, please check if using two different config map
> objects solves the issue.
>
>
>
> On Tue, Jan 23, 2018 at 1:32 PM, Teena Kappen // BPRISE <
> teena.kappen@bprise.com> wrote:
>
> Thanks Fabian. I will go through it and add info if required.
>
>
>
> *From:* Fabian Hueske [mailto:fhueske@gmail.com]
> *Sent:* 23 January 2018 15:20
> *To:* Teena Kappen // BPRISE <te...@bprise.com>
> *Cc:* Timo Walther <tw...@apache.org>; user@flink.apache.org
>
>
> *Subject:* Re: Multiple Elasticsearch sinks not working in Flink
>
>
>
> Hi Teena,
>
> I created FLINK-8489 [1] to track the issue.
>
> Please have a look and add information that might be relevant.
>
>
>
> Best, Fabian
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-8489
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C13d8e8f5a69346af547c08d56246ca5d%7Cb7cfea21c1ff4c0a8e2ee7ccd5d8f54b%7C0%7C0%7C636522978545765383&sdata=ZSiaFbWWlnKEgAPZMdqE%2F%2FmdiFixS2T8jDqjAtt%2Btzs%3D&reserved=0>
>
>
>
> 2018-01-18 14:16 GMT+01:00 Teena Kappen // BPRISE <teena.kappen@bprise.com
> >:
>
> Hi Timo,
>
>
>
> It works fine when the second sink is a Cassandra Sink. The data gets read
> from KafkaTopic2 and it gets written to Cassandra as expected.
>
>
>
> Regards,
>
> Teena
>
>
>
> *From:* Timo Walther [mailto:twalthr@apache.org]
> *Sent:* 18 January 2018 18:41
> *To:* user@flink.apache.org
> *Subject:* Re: Multiple Elasticsearch sinks not working in Flink
>
>
>
> Hi Teena,
>
> what happens if you replace the second sink with a non-ElasticSearchSink?
> Is there the same result? Is the data read from the KafkaTopic2?
>
> We should determine which system is the bottleneck.
>
> Regards,
> Timo
>
>
> Am 1/18/18 um 9:53 AM schrieb Teena Kappen // BPRISE:
>
> Hi,
>
>
>
> I am running flink 1.4 in single node. My job has two Kafka consumers
> reading from separate topics. After fetching the data, the job writes it to
> two separate Elasticsearch sinks. So the process is like this
>
>
>
> KafkaTopic1 -> Kafkaconsumer1 -> create output record ->
> Elasticsearchsink1
>
> KafkaTopic2 -> Kafkaconsumer2 -> create output record -> Elasticsearchsink2
>
>
>
> Both the streams and their processing are completely unrelated. The first
> sink works as expected and it writes the output for all input records. The
> second sink writes to Elasticsearch only once and after that it stops
> writing to Elasticsearch even if there is more data that gets fed into
> Kafka. Sometimes, it does not even write once. We tested this in two other
> jobs and the same issue is there in all of them.
>
>
>
> I have attached a sample code I had created to illustrate the issue. We
> are using Elasticsearch version 5.6.4 and hence the dependency used is
> ‘flink-connector-elasticsearch5_2.11’.
>
>
>
> Regards,
>
> Teena
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
RE: Multiple Elasticsearch sinks not working in Flink
Posted by Teena Kappen // BPRISE <te...@bprise.com>.
Hi Fabian,
We tried the fix that was merged and the sinks are working correctly now. Thank you for resolving the issue.
Regards,
Teena
From: Teena Kappen // BPRISE
Sent: 01 February 2018 19:12
To: Fabian Hueske <fh...@gmail.com>; Stephan Ewen <se...@apache.org>
Cc: Timo Walther <tw...@apache.org>; user@flink.apache.org
Subject: RE: Multiple Elasticsearch sinks not working in Flink
@Fabian: I will run the code with the Git repo source and let you know the results.
@Stephan: Sorry I missed the email from you somehow. I understand from the JIRA link that you already have the answer for this. Yet I tried using two separate config map objects in my code and that resolved the issue. Both the sinks wrote into Elasticsearch as expected.
Thank you for taking this up. I will report back on the test results soon.
Regards,
Teena
From: Fabian Hueske [mailto:fhueske@gmail.com]
Sent: 31 January 2018 19:41
To: Stephan Ewen <se...@apache.org>>
Cc: Teena Kappen // BPRISE <te...@bprise.com>>; Timo Walther <tw...@apache.org>>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Multiple Elasticsearch sinks not working in Flink
Hi Teena,
a potential fix for the issue has been merged: https://issues.apache.org/jira/browse/FLINK-8489<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C28211d80416345c8fecf08d568b49a35%7C9a9bc2522d894aabb48c9884315adfa9%7C0%7C0%7C636530047252331606&sdata=4VLZXSwvjt9dJtEQi8VvyhtrZ51JMDFJse5NRs5bKcQ%3D&reserved=0>
It would be great if you could check if that fixes the problem and report back.
Thank you,
Fabian
2018-01-23 20:04 GMT+01:00 Stephan Ewen <se...@apache.org>>:
As mentioned in the issue, please check if using two different config map objects solves the issue.
On Tue, Jan 23, 2018 at 1:32 PM, Teena Kappen // BPRISE <te...@bprise.com>> wrote:
Thanks Fabian. I will go through it and add info if required.
From: Fabian Hueske [mailto:fhueske@gmail.com<ma...@gmail.com>]
Sent: 23 January 2018 15:20
To: Teena Kappen // BPRISE <te...@bprise.com>>
Cc: Timo Walther <tw...@apache.org>>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Multiple Elasticsearch sinks not working in Flink
Hi Teena,
I created FLINK-8489 [1] to track the issue.
Please have a look and add information that might be relevant.
Best, Fabian
[1] https://issues.apache.org/jira/browse/FLINK-8489<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C13d8e8f5a69346af547c08d56246ca5d%7Cb7cfea21c1ff4c0a8e2ee7ccd5d8f54b%7C0%7C0%7C636522978545765383&sdata=ZSiaFbWWlnKEgAPZMdqE%2F%2FmdiFixS2T8jDqjAtt%2Btzs%3D&reserved=0>
2018-01-18 14:16 GMT+01:00 Teena Kappen // BPRISE <te...@bprise.com>>:
Hi Timo,
It works fine when the second sink is a Cassandra Sink. The data gets read from KafkaTopic2 and it gets written to Cassandra as expected.
Regards,
Teena
From: Timo Walther [mailto:twalthr@apache.org<ma...@apache.org>]
Sent: 18 January 2018 18:41
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Multiple Elasticsearch sinks not working in Flink
Hi Teena,
what happens if you replace the second sink with a non-ElasticSearchSink? Is there the same result? Is the data read from the KafkaTopic2?
We should determine which system is the bottleneck.
Regards,
Timo
Am 1/18/18 um 9:53 AM schrieb Teena Kappen // BPRISE:
Hi,
I am running flink 1.4 in single node. My job has two Kafka consumers reading from separate topics. After fetching the data, the job writes it to two separate Elasticsearch sinks. So the process is like this
KafkaTopic1 -> Kafkaconsumer1 -> create output record -> Elasticsearchsink1
KafkaTopic2 -> Kafkaconsumer2 -> create output record -> Elasticsearchsink2
Both the streams and their processing are completely unrelated. The first sink works as expected and it writes the output for all input records. The second sink writes to Elasticsearch only once and after that it stops writing to Elasticsearch even if there is more data that gets fed into Kafka. Sometimes, it does not even write once. We tested this in two other jobs and the same issue is there in all of them.
I have attached a sample code I had created to illustrate the issue. We are using Elasticsearch version 5.6.4 and hence the dependency used is ‘flink-connector-elasticsearch5_2.11’.
Regards,
Teena
RE: Multiple Elasticsearch sinks not working in Flink
Posted by Teena Kappen // BPRISE <te...@bprise.com>.
@Fabian: I will run the code with the Git repo source and let you know the results.
@Stephan: Sorry I missed the email from you somehow. I understand from the JIRA link that you already have the answer for this. Yet I tried using two separate config map objects in my code and that resolved the issue. Both the sinks wrote into Elasticsearch as expected.
Thank you for taking this up. I will report back on the test results soon.
Regards,
Teena
From: Fabian Hueske [mailto:fhueske@gmail.com]
Sent: 31 January 2018 19:41
To: Stephan Ewen <se...@apache.org>
Cc: Teena Kappen // BPRISE <te...@bprise.com>; Timo Walther <tw...@apache.org>; user@flink.apache.org
Subject: Re: Multiple Elasticsearch sinks not working in Flink
Hi Teena,
a potential fix for the issue has been merged: https://issues.apache.org/jira/browse/FLINK-8489<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C28211d80416345c8fecf08d568b49a35%7C9a9bc2522d894aabb48c9884315adfa9%7C0%7C0%7C636530047252331606&sdata=4VLZXSwvjt9dJtEQi8VvyhtrZ51JMDFJse5NRs5bKcQ%3D&reserved=0>
It would be great if you could check if that fixes the problem and report back.
Thank you,
Fabian
2018-01-23 20:04 GMT+01:00 Stephan Ewen <se...@apache.org>>:
As mentioned in the issue, please check if using two different config map objects solves the issue.
On Tue, Jan 23, 2018 at 1:32 PM, Teena Kappen // BPRISE <te...@bprise.com>> wrote:
Thanks Fabian. I will go through it and add info if required.
From: Fabian Hueske [mailto:fhueske@gmail.com<ma...@gmail.com>]
Sent: 23 January 2018 15:20
To: Teena Kappen // BPRISE <te...@bprise.com>>
Cc: Timo Walther <tw...@apache.org>>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Multiple Elasticsearch sinks not working in Flink
Hi Teena,
I created FLINK-8489 [1] to track the issue.
Please have a look and add information that might be relevant.
Best, Fabian
[1] https://issues.apache.org/jira/browse/FLINK-8489<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C13d8e8f5a69346af547c08d56246ca5d%7Cb7cfea21c1ff4c0a8e2ee7ccd5d8f54b%7C0%7C0%7C636522978545765383&sdata=ZSiaFbWWlnKEgAPZMdqE%2F%2FmdiFixS2T8jDqjAtt%2Btzs%3D&reserved=0>
2018-01-18 14:16 GMT+01:00 Teena Kappen // BPRISE <te...@bprise.com>>:
Hi Timo,
It works fine when the second sink is a Cassandra Sink. The data gets read from KafkaTopic2 and it gets written to Cassandra as expected.
Regards,
Teena
From: Timo Walther [mailto:twalthr@apache.org<ma...@apache.org>]
Sent: 18 January 2018 18:41
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Multiple Elasticsearch sinks not working in Flink
Hi Teena,
what happens if you replace the second sink with a non-ElasticSearchSink? Is there the same result? Is the data read from the KafkaTopic2?
We should determine which system is the bottleneck.
Regards,
Timo
Am 1/18/18 um 9:53 AM schrieb Teena Kappen // BPRISE:
Hi,
I am running flink 1.4 in single node. My job has two Kafka consumers reading from separate topics. After fetching the data, the job writes it to two separate Elasticsearch sinks. So the process is like this
KafkaTopic1 -> Kafkaconsumer1 -> create output record -> Elasticsearchsink1
KafkaTopic2 -> Kafkaconsumer2 -> create output record -> Elasticsearchsink2
Both the streams and their processing are completely unrelated. The first sink works as expected and it writes the output for all input records. The second sink writes to Elasticsearch only once and after that it stops writing to Elasticsearch even if there is more data that gets fed into Kafka. Sometimes, it does not even write once. We tested this in two other jobs and the same issue is there in all of them.
I have attached a sample code I had created to illustrate the issue. We are using Elasticsearch version 5.6.4 and hence the dependency used is ‘flink-connector-elasticsearch5_2.11’.
Regards,
Teena
Re: Multiple Elasticsearch sinks not working in Flink
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Teena,
a potential fix for the issue has been merged:
https://issues.apache.org/jira/browse/FLINK-8489
It would be great if you could check if that fixes the problem and report
back.
Thank you,
Fabian
2018-01-23 20:04 GMT+01:00 Stephan Ewen <se...@apache.org>:
> As mentioned in the issue, please check if using two different config map
> objects solves the issue.
>
> On Tue, Jan 23, 2018 at 1:32 PM, Teena Kappen // BPRISE <
> teena.kappen@bprise.com> wrote:
>
>> Thanks Fabian. I will go through it and add info if required.
>>
>>
>>
>> *From:* Fabian Hueske [mailto:fhueske@gmail.com]
>> *Sent:* 23 January 2018 15:20
>> *To:* Teena Kappen // BPRISE <te...@bprise.com>
>> *Cc:* Timo Walther <tw...@apache.org>; user@flink.apache.org
>>
>> *Subject:* Re: Multiple Elasticsearch sinks not working in Flink
>>
>>
>>
>> Hi Teena,
>>
>> I created FLINK-8489 [1] to track the issue.
>>
>> Please have a look and add information that might be relevant.
>>
>>
>>
>> Best, Fabian
>>
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-8489
>> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C13d8e8f5a69346af547c08d56246ca5d%7Cb7cfea21c1ff4c0a8e2ee7ccd5d8f54b%7C0%7C0%7C636522978545765383&sdata=ZSiaFbWWlnKEgAPZMdqE%2F%2FmdiFixS2T8jDqjAtt%2Btzs%3D&reserved=0>
>>
>>
>>
>> 2018-01-18 14:16 GMT+01:00 Teena Kappen // BPRISE <
>> teena.kappen@bprise.com>:
>>
>> Hi Timo,
>>
>>
>>
>> It works fine when the second sink is a Cassandra Sink. The data gets
>> read from KafkaTopic2 and it gets written to Cassandra as expected.
>>
>>
>>
>> Regards,
>>
>> Teena
>>
>>
>>
>> *From:* Timo Walther [mailto:twalthr@apache.org]
>> *Sent:* 18 January 2018 18:41
>> *To:* user@flink.apache.org
>> *Subject:* Re: Multiple Elasticsearch sinks not working in Flink
>>
>>
>>
>> Hi Teena,
>>
>> what happens if you replace the second sink with a non-ElasticSearchSink?
>> Is there the same result? Is the data read from the KafkaTopic2?
>>
>> We should determine which system is the bottleneck.
>>
>> Regards,
>> Timo
>>
>>
>> Am 1/18/18 um 9:53 AM schrieb Teena Kappen // BPRISE:
>>
>> Hi,
>>
>>
>>
>> I am running flink 1.4 in single node. My job has two Kafka consumers
>> reading from separate topics. After fetching the data, the job writes it to
>> two separate Elasticsearch sinks. So the process is like this
>>
>>
>>
>> KafkaTopic1 -> Kafkaconsumer1 -> create output record ->
>> Elasticsearchsink1
>>
>> KafkaTopic2 -> Kafkaconsumer2 -> create output record ->
>> Elasticsearchsink2
>>
>>
>>
>> Both the streams and their processing are completely unrelated. The first
>> sink works as expected and it writes the output for all input records. The
>> second sink writes to Elasticsearch only once and after that it stops
>> writing to Elasticsearch even if there is more data that gets fed into
>> Kafka. Sometimes, it does not even write once. We tested this in two other
>> jobs and the same issue is there in all of them.
>>
>>
>>
>> I have attached a sample code I had created to illustrate the issue. We
>> are using Elasticsearch version 5.6.4 and hence the dependency used is
>> ‘flink-connector-elasticsearch5_2.11’.
>>
>>
>>
>> Regards,
>>
>> Teena
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
Re: Multiple Elasticsearch sinks not working in Flink
Posted by Stephan Ewen <se...@apache.org>.
As mentioned in the issue, please check if using two different config map
objects solves the issue.
On Tue, Jan 23, 2018 at 1:32 PM, Teena Kappen // BPRISE <
teena.kappen@bprise.com> wrote:
> Thanks Fabian. I will go through it and add info if required.
>
>
>
> *From:* Fabian Hueske [mailto:fhueske@gmail.com]
> *Sent:* 23 January 2018 15:20
> *To:* Teena Kappen // BPRISE <te...@bprise.com>
> *Cc:* Timo Walther <tw...@apache.org>; user@flink.apache.org
>
> *Subject:* Re: Multiple Elasticsearch sinks not working in Flink
>
>
>
> Hi Teena,
>
> I created FLINK-8489 [1] to track the issue.
>
> Please have a look and add information that might be relevant.
>
>
>
> Best, Fabian
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-8489
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C13d8e8f5a69346af547c08d56246ca5d%7Cb7cfea21c1ff4c0a8e2ee7ccd5d8f54b%7C0%7C0%7C636522978545765383&sdata=ZSiaFbWWlnKEgAPZMdqE%2F%2FmdiFixS2T8jDqjAtt%2Btzs%3D&reserved=0>
>
>
>
> 2018-01-18 14:16 GMT+01:00 Teena Kappen // BPRISE <teena.kappen@bprise.com
> >:
>
> Hi Timo,
>
>
>
> It works fine when the second sink is a Cassandra Sink. The data gets read
> from KafkaTopic2 and it gets written to Cassandra as expected.
>
>
>
> Regards,
>
> Teena
>
>
>
> *From:* Timo Walther [mailto:twalthr@apache.org]
> *Sent:* 18 January 2018 18:41
> *To:* user@flink.apache.org
> *Subject:* Re: Multiple Elasticsearch sinks not working in Flink
>
>
>
> Hi Teena,
>
> what happens if you replace the second sink with a non-ElasticSearchSink?
> Is there the same result? Is the data read from the KafkaTopic2?
>
> We should determine which system is the bottleneck.
>
> Regards,
> Timo
>
>
> Am 1/18/18 um 9:53 AM schrieb Teena Kappen // BPRISE:
>
> Hi,
>
>
>
> I am running flink 1.4 in single node. My job has two Kafka consumers
> reading from separate topics. After fetching the data, the job writes it to
> two separate Elasticsearch sinks. So the process is like this
>
>
>
> KafkaTopic1 -> Kafkaconsumer1 -> create output record ->
> Elasticsearchsink1
>
> KafkaTopic2 -> Kafkaconsumer2 -> create output record -> Elasticsearchsink2
>
>
>
> Both the streams and their processing are completely unrelated. The first
> sink works as expected and it writes the output for all input records. The
> second sink writes to Elasticsearch only once and after that it stops
> writing to Elasticsearch even if there is more data that gets fed into
> Kafka. Sometimes, it does not even write once. We tested this in two other
> jobs and the same issue is there in all of them.
>
>
>
> I have attached a sample code I had created to illustrate the issue. We
> are using Elasticsearch version 5.6.4 and hence the dependency used is
> ‘flink-connector-elasticsearch5_2.11’.
>
>
>
> Regards,
>
> Teena
>
>
>
>
>
>
>
>
>
>
>
>
>
RE: Multiple Elasticsearch sinks not working in Flink
Posted by Teena Kappen // BPRISE <te...@bprise.com>.
Thanks Fabian. I will go through it and add info if required.
From: Fabian Hueske [mailto:fhueske@gmail.com]
Sent: 23 January 2018 15:20
To: Teena Kappen // BPRISE <te...@bprise.com>
Cc: Timo Walther <tw...@apache.org>; user@flink.apache.org
Subject: Re: Multiple Elasticsearch sinks not working in Flink
Hi Teena,
I created FLINK-8489 [1] to track the issue.
Please have a look and add information that might be relevant.
Best, Fabian
[1] https://issues.apache.org/jira/browse/FLINK-8489<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C13d8e8f5a69346af547c08d56246ca5d%7Cb7cfea21c1ff4c0a8e2ee7ccd5d8f54b%7C0%7C0%7C636522978545765383&sdata=ZSiaFbWWlnKEgAPZMdqE%2F%2FmdiFixS2T8jDqjAtt%2Btzs%3D&reserved=0>
2018-01-18 14:16 GMT+01:00 Teena Kappen // BPRISE <te...@bprise.com>>:
Hi Timo,
It works fine when the second sink is a Cassandra Sink. The data gets read from KafkaTopic2 and it gets written to Cassandra as expected.
Regards,
Teena
From: Timo Walther [mailto:twalthr@apache.org<ma...@apache.org>]
Sent: 18 January 2018 18:41
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Multiple Elasticsearch sinks not working in Flink
Hi Teena,
what happens if you replace the second sink with a non-ElasticSearchSink? Is there the same result? Is the data read from the KafkaTopic2?
We should determine which system is the bottleneck.
Regards,
Timo
Am 1/18/18 um 9:53 AM schrieb Teena Kappen // BPRISE:
Hi,
I am running flink 1.4 in single node. My job has two Kafka consumers reading from separate topics. After fetching the data, the job writes it to two separate Elasticsearch sinks. So the process is like this
KafkaTopic1 -> Kafkaconsumer1 -> create output record -> Elasticsearchsink1
KafkaTopic2 -> Kafkaconsumer2 -> create output record -> Elasticsearchsink2
Both the streams and their processing are completely unrelated. The first sink works as expected and it writes the output for all input records. The second sink writes to Elasticsearch only once and after that it stops writing to Elasticsearch even if there is more data that gets fed into Kafka. Sometimes, it does not even write once. We tested this in two other jobs and the same issue is there in all of them.
I have attached a sample code I had created to illustrate the issue. We are using Elasticsearch version 5.6.4 and hence the dependency used is ‘flink-connector-elasticsearch5_2.11’.
Regards,
Teena
Re: Multiple Elasticsearch sinks not working in Flink
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Teena,
I created FLINK-8489 [1] to track the issue.
Please have a look and add information that might be relevant.
Best, Fabian
[1] https://issues.apache.org/jira/browse/FLINK-8489
2018-01-18 14:16 GMT+01:00 Teena Kappen // BPRISE <te...@bprise.com>:
> Hi Timo,
>
>
>
> It works fine when the second sink is a Cassandra Sink. The data gets read
> from KafkaTopic2 and it gets written to Cassandra as expected.
>
>
>
> Regards,
>
> Teena
>
>
>
> *From:* Timo Walther [mailto:twalthr@apache.org]
> *Sent:* 18 January 2018 18:41
> *To:* user@flink.apache.org
> *Subject:* Re: Multiple Elasticsearch sinks not working in Flink
>
>
>
> Hi Teena,
>
> what happens if you replace the second sink with a non-ElasticSearchSink?
> Is there the same result? Is the data read from the KafkaTopic2?
>
> We should determine which system is the bottleneck.
>
> Regards,
> Timo
>
>
> Am 1/18/18 um 9:53 AM schrieb Teena Kappen // BPRISE:
>
> Hi,
>
>
>
> I am running flink 1.4 in single node. My job has two Kafka consumers
> reading from separate topics. After fetching the data, the job writes it to
> two separate Elasticsearch sinks. So the process is like this
>
>
>
> KafkaTopic1 -> Kafkaconsumer1 -> create output record ->
> Elasticsearchsink1
>
> KafkaTopic2 -> Kafkaconsumer2 -> create output record -> Elasticsearchsink2
>
>
>
> Both the streams and their processing are completely unrelated. The first
> sink works as expected and it writes the output for all input records. The
> second sink writes to Elasticsearch only once and after that it stops
> writing to Elasticsearch even if there is more data that gets fed into
> Kafka. Sometimes, it does not even write once. We tested this in two other
> jobs and the same issue is there in all of them.
>
>
>
> I have attached a sample code I had created to illustrate the issue. We
> are using Elasticsearch version 5.6.4 and hence the dependency used is
> ‘flink-connector-elasticsearch5_2.11’.
>
>
>
> Regards,
>
> Teena
>
>
>
>
>
>
>
>
>
>
>
RE: Multiple Elasticsearch sinks not working in Flink
Posted by Teena Kappen // BPRISE <te...@bprise.com>.
Hi Timo,
It works fine when the second sink is a Cassandra Sink. The data gets read from KafkaTopic2 and it gets written to Cassandra as expected.
Regards,
Teena
From: Timo Walther [mailto:twalthr@apache.org]
Sent: 18 January 2018 18:41
To: user@flink.apache.org
Subject: Re: Multiple Elasticsearch sinks not working in Flink
Hi Teena,
what happens if you replace the second sink with a non-ElasticSearchSink? Is there the same result? Is the data read from the KafkaTopic2?
We should determine which system is the bottleneck.
Regards,
Timo
Am 1/18/18 um 9:53 AM schrieb Teena Kappen // BPRISE:
Hi,
I am running flink 1.4 in single node. My job has two Kafka consumers reading from separate topics. After fetching the data, the job writes it to two separate Elasticsearch sinks. So the process is like this
KafkaTopic1 -> Kafkaconsumer1 -> create output record -> Elasticsearchsink1
KafkaTopic2 -> Kafkaconsumer2 -> create output record -> Elasticsearchsink2
Both the streams and their processing are completely unrelated. The first sink works as expected and it writes the output for all input records. The second sink writes to Elasticsearch only once and after that it stops writing to Elasticsearch even if there is more data that gets fed into Kafka. Sometimes, it does not even write once. We tested this in two other jobs and the same issue is there in all of them.
I have attached a sample code I had created to illustrate the issue. We are using Elasticsearch version 5.6.4 and hence the dependency used is 'flink-connector-elasticsearch5_2.11'.
Regards,
Teena
Re: Multiple Elasticsearch sinks not working in Flink
Posted by Timo Walther <tw...@apache.org>.
Hi Teena,
what happens if you replace the second sink with a
non-ElasticSearchSink? Is there the same result? Is the data read from
the KafkaTopic2?
We should determine which system is the bottleneck.
Regards,
Timo
Am 1/18/18 um 9:53 AM schrieb Teena Kappen // BPRISE:
>
> Hi,
>
> I am running flink 1.4 in single node. My job has two Kafka consumers
> reading from separate topics. After fetching the data, the job writes
> it to two separate Elasticsearch sinks. So the process is like this
>
> KafkaTopic1 -> Kafkaconsumer1 -> create output record ->
> Elasticsearchsink1
>
> KafkaTopic2 -> Kafkaconsumer2 -> create output record ->
> Elasticsearchsink2
>
> Both the streams and their processing are completely unrelated. The
> first sink works as expected and it writes the output for all input
> records. The second sink writes to Elasticsearch only once and after
> that it stops writing to Elasticsearch even if there is more data that
> gets fed into Kafka. Sometimes, it does not even write once. We tested
> this in two other jobs and the same issue is there in all of them.
>
> I have attached a sample code I had created to illustrate the issue.
> We are using Elasticsearch version 5.6.4 and hence the dependency used
> is ‘flink-connector-elasticsearch5_2.11’.
>
> Regards,
>
> Teena
>