You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Surendra Lalwani <su...@swiggy.in> on 2022/05/17 07:44:23 UTC
How to avoid multiple reading from same topic
Hi Team,
We have various SQL queries where we are querying the same kafka topic but
both of the queries are completely different. I wanted to explore if we can
avoid reading multiple times and read only once and perform different
processing according to the query and dump data accordingly.
Thanks and Regards ,
Surendra Lalwani
--
IMPORTANT NOTICE: This e-mail, including any attachments, may contain
confidential information and is intended only for the addressee(s) named
above. If you are not the intended recipient(s), you should not
disseminate, distribute, or copy this e-mail. Please notify the sender by
reply e-mail immediately if you have received this e-mail in error and
permanently delete all copies of the original message from your system.
E-mail transmission cannot be guaranteed to be secure as it could be
intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
contain viruses. Company accepts no liability for any damage or loss of
confidential information caused by this email or due to any virus
transmitted by this email or otherwise.
Re: How to avoid multiple reading from same topic
Posted by Caizhi Weng <ts...@gmail.com>.
Hi!
In that case just put these two SQL queries into one job and use the same
table as their source. The source task will be automatically reused
(actually controlled by a configuration [1] but it is enabled by default)
and the records will be only read once from Kafka and will be replicated
inside Flink if necessary.
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-optimizer-reuse-source-enabled
Surendra Lalwani <su...@swiggy.in> 于2022年5月17日周二 15:52写道:
> So let's assume if we are reading data from a topic named A and performing
> aggregation on 4 hour window and a different pipeline which does same
> aggregation for 6 hour window, so in this case we are reading same data
> couple of times and processing it for different intervals, can we do it
> using just a single read.
>
> Thanks and Regards ,
> Surendra Lalwani
>
>
> On Tue, May 17, 2022 at 1:19 PM Caizhi Weng <ts...@gmail.com> wrote:
>
>> Hi!
>>
>> I wanted to explore if we can avoid reading multiple times and read only
>>> once
>>>
>>
>> Could you elaborate more on the reason you need this? If both of your
>> queries need full data from that topic you'll have to read the whole topic
>> many times. If each query only need to consume a specific portion of that
>> topic I would suggest splitting it into multiple topics.
>>
>>
>> Surendra Lalwani <su...@swiggy.in> 于2022年5月17日周二 15:44写道:
>>
>>> Hi Team,
>>>
>>> We have various SQL queries where we are querying the same kafka topic
>>> but both of the queries are completely different. I wanted to explore if we
>>> can avoid reading multiple times and read only once and perform different
>>> processing according to the query and dump data accordingly.
>>>
>>> Thanks and Regards ,
>>> Surendra Lalwani
>>>
>>>
>>> ------------------------------
>>> IMPORTANT NOTICE: This e-mail, including any attachments, may contain
>>> confidential information and is intended only for the addressee(s) named
>>> above. If you are not the intended recipient(s), you should not
>>> disseminate, distribute, or copy this e-mail. Please notify the sender by
>>> reply e-mail immediately if you have received this e-mail in error and
>>> permanently delete all copies of the original message from your system.
>>> E-mail transmission cannot be guaranteed to be secure as it could be
>>> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
>>> contain viruses. Company accepts no liability for any damage or loss of
>>> confidential information caused by this email or due to any virus
>>> transmitted by this email or otherwise.
>>
>>
>
> ------------------------------
> IMPORTANT NOTICE: This e-mail, including any attachments, may contain
> confidential information and is intended only for the addressee(s) named
> above. If you are not the intended recipient(s), you should not
> disseminate, distribute, or copy this e-mail. Please notify the sender by
> reply e-mail immediately if you have received this e-mail in error and
> permanently delete all copies of the original message from your system.
> E-mail transmission cannot be guaranteed to be secure as it could be
> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> contain viruses. Company accepts no liability for any damage or loss of
> confidential information caused by this email or due to any virus
> transmitted by this email or otherwise.
Re: How to avoid multiple reading from same topic
Posted by Surendra Lalwani <su...@swiggy.in>.
So let's assume if we are reading data from a topic named A and performing
aggregation on 4 hour window and a different pipeline which does same
aggregation for 6 hour window, so in this case we are reading same data
couple of times and processing it for different intervals, can we do it
using just a single read.
Thanks and Regards ,
Surendra Lalwani
On Tue, May 17, 2022 at 1:19 PM Caizhi Weng <ts...@gmail.com> wrote:
> Hi!
>
> I wanted to explore if we can avoid reading multiple times and read only
>> once
>>
>
> Could you elaborate more on the reason you need this? If both of your
> queries need full data from that topic you'll have to read the whole topic
> many times. If each query only need to consume a specific portion of that
> topic I would suggest splitting it into multiple topics.
>
>
> Surendra Lalwani <su...@swiggy.in> 于2022年5月17日周二 15:44写道:
>
>> Hi Team,
>>
>> We have various SQL queries where we are querying the same kafka topic
>> but both of the queries are completely different. I wanted to explore if we
>> can avoid reading multiple times and read only once and perform different
>> processing according to the query and dump data accordingly.
>>
>> Thanks and Regards ,
>> Surendra Lalwani
>>
>>
>> ------------------------------
>> IMPORTANT NOTICE: This e-mail, including any attachments, may contain
>> confidential information and is intended only for the addressee(s) named
>> above. If you are not the intended recipient(s), you should not
>> disseminate, distribute, or copy this e-mail. Please notify the sender by
>> reply e-mail immediately if you have received this e-mail in error and
>> permanently delete all copies of the original message from your system.
>> E-mail transmission cannot be guaranteed to be secure as it could be
>> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
>> contain viruses. Company accepts no liability for any damage or loss of
>> confidential information caused by this email or due to any virus
>> transmitted by this email or otherwise.
>
>
--
IMPORTANT NOTICE: This e-mail, including any attachments, may contain
confidential information and is intended only for the addressee(s) named
above. If you are not the intended recipient(s), you should not
disseminate, distribute, or copy this e-mail. Please notify the sender by
reply e-mail immediately if you have received this e-mail in error and
permanently delete all copies of the original message from your system.
E-mail transmission cannot be guaranteed to be secure as it could be
intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
contain viruses. Company accepts no liability for any damage or loss of
confidential information caused by this email or due to any virus
transmitted by this email or otherwise.
Re: How to avoid multiple reading from same topic
Posted by Caizhi Weng <ts...@gmail.com>.
Hi!
I wanted to explore if we can avoid reading multiple times and read only
> once
>
Could you elaborate more on the reason you need this? If both of your
queries need full data from that topic you'll have to read the whole topic
many times. If each query only need to consume a specific portion of that
topic I would suggest splitting it into multiple topics.
Surendra Lalwani <su...@swiggy.in> 于2022年5月17日周二 15:44写道:
> Hi Team,
>
> We have various SQL queries where we are querying the same kafka topic but
> both of the queries are completely different. I wanted to explore if we can
> avoid reading multiple times and read only once and perform different
> processing according to the query and dump data accordingly.
>
> Thanks and Regards ,
> Surendra Lalwani
>
>
> ------------------------------
> IMPORTANT NOTICE: This e-mail, including any attachments, may contain
> confidential information and is intended only for the addressee(s) named
> above. If you are not the intended recipient(s), you should not
> disseminate, distribute, or copy this e-mail. Please notify the sender by
> reply e-mail immediately if you have received this e-mail in error and
> permanently delete all copies of the original message from your system.
> E-mail transmission cannot be guaranteed to be secure as it could be
> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> contain viruses. Company accepts no liability for any damage or loss of
> confidential information caused by this email or due to any virus
> transmitted by this email or otherwise.