You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Kamal Mittal <ks...@gmail.com> on 2023/05/18 09:03:23 UTC
Backpressure handling in FileSource APIs - Flink 1.16
Hello Community,
Does FileSource APIs for Bulk and Record stream formats handle back
pressure by any way like slowing down sending data in piepline further or
reading data from source somehow?
Or does it give any callback/handle so that any action can be taken? Can
you please share details if any?
Rgds,
Kamal
Re: Backpressure handling in FileSource APIs - Flink 1.16
Posted by Shammon FY <zj...@gmail.com>.
Hi Kamal,
The network buffer will be full for specific `FileSource` when the job has
back pressure which will block the source subtask. You can refer to network
buffer [1] for more information.
[1]
https://flink.apache.org/2019/06/05/a-deep-dive-into-flinks-network-stack/
Best,
Shammon FY
On Fri, May 26, 2023 at 7:13 PM Kamal Mittal <ks...@gmail.com> wrote:
> Hello Shammon,
>
> Can you please point out the classes where like for "FileSource" slow down
> logic is placed?
>
> Just wanted to understand it more better and try it at my end by running
> various perf. runs, also apply changes in my application if any.
>
> Rgds,
> Kamal
>
> On Thu, May 25, 2023 at 9:16 AM Kamal Mittal <ks...@gmail.com> wrote:
>
>> Hello Shammon,
>>
>> Can you please point out the classes where like for "FileSource" slow
>> down logic is placed?
>>
>> Just wanted to understand it more better and try it at my end by running
>> various perf. runs, also apply changes in my application if any.
>>
>> Rgds,
>> Kamal
>>
>> On Wed, May 24, 2023 at 11:41 AM Kamal Mittal <ks...@gmail.com> wrote:
>>
>>> Thanks Shammon for clarification.
>>>
>>> On Wed, May 24, 2023 at 11:01 AM Shammon FY <zj...@gmail.com> wrote:
>>>
>>>> Hi Kamal,
>>>>
>>>> The source will slow down when there is backpressure in the flink job,
>>>> you can refer to docs [1] and [2] to get more detailed information about
>>>> backpressure mechanism.
>>>>
>>>> Currently there's no API or Callback in source for users to do some
>>>> customized operations for backpressure, but users can collect the metrics
>>>> of the job and analysis, for example, the metrics in [1] and [3]. I hope
>>>> this can help you.
>>>>
>>>> [1]
>>>> https://flink.apache.org/2021/07/07/how-to-identify-the-source-of-backpressure/#:~:text=Backpressure%20is%20an%20indicator%20that,the%20queues%20before%20being%20processed
>>>> .
>>>> [2]
>>>> https://www.alibabacloud.com/blog/analysis-of-network-flow-control-and-back-pressure-flink-advanced-tutorials_596632
>>>> [3]
>>>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/monitoring/back_pressure/
>>>>
>>>> On Tue, May 23, 2023 at 9:40 PM Kamal Mittal <ks...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello Community,
>>>>>
>>>>> Can you please share views about the query asked above w.r.t back
>>>>> pressure for FileSource APIs for Bulk and Record stream formats.
>>>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa
>>>>> conversion.
>>>>>
>>>>> Rgds,
>>>>> Kamal
>>>>>
>>>>> On Tue, 23 May 2023, 12:26 pm Kamal Mittal, <ks...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Added Flink community DL as well.
>>>>>>
>>>>>> ---------- Forwarded message ---------
>>>>>> From: Kamal Mittal <ks...@gmail.com>
>>>>>> Date: Tue, May 23, 2023 at 7:57 AM
>>>>>> Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16
>>>>>> To: Shammon FY <zj...@gmail.com>
>>>>>>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Yes, want to take some custom actions and also if there is any
>>>>>> default behavior of slowing down sending data in pipeline further or
>>>>>> reading data from source somehow?
>>>>>>
>>>>>> Rgds,
>>>>>> Kamal
>>>>>>
>>>>>> On Tue, May 23, 2023 at 6:06 AM Shammon FY <zj...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Kamal,
>>>>>>>
>>>>>>> If I understand correctly, do you want the source to do some custom
>>>>>>> actions, such as current limiting, when there is backpressure in the job?
>>>>>>>
>>>>>>> Best,
>>>>>>> Shammon FY
>>>>>>>
>>>>>>>
>>>>>>> On Mon, May 22, 2023 at 2:12 PM Kamal Mittal <ks...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello Community,
>>>>>>>>
>>>>>>>> Can you please share views about the query asked above w.r.t back
>>>>>>>> pressure for FileSource APIs for Bulk and Record stream formats.
>>>>>>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa
>>>>>>>> conversion.
>>>>>>>>
>>>>>>>> Rgds,
>>>>>>>> Kamal
>>>>>>>>
>>>>>>>> On Thu, May 18, 2023 at 2:33 PM Kamal Mittal <ks...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello Community,
>>>>>>>>>
>>>>>>>>> Does FileSource APIs for Bulk and Record stream formats handle
>>>>>>>>> back pressure by any way like slowing down sending data in piepline further
>>>>>>>>> or reading data from source somehow?
>>>>>>>>> Or does it give any callback/handle so that any action can be
>>>>>>>>> taken? Can you please share details if any?
>>>>>>>>>
>>>>>>>>> Rgds,
>>>>>>>>> Kamal
>>>>>>>>>
>>>>>>>>
Re: Backpressure handling in FileSource APIs - Flink 1.16
Posted by Kamal Mittal <ks...@gmail.com>.
Hello Shammon,
Can you please point out the classes where like for "FileSource" slow down
logic is placed?
Just wanted to understand it more better and try it at my end by running
various perf. runs, also apply changes in my application if any.
Rgds,
Kamal
On Thu, May 25, 2023 at 9:16 AM Kamal Mittal <ks...@gmail.com> wrote:
> Hello Shammon,
>
> Can you please point out the classes where like for "FileSource" slow down
> logic is placed?
>
> Just wanted to understand it more better and try it at my end by running
> various perf. runs, also apply changes in my application if any.
>
> Rgds,
> Kamal
>
> On Wed, May 24, 2023 at 11:41 AM Kamal Mittal <ks...@gmail.com> wrote:
>
>> Thanks Shammon for clarification.
>>
>> On Wed, May 24, 2023 at 11:01 AM Shammon FY <zj...@gmail.com> wrote:
>>
>>> Hi Kamal,
>>>
>>> The source will slow down when there is backpressure in the flink job,
>>> you can refer to docs [1] and [2] to get more detailed information about
>>> backpressure mechanism.
>>>
>>> Currently there's no API or Callback in source for users to do some
>>> customized operations for backpressure, but users can collect the metrics
>>> of the job and analysis, for example, the metrics in [1] and [3]. I hope
>>> this can help you.
>>>
>>> [1]
>>> https://flink.apache.org/2021/07/07/how-to-identify-the-source-of-backpressure/#:~:text=Backpressure%20is%20an%20indicator%20that,the%20queues%20before%20being%20processed
>>> .
>>> [2]
>>> https://www.alibabacloud.com/blog/analysis-of-network-flow-control-and-back-pressure-flink-advanced-tutorials_596632
>>> [3]
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/monitoring/back_pressure/
>>>
>>> On Tue, May 23, 2023 at 9:40 PM Kamal Mittal <ks...@gmail.com> wrote:
>>>
>>>> Hello Community,
>>>>
>>>> Can you please share views about the query asked above w.r.t back
>>>> pressure for FileSource APIs for Bulk and Record stream formats.
>>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa
>>>> conversion.
>>>>
>>>> Rgds,
>>>> Kamal
>>>>
>>>> On Tue, 23 May 2023, 12:26 pm Kamal Mittal, <ks...@gmail.com> wrote:
>>>>
>>>>> Added Flink community DL as well.
>>>>>
>>>>> ---------- Forwarded message ---------
>>>>> From: Kamal Mittal <ks...@gmail.com>
>>>>> Date: Tue, May 23, 2023 at 7:57 AM
>>>>> Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16
>>>>> To: Shammon FY <zj...@gmail.com>
>>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>> Yes, want to take some custom actions and also if there is any default
>>>>> behavior of slowing down sending data in pipeline further or reading data
>>>>> from source somehow?
>>>>>
>>>>> Rgds,
>>>>> Kamal
>>>>>
>>>>> On Tue, May 23, 2023 at 6:06 AM Shammon FY <zj...@gmail.com> wrote:
>>>>>
>>>>>> Hi Kamal,
>>>>>>
>>>>>> If I understand correctly, do you want the source to do some custom
>>>>>> actions, such as current limiting, when there is backpressure in the job?
>>>>>>
>>>>>> Best,
>>>>>> Shammon FY
>>>>>>
>>>>>>
>>>>>> On Mon, May 22, 2023 at 2:12 PM Kamal Mittal <ks...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello Community,
>>>>>>>
>>>>>>> Can you please share views about the query asked above w.r.t back
>>>>>>> pressure for FileSource APIs for Bulk and Record stream formats.
>>>>>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa
>>>>>>> conversion.
>>>>>>>
>>>>>>> Rgds,
>>>>>>> Kamal
>>>>>>>
>>>>>>> On Thu, May 18, 2023 at 2:33 PM Kamal Mittal <ks...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello Community,
>>>>>>>>
>>>>>>>> Does FileSource APIs for Bulk and Record stream formats handle back
>>>>>>>> pressure by any way like slowing down sending data in piepline further or
>>>>>>>> reading data from source somehow?
>>>>>>>> Or does it give any callback/handle so that any action can be
>>>>>>>> taken? Can you please share details if any?
>>>>>>>>
>>>>>>>> Rgds,
>>>>>>>> Kamal
>>>>>>>>
>>>>>>>
Re: Backpressure handling in FileSource APIs - Flink 1.16
Posted by Kamal Mittal <ks...@gmail.com>.
Hello Shammon,
Can you please point out the classes where like for "FileSource" slow down
logic is placed?
Just wanted to understand it more better and try it at my end by running
various perf. runs, also apply changes in my application if any.
Rgds,
Kamal
On Wed, May 24, 2023 at 11:41 AM Kamal Mittal <ks...@gmail.com> wrote:
> Thanks Shammon for clarification.
>
> On Wed, May 24, 2023 at 11:01 AM Shammon FY <zj...@gmail.com> wrote:
>
>> Hi Kamal,
>>
>> The source will slow down when there is backpressure in the flink job,
>> you can refer to docs [1] and [2] to get more detailed information about
>> backpressure mechanism.
>>
>> Currently there's no API or Callback in source for users to do some
>> customized operations for backpressure, but users can collect the metrics
>> of the job and analysis, for example, the metrics in [1] and [3]. I hope
>> this can help you.
>>
>> [1]
>> https://flink.apache.org/2021/07/07/how-to-identify-the-source-of-backpressure/#:~:text=Backpressure%20is%20an%20indicator%20that,the%20queues%20before%20being%20processed
>> .
>> [2]
>> https://www.alibabacloud.com/blog/analysis-of-network-flow-control-and-back-pressure-flink-advanced-tutorials_596632
>> [3]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/monitoring/back_pressure/
>>
>> On Tue, May 23, 2023 at 9:40 PM Kamal Mittal <ks...@gmail.com> wrote:
>>
>>> Hello Community,
>>>
>>> Can you please share views about the query asked above w.r.t back
>>> pressure for FileSource APIs for Bulk and Record stream formats.
>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa
>>> conversion.
>>>
>>> Rgds,
>>> Kamal
>>>
>>> On Tue, 23 May 2023, 12:26 pm Kamal Mittal, <ks...@gmail.com> wrote:
>>>
>>>> Added Flink community DL as well.
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: Kamal Mittal <ks...@gmail.com>
>>>> Date: Tue, May 23, 2023 at 7:57 AM
>>>> Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16
>>>> To: Shammon FY <zj...@gmail.com>
>>>>
>>>>
>>>> Hello,
>>>>
>>>> Yes, want to take some custom actions and also if there is any default
>>>> behavior of slowing down sending data in pipeline further or reading data
>>>> from source somehow?
>>>>
>>>> Rgds,
>>>> Kamal
>>>>
>>>> On Tue, May 23, 2023 at 6:06 AM Shammon FY <zj...@gmail.com> wrote:
>>>>
>>>>> Hi Kamal,
>>>>>
>>>>> If I understand correctly, do you want the source to do some custom
>>>>> actions, such as current limiting, when there is backpressure in the job?
>>>>>
>>>>> Best,
>>>>> Shammon FY
>>>>>
>>>>>
>>>>> On Mon, May 22, 2023 at 2:12 PM Kamal Mittal <ks...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello Community,
>>>>>>
>>>>>> Can you please share views about the query asked above w.r.t back
>>>>>> pressure for FileSource APIs for Bulk and Record stream formats.
>>>>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa
>>>>>> conversion.
>>>>>>
>>>>>> Rgds,
>>>>>> Kamal
>>>>>>
>>>>>> On Thu, May 18, 2023 at 2:33 PM Kamal Mittal <ks...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello Community,
>>>>>>>
>>>>>>> Does FileSource APIs for Bulk and Record stream formats handle back
>>>>>>> pressure by any way like slowing down sending data in piepline further or
>>>>>>> reading data from source somehow?
>>>>>>> Or does it give any callback/handle so that any action can be taken?
>>>>>>> Can you please share details if any?
>>>>>>>
>>>>>>> Rgds,
>>>>>>> Kamal
>>>>>>>
>>>>>>
Re: Backpressure handling in FileSource APIs - Flink 1.16
Posted by Kamal Mittal <ks...@gmail.com>.
Thanks Shammon for clarification.
On Wed, May 24, 2023 at 11:01 AM Shammon FY <zj...@gmail.com> wrote:
> Hi Kamal,
>
> The source will slow down when there is backpressure in the flink job, you
> can refer to docs [1] and [2] to get more detailed information about
> backpressure mechanism.
>
> Currently there's no API or Callback in source for users to do some
> customized operations for backpressure, but users can collect the metrics
> of the job and analysis, for example, the metrics in [1] and [3]. I hope
> this can help you.
>
> [1]
> https://flink.apache.org/2021/07/07/how-to-identify-the-source-of-backpressure/#:~:text=Backpressure%20is%20an%20indicator%20that,the%20queues%20before%20being%20processed
> .
> [2]
> https://www.alibabacloud.com/blog/analysis-of-network-flow-control-and-back-pressure-flink-advanced-tutorials_596632
> [3]
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/monitoring/back_pressure/
>
> On Tue, May 23, 2023 at 9:40 PM Kamal Mittal <ks...@gmail.com> wrote:
>
>> Hello Community,
>>
>> Can you please share views about the query asked above w.r.t back
>> pressure for FileSource APIs for Bulk and Record stream formats.
>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa
>> conversion.
>>
>> Rgds,
>> Kamal
>>
>> On Tue, 23 May 2023, 12:26 pm Kamal Mittal, <ks...@gmail.com> wrote:
>>
>>> Added Flink community DL as well.
>>>
>>> ---------- Forwarded message ---------
>>> From: Kamal Mittal <ks...@gmail.com>
>>> Date: Tue, May 23, 2023 at 7:57 AM
>>> Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16
>>> To: Shammon FY <zj...@gmail.com>
>>>
>>>
>>> Hello,
>>>
>>> Yes, want to take some custom actions and also if there is any default
>>> behavior of slowing down sending data in pipeline further or reading data
>>> from source somehow?
>>>
>>> Rgds,
>>> Kamal
>>>
>>> On Tue, May 23, 2023 at 6:06 AM Shammon FY <zj...@gmail.com> wrote:
>>>
>>>> Hi Kamal,
>>>>
>>>> If I understand correctly, do you want the source to do some custom
>>>> actions, such as current limiting, when there is backpressure in the job?
>>>>
>>>> Best,
>>>> Shammon FY
>>>>
>>>>
>>>> On Mon, May 22, 2023 at 2:12 PM Kamal Mittal <ks...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello Community,
>>>>>
>>>>> Can you please share views about the query asked above w.r.t back
>>>>> pressure for FileSource APIs for Bulk and Record stream formats.
>>>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa
>>>>> conversion.
>>>>>
>>>>> Rgds,
>>>>> Kamal
>>>>>
>>>>> On Thu, May 18, 2023 at 2:33 PM Kamal Mittal <ks...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello Community,
>>>>>>
>>>>>> Does FileSource APIs for Bulk and Record stream formats handle back
>>>>>> pressure by any way like slowing down sending data in piepline further or
>>>>>> reading data from source somehow?
>>>>>> Or does it give any callback/handle so that any action can be taken?
>>>>>> Can you please share details if any?
>>>>>>
>>>>>> Rgds,
>>>>>> Kamal
>>>>>>
>>>>>
Re: Backpressure handling in FileSource APIs - Flink 1.16
Posted by Shammon FY <zj...@gmail.com>.
Hi Kamal,
The source will slow down when there is backpressure in the flink job, you
can refer to docs [1] and [2] to get more detailed information about
backpressure mechanism.
Currently there's no API or Callback in source for users to do some
customized operations for backpressure, but users can collect the metrics
of the job and analysis, for example, the metrics in [1] and [3]. I hope
this can help you.
[1]
https://flink.apache.org/2021/07/07/how-to-identify-the-source-of-backpressure/#:~:text=Backpressure%20is%20an%20indicator%20that,the%20queues%20before%20being%20processed
.
[2]
https://www.alibabacloud.com/blog/analysis-of-network-flow-control-and-back-pressure-flink-advanced-tutorials_596632
[3]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/monitoring/back_pressure/
On Tue, May 23, 2023 at 9:40 PM Kamal Mittal <ks...@gmail.com> wrote:
> Hello Community,
>
> Can you please share views about the query asked above w.r.t back pressure
> for FileSource APIs for Bulk and Record stream formats.
> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa conversion.
>
> Rgds,
> Kamal
>
> On Tue, 23 May 2023, 12:26 pm Kamal Mittal, <ks...@gmail.com> wrote:
>
>> Added Flink community DL as well.
>>
>> ---------- Forwarded message ---------
>> From: Kamal Mittal <ks...@gmail.com>
>> Date: Tue, May 23, 2023 at 7:57 AM
>> Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16
>> To: Shammon FY <zj...@gmail.com>
>>
>>
>> Hello,
>>
>> Yes, want to take some custom actions and also if there is any default
>> behavior of slowing down sending data in pipeline further or reading data
>> from source somehow?
>>
>> Rgds,
>> Kamal
>>
>> On Tue, May 23, 2023 at 6:06 AM Shammon FY <zj...@gmail.com> wrote:
>>
>>> Hi Kamal,
>>>
>>> If I understand correctly, do you want the source to do some custom
>>> actions, such as current limiting, when there is backpressure in the job?
>>>
>>> Best,
>>> Shammon FY
>>>
>>>
>>> On Mon, May 22, 2023 at 2:12 PM Kamal Mittal <ks...@gmail.com> wrote:
>>>
>>>> Hello Community,
>>>>
>>>> Can you please share views about the query asked above w.r.t back
>>>> pressure for FileSource APIs for Bulk and Record stream formats.
>>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa
>>>> conversion.
>>>>
>>>> Rgds,
>>>> Kamal
>>>>
>>>> On Thu, May 18, 2023 at 2:33 PM Kamal Mittal <ks...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello Community,
>>>>>
>>>>> Does FileSource APIs for Bulk and Record stream formats handle back
>>>>> pressure by any way like slowing down sending data in piepline further or
>>>>> reading data from source somehow?
>>>>> Or does it give any callback/handle so that any action can be taken?
>>>>> Can you please share details if any?
>>>>>
>>>>> Rgds,
>>>>> Kamal
>>>>>
>>>>
Re: Backpressure handling in FileSource APIs - Flink 1.16
Posted by Kamal Mittal <ks...@gmail.com>.
Hello Community,
Can you please share views about the query asked above w.r.t back pressure
for FileSource APIs for Bulk and Record stream formats.
Planning to use these APIs w.r.t AVRO to Parquet and vice-versa conversion.
Rgds,
Kamal
On Tue, 23 May 2023, 12:26 pm Kamal Mittal, <ks...@gmail.com> wrote:
> Added Flink community DL as well.
>
> ---------- Forwarded message ---------
> From: Kamal Mittal <ks...@gmail.com>
> Date: Tue, May 23, 2023 at 7:57 AM
> Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16
> To: Shammon FY <zj...@gmail.com>
>
>
> Hello,
>
> Yes, want to take some custom actions and also if there is any default
> behavior of slowing down sending data in pipeline further or reading data
> from source somehow?
>
> Rgds,
> Kamal
>
> On Tue, May 23, 2023 at 6:06 AM Shammon FY <zj...@gmail.com> wrote:
>
>> Hi Kamal,
>>
>> If I understand correctly, do you want the source to do some custom
>> actions, such as current limiting, when there is backpressure in the job?
>>
>> Best,
>> Shammon FY
>>
>>
>> On Mon, May 22, 2023 at 2:12 PM Kamal Mittal <ks...@gmail.com> wrote:
>>
>>> Hello Community,
>>>
>>> Can you please share views about the query asked above w.r.t back
>>> pressure for FileSource APIs for Bulk and Record stream formats.
>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa
>>> conversion.
>>>
>>> Rgds,
>>> Kamal
>>>
>>> On Thu, May 18, 2023 at 2:33 PM Kamal Mittal <ks...@gmail.com> wrote:
>>>
>>>> Hello Community,
>>>>
>>>> Does FileSource APIs for Bulk and Record stream formats handle back
>>>> pressure by any way like slowing down sending data in piepline further or
>>>> reading data from source somehow?
>>>> Or does it give any callback/handle so that any action can be taken?
>>>> Can you please share details if any?
>>>>
>>>> Rgds,
>>>> Kamal
>>>>
>>>
Fwd: Backpressure handling in FileSource APIs - Flink 1.16
Posted by Kamal Mittal <ks...@gmail.com>.
Added Flink community DL as well.
---------- Forwarded message ---------
From: Kamal Mittal <ks...@gmail.com>
Date: Tue, May 23, 2023 at 7:57 AM
Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16
To: Shammon FY <zj...@gmail.com>
Hello,
Yes, want to take some custom actions and also if there is any default
behavior of slowing down sending data in pipeline further or reading data
from source somehow?
Rgds,
Kamal
On Tue, May 23, 2023 at 6:06 AM Shammon FY <zj...@gmail.com> wrote:
> Hi Kamal,
>
> If I understand correctly, do you want the source to do some custom
> actions, such as current limiting, when there is backpressure in the job?
>
> Best,
> Shammon FY
>
>
> On Mon, May 22, 2023 at 2:12 PM Kamal Mittal <ks...@gmail.com> wrote:
>
>> Hello Community,
>>
>> Can you please share views about the query asked above w.r.t back
>> pressure for FileSource APIs for Bulk and Record stream formats.
>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa
>> conversion.
>>
>> Rgds,
>> Kamal
>>
>> On Thu, May 18, 2023 at 2:33 PM Kamal Mittal <ks...@gmail.com> wrote:
>>
>>> Hello Community,
>>>
>>> Does FileSource APIs for Bulk and Record stream formats handle back
>>> pressure by any way like slowing down sending data in piepline further or
>>> reading data from source somehow?
>>> Or does it give any callback/handle so that any action can be taken? Can
>>> you please share details if any?
>>>
>>> Rgds,
>>> Kamal
>>>
>>
Re: Backpressure handling in FileSource APIs - Flink 1.16
Posted by Shammon FY <zj...@gmail.com>.
Hi Kamal,
If I understand correctly, do you want the source to do some custom
actions, such as current limiting, when there is backpressure in the job?
Best,
Shammon FY
On Mon, May 22, 2023 at 2:12 PM Kamal Mittal <ks...@gmail.com> wrote:
> Hello Community,
>
> Can you please share views about the query asked above w.r.t back pressure
> for FileSource APIs for Bulk and Record stream formats.
> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa conversion.
>
> Rgds,
> Kamal
>
> On Thu, May 18, 2023 at 2:33 PM Kamal Mittal <ks...@gmail.com> wrote:
>
>> Hello Community,
>>
>> Does FileSource APIs for Bulk and Record stream formats handle back
>> pressure by any way like slowing down sending data in piepline further or
>> reading data from source somehow?
>> Or does it give any callback/handle so that any action can be taken? Can
>> you please share details if any?
>>
>> Rgds,
>> Kamal
>>
>
Re: Backpressure handling in FileSource APIs - Flink 1.16
Posted by Kamal Mittal <ks...@gmail.com>.
Hello Community,
Can you please share views about the query asked above w.r.t back pressure
for FileSource APIs for Bulk and Record stream formats.
Planning to use these APIs w.r.t AVRO to Parquet and vice-versa conversion.
Rgds,
Kamal
On Thu, May 18, 2023 at 2:33 PM Kamal Mittal <ks...@gmail.com> wrote:
> Hello Community,
>
> Does FileSource APIs for Bulk and Record stream formats handle back
> pressure by any way like slowing down sending data in piepline further or
> reading data from source somehow?
> Or does it give any callback/handle so that any action can be taken? Can
> you please share details if any?
>
> Rgds,
> Kamal
>