You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by 大森林 <ap...@foxmail.com> on 2020/10/06 14:10:49 UTC
how to simulate the scene "back pressure" in flink?Thanks~!
I want to learn the concept "back pressure".
but I can not find the datastream generator example to generate a lot of data.
besides,
is there any example on how to simulate the scene "back pressure" in WEB UI?
Thanks for your help~
Re: how to simulate the scene "back pressure" in flink?Thanks~!
Posted by David Anderson <da...@apache.org>.
>
> Could I use your command with no docker?
Hypothetically, yes, but it's a somewhat impractical idea. The
ClickCountJob needs Flink and Kafka, and there is another java application
(the clickevent-generator) that writes into Kafka the data that is being
processed.
On Sat, Oct 10, 2020 at 5:32 PM 大森林 <ap...@foxmail.com> wrote:
>
> Could I use your command with no docker?
>
> ------------------ 原始邮件 ------------------
> *发件人:* "David Anderson" <da...@apache.org>;
> *发送时间:* 2020年10月10日(星期六) 晚上10:30
> *收件人:* "大森林"<ap...@foxmail.com>;
> *抄送:* "Arvid Heise"<ar...@flink.apache.org>;
> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>
> The ClickCountJob used in the operations playground accepts an application
> parameter, like this:
>
> flink run -d /opt/ClickCountJob.jar --bootstrap.servers kafka:9092
> --checkpointing --event-time --backpressure
>
> To try this, you would modify the docker-compose.yaml file in [1]. If you
> want to see how it is implemented, see [2].
>
> You can not use this --backpressure option with any other application.
>
> [1]
> https://github.com/apache/flink-playgrounds/blob/release-1.11/operations-playground/docker-compose.yaml
> [2]
> https://github.com/apache/flink-playgrounds/blob/release-1.11/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/BackpressureMap.java
>
>
> On Sat, Oct 10, 2020 at 7:26 AM 大森林 <ap...@foxmail.com> wrote:
>
>> Thanks for both of your help...
>> but...
>>
>> I can not understand both:
>>
>> ------------------------------------------------------------------------------------------------
>> Dear David Anderson:
>> Is the whole command like this?
>> flink run *--backpressure* -c wordcount_increstate
>> datastream_api-1.0-SNAPSHOT.jar
>>
>>
>> ------------------------------------------------------------------------------------------------
>> Dear Arvid Heise:
>> For conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
>> will this settings work to sleep when the output stream is generating?
>>
>>
>> ------------------------------------------------------------------------------------------------
>> apologise for my poor basic knowledge of flink~
>> Thanks for both of your help~
>>
>>
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "David Anderson" <da...@apache.org>;
>> *发送时间:* 2020年10月9日(星期五) 晚上8:23
>> *收件人:* "Arvid Heise"<ar...@ververica.com>;
>> *抄送:* "大森林"<ap...@flink.apache.org>;
>> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>>
>> The Flink Operations Playground includes an optional backpressure
>> simulation you can experiment with. It is described at the end of [1].
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/flink-operations-playground.html#variants
>>
>>
>> On Fri, Oct 9, 2020 at 10:02 AM Arvid Heise <ar...@ververica.com> wrote:
>>
>>> You can add
>>>
>>> conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
>>>
>>> at any place before creating the environment [1]. Default value is 8081,
>>> so you can access web ui through http://localhost:8081, but you can
>>> really choose any other free port.
>>>
>>> [1]
>>> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L198
>>> On Fri, Oct 9, 2020 at 9:24 AM 大森林 <ap...@foxmail.com> wrote:
>>>
>>>> Thanks for your repies,
>>>> could you tell me where to set RestOption.POPT?in configuration
>>>> what's the value should I set for RestOption.PORT?
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> ------------------ 原始邮件 ------------------
>>>> *发件人:* "Arvid Heise" <ar...@ververica.com>;
>>>> *发送时间:* 2020年10月9日(星期五) 下午3:00
>>>> *收件人:* "大森林"<ap...@foxmail.com>;
>>>> *抄送:* "David ,Anderson"<da...@apache.org>;"user"<
>>>> user@flink.apache.org>;
>>>> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>>>>
>>>> The easiest way to see backpressure is to add some sleep to your sink,
>>>> check [1] for an example.
>>>> If you execute that unit test with a RestOption.PORT set in the
>>>> configuration, you can even load the Web UI and watch the backpressure
>>>> accumulate and finally go away at the end of the test.
>>>>
>>>> [1]
>>>> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536
>>>>
>>>> On Tue, Oct 6, 2020 at 4:13 PM 大森林 <ap...@foxmail.com> wrote:
>>>>
>>>>>
>>>>> I want to learn the concept "back pressure".
>>>>> but I can not find the datastream generator example to generate a lot
>>>>> of data.
>>>>>
>>>>> besides,
>>>>> is there any example on how to simulate the scene "back pressure" in
>>>>> WEB UI?
>>>>>
>>>>> Thanks for your help~
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Arvid Heise | Senior Java Developer
>>>>
>>>> <https://www.ververica.com/>
>>>>
>>>> Follow us @VervericaData
>>>>
>>>> --
>>>>
>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>>> Conference
>>>>
>>>> Stream Processing | Event Driven | Real Time
>>>>
>>>> --
>>>>
>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>
>>>> --
>>>> Ververica GmbH
>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>>>> (Toni) Cheng
>>>>
>>>
>>>
>>> --
>>>
>>> Arvid Heise | Senior Java Developer
>>>
>>> <https://www.ververica.com/>
>>>
>>> Follow us @VervericaData
>>>
>>> --
>>>
>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>> Conference
>>>
>>> Stream Processing | Event Driven | Real Time
>>>
>>> --
>>>
>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>
>>> --
>>> Ververica GmbH
>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>>> (Toni) Cheng
>>>
>>
回复： how to simulate the scene "back pressure" in flink?Thanks~!
Posted by 大森林 <ap...@foxmail.com>.
Could I use your command with no docker?
------------------ 原始邮件 ------------------
发件人: "David Anderson" <danderson@apache.org>;
发送时间: 2020年10月10日(星期六) 晚上10:30
收件人: "大森林"<appleyuchi@foxmail.com>;
抄送: "Arvid Heise"<arvid@ververica.com>;"user"<user@flink.apache.org>;
主题: Re: how to simulate the scene "back pressure" in flink?Thanks~!
The ClickCountJob used in the operations playground accepts an application parameter, like this:
flink run -d /opt/ClickCountJob.jar --bootstrap.servers kafka:9092 --checkpointing --event-time --backpressure
To try this, you would modify the docker-compose.yaml file in [1]. If you want to see how it is implemented, see [2].
You can not use this --backpressure option with any other application.
[1] https://github.com/apache/flink-playgrounds/blob/release-1.11/operations-playground/docker-compose.yaml
[2] https://github.com/apache/flink-playgrounds/blob/release-1.11/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/BackpressureMap.java
On Sat, Oct 10, 2020 at 7:26 AM 大森林 <appleyuchi@foxmail.com> wrote:
Thanks for both of your help...
but...
I can not understand both:
------------------------------------------------------------------------------------------------
Dear David Anderson:
Is the whole command like this?
flink run --backpressure -c wordcount_increstate datastream_api-1.0-SNAPSHOT.jar
------------------------------------------------------------------------------------------------
Dear Arvid Heise:
For conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
will this settings work to sleep when the output stream is generating?
------------------------------------------------------------------------------------------------
apologise for my poor basic knowledge of flink~
Thanks for both of your help~
------------------ 原始邮件 ------------------
发件人: "David Anderson" <danderson@apache.org>;
发送时间: 2020年10月9日(星期五) 晚上8:23
收件人: "Arvid Heise"<arvid@ververica.com>;
抄送: "大森林"<appleyuchi@foxmail.com>;"user"<user@flink.apache.org>;
主题: Re: how to simulate the scene "back pressure" in flink?Thanks~!
The Flink Operations Playground includes an optional backpressure simulation you can experiment with. It is described at the end of [1].
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/flink-operations-playground.html#variants
On Fri, Oct 9, 2020 at 10:02 AM Arvid Heise <arvid@ververica.com> wrote:
You can add
conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
at any place before creating the environment [1]. Default value is 8081, so you can access web ui through http://localhost:8081, but you can really choose any other free port.
[1] https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L198
On Fri, Oct 9, 2020 at 9:24 AM 大森林 <appleyuchi@foxmail.com> wrote:
Thanks for your repies,
could you tell me where to set RestOption.POPT?in configuration
what's the value should I set for RestOption.PORT?
Thanks.
------------------ 原始邮件 ------------------
发件人: "Arvid Heise" <arvid@ververica.com>;
发送时间: 2020年10月9日(星期五) 下午3:00
收件人: "大森林"<appleyuchi@foxmail.com>;
抄送: "David&nbsp,Anderson"<danderson@apache.org>;"user"<user@flink.apache.org>;
主题: Re: how to simulate the scene "back pressure" in flink?Thanks~!
The easiest way to see backpressure is to add some sleep to your sink, check [1] for an example.
If you execute that unit test with a RestOption.PORT set in the configuration, you can even load the Web UI and watch the backpressure accumulate and finally go away at the end of the test.
[1] https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536
On Tue, Oct 6, 2020 at 4:13 PM 大森林 <appleyuchi@foxmail.com> wrote:
I want to learn the concept "back pressure".
but I can not find the datastream generator example to generate a lot of data.
besides,
is there any example on how to simulate the scene "back pressure" in WEB UI?
Thanks for your help~
--
Arvid Heise | Senior Java Developer
Follow us @VervericaData
--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
--
Arvid Heise | Senior Java Developer
Follow us @VervericaData
--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
Re: how to simulate the scene "back pressure" in flink?Thanks~!
Posted by David Anderson <da...@apache.org>.
The ClickCountJob used in the operations playground accepts an application
parameter, like this:
flink run -d /opt/ClickCountJob.jar --bootstrap.servers kafka:9092
--checkpointing --event-time --backpressure
To try this, you would modify the docker-compose.yaml file in [1]. If you
want to see how it is implemented, see [2].
You can not use this --backpressure option with any other application.
[1]
https://github.com/apache/flink-playgrounds/blob/release-1.11/operations-playground/docker-compose.yaml
[2]
https://github.com/apache/flink-playgrounds/blob/release-1.11/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/BackpressureMap.java
On Sat, Oct 10, 2020 at 7:26 AM 大森林 <ap...@foxmail.com> wrote:
> Thanks for both of your help...
> but...
>
> I can not understand both:
>
> ------------------------------------------------------------------------------------------------
> Dear David Anderson:
> Is the whole command like this?
> flink run *--backpressure* -c wordcount_increstate
> datastream_api-1.0-SNAPSHOT.jar
>
>
> ------------------------------------------------------------------------------------------------
> Dear Arvid Heise:
> For conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
> will this settings work to sleep when the output stream is generating?
>
>
> ------------------------------------------------------------------------------------------------
> apologise for my poor basic knowledge of flink~
> Thanks for both of your help~
>
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "David Anderson" <da...@apache.org>;
> *发送时间:* 2020年10月9日(星期五) 晚上8:23
> *收件人:* "Arvid Heise"<ar...@ververica.com>;
> *抄送:* "大森林"<ap...@flink.apache.org>;
> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>
> The Flink Operations Playground includes an optional backpressure
> simulation you can experiment with. It is described at the end of [1].
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/flink-operations-playground.html#variants
>
>
> On Fri, Oct 9, 2020 at 10:02 AM Arvid Heise <ar...@ververica.com> wrote:
>
>> You can add
>>
>> conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
>>
>> at any place before creating the environment [1]. Default value is 8081,
>> so you can access web ui through http://localhost:8081, but you can
>> really choose any other free port.
>>
>> [1]
>> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L198
>> On Fri, Oct 9, 2020 at 9:24 AM 大森林 <ap...@foxmail.com> wrote:
>>
>>> Thanks for your repies,
>>> could you tell me where to set RestOption.POPT?in configuration
>>> what's the value should I set for RestOption.PORT?
>>>
>>> Thanks.
>>>
>>>
>>> ------------------ 原始邮件 ------------------
>>> *发件人:* "Arvid Heise" <ar...@ververica.com>;
>>> *发送时间:* 2020年10月9日(星期五) 下午3:00
>>> *收件人:* "大森林"<ap...@foxmail.com>;
>>> *抄送:* "David ,Anderson"<da...@apache.org>;"user"<
>>> user@flink.apache.org>;
>>> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>>>
>>> The easiest way to see backpressure is to add some sleep to your sink,
>>> check [1] for an example.
>>> If you execute that unit test with a RestOption.PORT set in the
>>> configuration, you can even load the Web UI and watch the backpressure
>>> accumulate and finally go away at the end of the test.
>>>
>>> [1]
>>> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536
>>>
>>> On Tue, Oct 6, 2020 at 4:13 PM 大森林 <ap...@foxmail.com> wrote:
>>>
>>>>
>>>> I want to learn the concept "back pressure".
>>>> but I can not find the datastream generator example to generate a lot
>>>> of data.
>>>>
>>>> besides,
>>>> is there any example on how to simulate the scene "back pressure" in
>>>> WEB UI?
>>>>
>>>> Thanks for your help~
>>>>
>>>
>>>
>>> --
>>>
>>> Arvid Heise | Senior Java Developer
>>>
>>> <https://www.ververica.com/>
>>>
>>> Follow us @VervericaData
>>>
>>> --
>>>
>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>> Conference
>>>
>>> Stream Processing | Event Driven | Real Time
>>>
>>> --
>>>
>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>
>>> --
>>> Ververica GmbH
>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>>> (Toni) Cheng
>>>
>>
>>
>> --
>>
>> Arvid Heise | Senior Java Developer
>>
>> <https://www.ververica.com/>
>>
>> Follow us @VervericaData
>>
>> --
>>
>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>> Conference
>>
>> Stream Processing | Event Driven | Real Time
>>
>> --
>>
>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>
>> --
>> Ververica GmbH
>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>> (Toni) Cheng
>>
>
回复： how to simulate the scene "back pressure" in flink?Thanks~!
Posted by 大森林 <ap...@foxmail.com>.
Thanks for both of your help...
but...
I can not understand both:
------------------------------------------------------------------------------------------------
Dear David Anderson:
Is the whole command like this?
flink run --backpressure -c wordcount_increstate datastream_api-1.0-SNAPSHOT.jar
------------------------------------------------------------------------------------------------
Dear Arvid Heise:
For conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
will this settings work to sleep when the output stream is generating?
------------------------------------------------------------------------------------------------
apologise for my poor basic knowledge of flink~
Thanks for both of your help~
------------------ 原始邮件 ------------------
发件人: "David Anderson" <danderson@apache.org>;
发送时间: 2020年10月9日(星期五) 晚上8:23
收件人: "Arvid Heise"<arvid@ververica.com>;
抄送: "大森林"<appleyuchi@foxmail.com>;"user"<user@flink.apache.org>;
主题: Re: how to simulate the scene "back pressure" in flink?Thanks~!
The Flink Operations Playground includes an optional backpressure simulation you can experiment with. It is described at the end of [1].
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/flink-operations-playground.html#variants
On Fri, Oct 9, 2020 at 10:02 AM Arvid Heise <arvid@ververica.com> wrote:
You can add
conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
at any place before creating the environment [1]. Default value is 8081, so you can access web ui through http://localhost:8081, but you can really choose any other free port.
[1] https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L198
On Fri, Oct 9, 2020 at 9:24 AM 大森林 <appleyuchi@foxmail.com> wrote:
Thanks for your repies,
could you tell me where to set RestOption.POPT?in configuration
what's the value should I set for RestOption.PORT?
Thanks.
------------------ 原始邮件 ------------------
发件人: "Arvid Heise" <arvid@ververica.com>;
发送时间: 2020年10月9日(星期五) 下午3:00
收件人: "大森林"<appleyuchi@foxmail.com>;
抄送: "David&nbsp,Anderson"<danderson@apache.org>;"user"<user@flink.apache.org>;
主题: Re: how to simulate the scene "back pressure" in flink?Thanks~!
The easiest way to see backpressure is to add some sleep to your sink, check [1] for an example.
If you execute that unit test with a RestOption.PORT set in the configuration, you can even load the Web UI and watch the backpressure accumulate and finally go away at the end of the test.
[1] https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536
On Tue, Oct 6, 2020 at 4:13 PM 大森林 <appleyuchi@foxmail.com> wrote:
I want to learn the concept "back pressure".
but I can not find the datastream generator example to generate a lot of data.
besides,
is there any example on how to simulate the scene "back pressure" in WEB UI?
Thanks for your help~
--
Arvid Heise | Senior Java Developer
Follow us @VervericaData
--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
--
Arvid Heise | Senior Java Developer
Follow us @VervericaData
--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
Re: how to simulate the scene "back pressure" in flink?Thanks~!
Posted by David Anderson <da...@apache.org>.
The Flink Operations Playground includes an optional backpressure
simulation you can experiment with. It is described at the end of [1].
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/flink-operations-playground.html#variants
On Fri, Oct 9, 2020 at 10:02 AM Arvid Heise <ar...@ververica.com> wrote:
> You can add
>
> conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
>
> at any place before creating the environment [1]. Default value is 8081,
> so you can access web ui through http://localhost:8081, but you can
> really choose any other free port.
>
> [1]
> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L198
> On Fri, Oct 9, 2020 at 9:24 AM 大森林 <ap...@foxmail.com> wrote:
>
>> Thanks for your repies,
>> could you tell me where to set RestOption.POPT?in configuration
>> what's the value should I set for RestOption.PORT?
>>
>> Thanks.
>>
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "Arvid Heise" <ar...@ververica.com>;
>> *发送时间:* 2020年10月9日(星期五) 下午3:00
>> *收件人:* "大森林"<ap...@foxmail.com>;
>> *抄送:* "David ,Anderson"<da...@apache.org>;"user"<
>> user@flink.apache.org>;
>> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>>
>> The easiest way to see backpressure is to add some sleep to your sink,
>> check [1] for an example.
>> If you execute that unit test with a RestOption.PORT set in the
>> configuration, you can even load the Web UI and watch the backpressure
>> accumulate and finally go away at the end of the test.
>>
>> [1]
>> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536
>>
>> On Tue, Oct 6, 2020 at 4:13 PM 大森林 <ap...@foxmail.com> wrote:
>>
>>>
>>> I want to learn the concept "back pressure".
>>> but I can not find the datastream generator example to generate a lot of
>>> data.
>>>
>>> besides,
>>> is there any example on how to simulate the scene "back pressure" in
>>> WEB UI?
>>>
>>> Thanks for your help~
>>>
>>
>>
>> --
>>
>> Arvid Heise | Senior Java Developer
>>
>> <https://www.ververica.com/>
>>
>> Follow us @VervericaData
>>
>> --
>>
>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>> Conference
>>
>> Stream Processing | Event Driven | Real Time
>>
>> --
>>
>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>
>> --
>> Ververica GmbH
>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>> (Toni) Cheng
>>
>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> <https://www.ververica.com/>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>
Re: how to simulate the scene "back pressure" in flink?Thanks~!
Posted by Arvid Heise <ar...@ververica.com>.
You can add
conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue());
at any place before creating the environment [1]. Default value is 8081, so
you can access web ui through http://localhost:8081, but you can really
choose any other free port.
[1]
https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L198
On Fri, Oct 9, 2020 at 9:24 AM 大森林 <ap...@foxmail.com> wrote:
> Thanks for your repies,
> could you tell me where to set RestOption.POPT?in configuration
> what's the value should I set for RestOption.PORT?
>
> Thanks.
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Arvid Heise" <ar...@ververica.com>;
> *发送时间:* 2020年10月9日(星期五) 下午3:00
> *收件人:* "大森林"<ap...@foxmail.com>;
> *抄送:* "David ,Anderson"<da...@apache.org>;"user"<
> user@flink.apache.org>;
> *主题:* Re: how to simulate the scene "back pressure" in flink?Thanks~!
>
> The easiest way to see backpressure is to add some sleep to your sink,
> check [1] for an example.
> If you execute that unit test with a RestOption.PORT set in the
> configuration, you can even load the Web UI and watch the backpressure
> accumulate and finally go away at the end of the test.
>
> [1]
> https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536
>
> On Tue, Oct 6, 2020 at 4:13 PM 大森林 <ap...@foxmail.com> wrote:
>
>>
>> I want to learn the concept "back pressure".
>> but I can not find the datastream generator example to generate a lot of
>> data.
>>
>> besides,
>> is there any example on how to simulate the scene "back pressure" in WEB
>> UI?
>>
>> Thanks for your help~
>>
>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> <https://www.ververica.com/>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>
--
Arvid Heise | Senior Java Developer
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng
回复： how to simulate the scene "back pressure" in flink?Thanks~!
Posted by 大森林 <ap...@foxmail.com>.
Thanks for your repies,
could you tell me where to set RestOption.POPT?in configuration
what's the value should I set for RestOption.PORT?
Thanks.
------------------ 原始邮件 ------------------
发件人: "Arvid Heise" <arvid@ververica.com>;
发送时间: 2020年10月9日(星期五) 下午3:00
收件人: "大森林"<appleyuchi@foxmail.com>;
抄送: "David&nbsp,Anderson"<danderson@apache.org>;"user"<user@flink.apache.org>;
主题: Re: how to simulate the scene "back pressure" in flink?Thanks~!
The easiest way to see backpressure is to add some sleep to your sink, check [1] for an example.
If you execute that unit test with a RestOption.PORT set in the configuration, you can even load the Web UI and watch the backpressure accumulate and finally go away at the end of the test.
[1] https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536
On Tue, Oct 6, 2020 at 4:13 PM 大森林 <appleyuchi@foxmail.com> wrote:
I want to learn the concept "back pressure".
but I can not find the datastream generator example to generate a lot of data.
besides,
is there any example on how to simulate the scene "back pressure" in WEB UI?
Thanks for your help~
--
Arvid Heise | Senior Java Developer
Follow us @VervericaData
--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
Re: how to simulate the scene "back pressure" in flink?Thanks~!
Posted by Arvid Heise <ar...@ververica.com>.
The easiest way to see backpressure is to add some sleep to your sink,
check [1] for an example.
If you execute that unit test with a RestOption.PORT set in the
configuration, you can even load the Web UI and watch the backpressure
accumulate and finally go away at the end of the test.
[1]
https://github.com/apache/flink/blob/61a997364b020b44bd26df76208e76106c6390b5/flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java#L536
On Tue, Oct 6, 2020 at 4:13 PM 大森林 <ap...@foxmail.com> wrote:
>
> I want to learn the concept "back pressure".
> but I can not find the datastream generator example to generate a lot of
> data.
>
> besides,
> is there any example on how to simulate the scene "back pressure" in WEB
> UI?
>
> Thanks for your help~
>
--
Arvid Heise | Senior Java Developer
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng