You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn> on 2019/10/16 06:09:51 UTC
NiFi backpressure not work
As the picture shows.
I have set the backpressure object threshold to 10000, but picture shows there's more than 1,000,000. Why this happens?
Thanks, Lei
wanglei2@geekplus.com.cn
Re: Re: NiFi backpressure not work
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
Thanks very much. I got it.
wanglei2@geekplus.com.cn
From: Joe Witt
Date: 2019-10-16 17:24
To: users@nifi.apache.org
Subject: Re: Re: NiFi backpressure not work
Lei
Please do not post to both users and dev list. Dropping dev.
Yes these scenarios are based on limitations of a given processor implementation or the nature of a given protocol/mechanism. NiFi enforces back pressure by slowing/stopping scheduling a component and making fact of back pressure available to components.
I dont know about this particular processor but it likely needs to be updated so the max amount of data it will return in any one cycle is small enough or controllable enough to avoud what youre seeing.
Thanks
Joe
On Wed, Oct 16, 2019 at 3:06 AM wanglei2@geekplus.com.cn <wa...@geekplus.com.cn> wrote:
CaptureChangeMySQLSelf is from CaptureChangeMySQL and I fix a small bug about binlog parsing. But i am sure there's no change that will affect the backpressure mechanism.
Actually I don't quite understand it.
It is related with the implementation of the processor? In order to use the backpressure mechanism, the processor must do something?
In my case, what it will happen if the queue continue to grow(for example to 100G)? Something bad will happen?
Thanks,
Lei
wanglei2@geekplus.com.cn
From: Josef.Zahner1@swisscom.com
Date: 2019-10-16 14:17
To: users@nifi.apache.org; dev@nifi.apache.org
Subject: Re: NiFi backpressure not work
Hi Lei
I assume it’s the same as for the ListSFTP. As I don’t know the CaptureChangeMySQLSelf processor I try to explain it for the ListSFTP. There with one single execution of the processor you get as much flowfiles as you have files on your disk. If it finds 1000 it sends out 1000 flowfiles. If it finds 1’000’000 it sends out 1’000’000 flowfiles. The processor can’t hold back some flows and knows nothing about your queue, so it sends out everything in one “batch” per execution – even if it is more than the queue size. Hope this explains the behavior a bit.
Cheers
From: "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>
Reply to: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 16 October 2019 at 08:10
To: users <us...@nifi.apache.org>, dev <de...@nifi.apache.org>
Subject: NiFi backpressure not work
As the picture shows.
I have set the backpressure object threshold to 10000, but picture shows there's more than 1,000,000. Why this happens?
Thanks, Lei
wanglei2@geekplus.com.cn
Re: Re: NiFi backpressure not work
Posted by Joe Witt <jo...@gmail.com>.
Lei
Please do not post to both users and dev list. Dropping dev.
Yes these scenarios are based on limitations of a given processor
implementation or the nature of a given protocol/mechanism. NiFi enforces
back pressure by slowing/stopping scheduling a component and making fact of
back pressure available to components.
I dont know about this particular processor but it likely needs to be
updated so the max amount of data it will return in any one cycle is small
enough or controllable enough to avoud what youre seeing.
Thanks
Joe
On Wed, Oct 16, 2019 at 3:06 AM wanglei2@geekplus.com.cn <
wanglei2@geekplus.com.cn> wrote:
> CaptureChangeMySQLSelf is from CaptureChangeMySQL and I fix a small bug
> about binlog parsing. But i am sure there's no change that will affect the
> backpressure mechanism.
>
> Actually I don't quite understand it.
> It is related with the implementation of the processor? In order to use
> the backpressure mechanism, the processor must do something?
> In my case, what it will happen if the queue continue to grow(for example
> to 100G)? Something bad will happen?
>
> Thanks,
> Lei
>
>
> ------------------------------
> wanglei2@geekplus.com.cn
>
>
> *From:* Josef.Zahner1@swisscom.com
> *Date:* 2019-10-16 14:17
> *To:* users@nifi.apache.org; dev@nifi.apache.org
> *Subject:* Re: NiFi backpressure not work
>
> Hi Lei
>
>
>
> I assume it’s the same as for the ListSFTP. As I don’t know the
> CaptureChangeMySQLSelf processor I try to explain it for the ListSFTP.
> There with one single execution of the processor you get as much flowfiles
> as you have files on your disk. If it finds 1000 it sends out 1000
> flowfiles. If it finds 1’000’000 it sends out 1’000’000 flowfiles. The
> processor can’t hold back some flows and knows nothing about your queue, so
> it sends out everything in one “batch” per execution – even if it is more
> than the queue size. Hope this explains the behavior a bit.
>
>
>
> Cheers
>
>
>
>
>
> *From: *"wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>
> *Reply to: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Wednesday, 16 October 2019 at 08:10
> *To: *users <us...@nifi.apache.org>, dev <de...@nifi.apache.org>
> *Subject: *NiFi backpressure not work
>
>
>
> As the picture shows.
>
> I have set the backpressure object threshold to 10000, but picture shows
> there's more than 1,000,000. Why this happens?
>
>
>
>
>
>
>
> Thanks, Lei
> ------------------------------
>
> wanglei2@geekplus.com.cn
>
>
Re: Re: NiFi backpressure not work
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
CaptureChangeMySQLSelf is from CaptureChangeMySQL and I fix a small bug about binlog parsing. But i am sure there's no change that will affect the backpressure mechanism.
Actually I don't quite understand it.
It is related with the implementation of the processor? In order to use the backpressure mechanism, the processor must do something?
In my case, what it will happen if the queue continue to grow(for example to 100G)? Something bad will happen?
Thanks,
Lei
wanglei2@geekplus.com.cn
From: Josef.Zahner1@swisscom.com
Date: 2019-10-16 14:17
To: users@nifi.apache.org; dev@nifi.apache.org
Subject: Re: NiFi backpressure not work
Hi Lei
I assume it’s the same as for the ListSFTP. As I don’t know the CaptureChangeMySQLSelf processor I try to explain it for the ListSFTP. There with one single execution of the processor you get as much flowfiles as you have files on your disk. If it finds 1000 it sends out 1000 flowfiles. If it finds 1’000’000 it sends out 1’000’000 flowfiles. The processor can’t hold back some flows and knows nothing about your queue, so it sends out everything in one “batch” per execution – even if it is more than the queue size. Hope this explains the behavior a bit.
Cheers
From: "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>
Reply to: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 16 October 2019 at 08:10
To: users <us...@nifi.apache.org>, dev <de...@nifi.apache.org>
Subject: NiFi backpressure not work
As the picture shows.
I have set the backpressure object threshold to 10000, but picture shows there's more than 1,000,000. Why this happens?
Thanks, Lei
wanglei2@geekplus.com.cn
Re: Re: NiFi backpressure not work
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
CaptureChangeMySQLSelf is from CaptureChangeMySQL and I fix a small bug about binlog parsing. But i am sure there's no change that will affect the backpressure mechanism.
Actually I don't quite understand it.
It is related with the implementation of the processor? In order to use the backpressure mechanism, the processor must do something?
In my case, what it will happen if the queue continue to grow(for example to 100G)? Something bad will happen?
Thanks,
Lei
wanglei2@geekplus.com.cn
From: Josef.Zahner1@swisscom.com
Date: 2019-10-16 14:17
To: users@nifi.apache.org; dev@nifi.apache.org
Subject: Re: NiFi backpressure not work
Hi Lei
I assume it’s the same as for the ListSFTP. As I don’t know the CaptureChangeMySQLSelf processor I try to explain it for the ListSFTP. There with one single execution of the processor you get as much flowfiles as you have files on your disk. If it finds 1000 it sends out 1000 flowfiles. If it finds 1’000’000 it sends out 1’000’000 flowfiles. The processor can’t hold back some flows and knows nothing about your queue, so it sends out everything in one “batch” per execution – even if it is more than the queue size. Hope this explains the behavior a bit.
Cheers
From: "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>
Reply to: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 16 October 2019 at 08:10
To: users <us...@nifi.apache.org>, dev <de...@nifi.apache.org>
Subject: NiFi backpressure not work
As the picture shows.
I have set the backpressure object threshold to 10000, but picture shows there's more than 1,000,000. Why this happens?
Thanks, Lei
wanglei2@geekplus.com.cn
Re: NiFi backpressure not work
Posted by Jo...@swisscom.com.
Hi Lei
I assume it’s the same as for the ListSFTP. As I don’t know the CaptureChangeMySQLSelf processor I try to explain it for the ListSFTP. There with one single execution of the processor you get as much flowfiles as you have files on your disk. If it finds 1000 it sends out 1000 flowfiles. If it finds 1’000’000 it sends out 1’000’000 flowfiles. The processor can’t hold back some flows and knows nothing about your queue, so it sends out everything in one “batch” per execution – even if it is more than the queue size. Hope this explains the behavior a bit.
Cheers
From: "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>
Reply to: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 16 October 2019 at 08:10
To: users <us...@nifi.apache.org>, dev <de...@nifi.apache.org>
Subject: NiFi backpressure not work
As the picture shows.
I have set the backpressure object threshold to 10000, but picture shows there's more than 1,000,000. Why this happens?
[cid:image001.jpg@01D583FA.26EAC380]
Thanks, Lei
________________________________
wanglei2@geekplus.com.cn
Re: NiFi backpressure not work
Posted by Jo...@swisscom.com.
Hi Lei
I assume it’s the same as for the ListSFTP. As I don’t know the CaptureChangeMySQLSelf processor I try to explain it for the ListSFTP. There with one single execution of the processor you get as much flowfiles as you have files on your disk. If it finds 1000 it sends out 1000 flowfiles. If it finds 1’000’000 it sends out 1’000’000 flowfiles. The processor can’t hold back some flows and knows nothing about your queue, so it sends out everything in one “batch” per execution – even if it is more than the queue size. Hope this explains the behavior a bit.
Cheers
From: "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>
Reply to: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 16 October 2019 at 08:10
To: users <us...@nifi.apache.org>, dev <de...@nifi.apache.org>
Subject: NiFi backpressure not work
As the picture shows.
I have set the backpressure object threshold to 10000, but picture shows there's more than 1,000,000. Why this happens?
[cid:image001.jpg@01D583FA.26EAC380]
Thanks, Lei
________________________________
wanglei2@geekplus.com.cn