You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by pradeepbill <pr...@gmail.com> on 2016/06/21 16:07:05 UTC
ConvertJSONToAvro processor is slow?
Hi there, I am seeing huge delays in ConvertJSONToAvro processor where the
converted data trickles down to the next step.Please see attachment.I have
increased the concurrent tasks =100. Is there anything else I can
configure/tune up.
nifi_Capture.PNG
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/n11957/nifi_Capture.PNG>
Thanks
Pradeep
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertJSONToAvro-processor-is-slow-tp11957.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: ConvertJSONToAvro processor is slow?
Posted by pradeepbill <pr...@gmail.com>.
Thanks Joe, I could not respond back soon, because I was verifying and
validating the details from this post.
You said, ConvertJSONtoAvro at 270 messages per second and ReplaceText is
processing at 70 messages per second.
>How did you do the math here ?
StoreInKite is doing very very little. And I doubt back pressure is a
factor in this config as-is.
>I am saving very little data here, will confirm though.
Will update the thread soon with more details.
THanks
Pradeep
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertJSONToAvro-processor-is-slow-tp11957p12101.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: ConvertJSONToAvro processor is slow?
Posted by Joe Witt <jo...@gmail.com>.
Couple of things to notice just from the screenshot...
As Andy points out the store in kite dataset processor appears to be
doing very little. The preceding two processors are
ConvertJSONtoAvro at 270 messages per second. The resulting data is
far smaller than the input which is interesting.
ReplaceText is processing at 70 messages per second
StoreInKite is doing very very little. And I doubt back pressure is a
factor in this config as-is.
Neither of those processors appear to be actively executing threads
when this snapshot was taken. That makes sense since neither has much
of a queue.
ListenSyslog has 4 active threads. You likely won't need that many.
I would avoid adding 100 parallel threads to any processor.
But as for speed of the convert and replace they seem to be going as
fast as needed to receive the data...
On Tue, Jun 21, 2016 at 1:18 PM, Andy LoPresto <al...@apache.org> wrote:
> It looks like the StoreInKiteDataset processor is the issue. It has read
> 1.85 MB in, but written 0 B out. What are the back pressure settings on that
> processor?
>
>
> Andy LoPresto
> alopresto@apache.org
> alopresto.apache@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Jun 21, 2016, at 9:07 AM, pradeepbill <pr...@gmail.com> wrote:
>
> Hi there, I am seeing huge delays in ConvertJSONToAvro processor where the
> converted data trickles down to the next step.Please see attachment.I have
> increased the concurrent tasks =100. Is there anything else I can
> configure/tune up.
>
> nifi_Capture.PNG
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n11957/nifi_Capture.PNG>
>
> Thanks
> Pradeep
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertJSONToAvro-processor-is-slow-tp11957.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
>
>
Re: ConvertJSONToAvro processor is slow?
Posted by Andy LoPresto <al...@apache.org>.
Yeah, I noticed that as well after sending the message. This could still be because of back pressure on the Kite processor (if the Kite processor is set to delay incoming data until it processes a certain amount / the incoming queue is below a certain threshold, the previous processor will yield its execution until that level is met).
Are there any errors in the logs/nifi-app.log file?
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 21, 2016, at 9:13 AM, pradeepbill <pr...@gmail.com> wrote:
>
> Thanks Andy, but I see ConvertJSONToAvro read 680MB and wrote 2.86MB to
> disk in ConvertJSONToAvro, is it not true ?
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertJSONToAvro-processor-is-slow-tp11957p11960.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: ConvertJSONToAvro processor is slow?
Posted by pradeepbill <pr...@gmail.com>.
Thanks Andy, but I see ConvertJSONToAvro read 680MB and wrote 2.86MB to
disk in ConvertJSONToAvro, is it not true ?
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertJSONToAvro-processor-is-slow-tp11957p11960.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: ConvertJSONToAvro processor is slow?
Posted by Andy LoPresto <al...@apache.org>.
It looks like the StoreInKiteDataset processor is the issue. It has read 1.85 MB in, but written 0 B out. What are the back pressure settings on that processor?
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 21, 2016, at 9:07 AM, pradeepbill <pr...@gmail.com> wrote:
>
> Hi there, I am seeing huge delays in ConvertJSONToAvro processor where the
> converted data trickles down to the next step.Please see attachment.I have
> increased the concurrent tasks =100. Is there anything else I can
> configure/tune up.
>
> nifi_Capture.PNG
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n11957/nifi_Capture.PNG>
>
> Thanks
> Pradeep
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertJSONToAvro-processor-is-slow-tp11957.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.