You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Edgardo Vega <ed...@gmail.com> on 2015/05/06 23:41:42 UTC
Provenance tuning
I am seeing warnings that provenance is slowing down the flows to allow it
to catch up. after reading the docs I am still not sure what should be done
to speed up provenance to make it go faster. Also is there a way to turn it
off to see the impact on performance?
Thanks,
Edgardo
--
Cheers,
Edgardo
Sent from Gmail Mobile
Re: Provenance tuning
Posted by Joe Witt <jo...@gmail.com>.
Ian,
You can effectively turn off the NiFi provenance reporter by switching
to an in-memory repository. So for example edit ./conf/nif.properties
Change this line
nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
To
nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository
You can also change this line to some smaller or other number to match
your needs.
nifi.provenance.repository.buffer.size=100000
Thanks
Joe
On Mon, Nov 30, 2015 at 2:21 PM, ianwork <ia...@gmail.com> wrote:
> My instance is running on a VM with 8cpu's. The repositories are all
> stored on the same virtual disks.
>
> a snapshot of io when the throttling kicks in
> Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
> vda 133.40 0.00 56982.40 0 284912
>
> On Mon, Nov 30, 2015 at 12:21 PM, Aldrin Piri [via Apache NiFi Developer
> List] <ml...@n7.nabble.com> wrote:
>
>> Ian,
>>
>> Could you provide information as to how your repositories are configured?
>> Specifically, what the layout is to physical volume(s)/disks?
>>
>> On Monday, November 30, 2015, ianwork <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=5524&i=0>> wrote:
>>
>> > I am also experiencing a slowdown with the provenance reporter and would
>> > like
>> > information on tuning. My application is performing log processing of
>> 20k
>> > flowfiles a second. As a workaround batch processing was implemented but
>> I
>> > would also like to see if it can be done without batching. I would
>> prefer
>> > to turn off the reporter, but since that is not possible, do you suggest
>> > that I move it to memory? And if so what setting would that be?
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> >
>> http://apache-nifi-developer-list.39713.n7.nabble.com/Provenance-tuning-tp1427p5522.html
>> > Sent from the Apache NiFi Developer List mailing list archive at
>> > Nabble.com.
>> >
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-nifi-developer-list.39713.n7.nabble.com/Provenance-tuning-tp1427p5524.html
>> To unsubscribe from Provenance tuning, click here
>> <http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1427&code=aWFud29yazAwMUBnbWFpbC5jb218MTQyN3wxMjM1MTk1NzEz>
>> .
>> NAML
>> <http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Provenance-tuning-tp1427p5525.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Provenance tuning
Posted by ianwork <ia...@gmail.com>.
My instance is running on a VM with 8cpu's. The repositories are all
stored on the same virtual disks.
a snapshot of io when the throttling kicks in
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
vda 133.40 0.00 56982.40 0 284912
On Mon, Nov 30, 2015 at 12:21 PM, Aldrin Piri [via Apache NiFi Developer
List] <ml...@n7.nabble.com> wrote:
> Ian,
>
> Could you provide information as to how your repositories are configured?
> Specifically, what the layout is to physical volume(s)/disks?
>
> On Monday, November 30, 2015, ianwork <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5524&i=0>> wrote:
>
> > I am also experiencing a slowdown with the provenance reporter and would
> > like
> > information on tuning. My application is performing log processing of
> 20k
> > flowfiles a second. As a workaround batch processing was implemented but
> I
> > would also like to see if it can be done without batching. I would
> prefer
> > to turn off the reporter, but since that is not possible, do you suggest
> > that I move it to memory? And if so what setting would that be?
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-nifi-developer-list.39713.n7.nabble.com/Provenance-tuning-tp1427p5522.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> > Nabble.com.
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-nifi-developer-list.39713.n7.nabble.com/Provenance-tuning-tp1427p5524.html
> To unsubscribe from Provenance tuning, click here
> <http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1427&code=aWFud29yazAwMUBnbWFpbC5jb218MTQyN3wxMjM1MTk1NzEz>
> .
> NAML
> <http://apache-nifi-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Provenance-tuning-tp1427p5525.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Provenance tuning
Posted by Aldrin Piri <al...@gmail.com>.
Ian,
Could you provide information as to how your repositories are configured?
Specifically, what the layout is to physical volume(s)/disks?
On Monday, November 30, 2015, ianwork <ia...@gmail.com> wrote:
> I am also experiencing a slowdown with the provenance reporter and would
> like
> information on tuning. My application is performing log processing of 20k
> flowfiles a second. As a workaround batch processing was implemented but I
> would also like to see if it can be done without batching. I would prefer
> to turn off the reporter, but since that is not possible, do you suggest
> that I move it to memory? And if so what setting would that be?
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Provenance-tuning-tp1427p5522.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Re: Provenance tuning
Posted by ianwork <ia...@gmail.com>.
I am also experiencing a slowdown with the provenance reporter and would like
information on tuning. My application is performing log processing of 20k
flowfiles a second. As a workaround batch processing was implemented but I
would also like to see if it can be done without batching. I would prefer
to turn off the reporter, but since that is not possible, do you suggest
that I move it to memory? And if so what setting would that be?
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Provenance-tuning-tp1427p5522.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Provenance tuning
Posted by Joe Witt <jo...@gmail.com>.
Edgardo
There isn't a totally 'off' version of provenance that I am aware of.
But you can certainly set it to be an in memory one only. I don't
have the settings for that handy but can look that up if you're
interested in walking that path.
Now having said this I'm really curious what your box/system
configuration is like and what your flow is like to run into this. We
only recently added this provenance back pressure capability. Perhaps
we don't have the knobs quite right. Can you share more details? For
easier sharing of templates, screenshots, logs, etc.. you may wish to
start a JIRA. Happy to help there.
Thanks
Joe
On Wed, May 6, 2015 at 5:41 PM, Edgardo Vega <ed...@gmail.com> wrote:
> I am seeing warnings that provenance is slowing down the flows to allow it
> to catch up. after reading the docs I am still not sure what should be done
> to speed up provenance to make it go faster. Also is there a way to turn it
> off to see the impact on performance?
>
> Thanks,
>
> Edgardo
>
>
> --
> Cheers,
>
> Edgardo
>
> Sent from Gmail Mobile