You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Joe Witt <jo...@gmail.com> on 2016/10/27 12:08:30 UTC
Re: Nifi ExecuteScript slow performance
Bala,
Are you saying that step 2 (executing the script) is taking three
seconds? Is that per message? Can you show the log or screenshot of
how you're tracking that?
Thanks
Joe
On Thu, Oct 27, 2016 at 6:02 AM, balacode63 <ba...@gmail.com> wrote:
> Dear All,
>
> I've added a ExecuteScript in python. This script is a simple script which
> will just add one attribute to the flowfile. The flow is definded as below,
>
> 1. Listen from mqtt (ConsumeMQTT)
> 2. Add attribute to nifi flow (Execute script)
> 3. Write to a file/post/ or some custom logic
>
> But its taking 3 seconds to process. Please guide me.
>
> Thanks,
> Bala
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Nifi ExecuteScript slow performance
Posted by balacode63 <ba...@gmail.com>.
Thanks for the reply Joe,
As matt suggested we moved to groovy.
Thanks
Bala
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13782.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Nifi ExecuteScript slow performance
Posted by balacode63 <ba...@gmail.com>.
Thanks for the reply matt,
Jython is slow, As you suggested we moved to groovy. ExecuteScript is taking
more than 2 seconds to execute below high level logic
The logic:
1) Split data from flowfile
2) Query influxdb with splitted data (influxdb jar file used/external
dependency)
3) write to influxdb
4) write flow file attributes
Without influxdb calling, the code is fast. Can you suggest me about
improving the performance of the below mentioned groovy script
Thanks
Bala
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13781.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Nifi ExecuteScript slow performance
Posted by Mike Thomsen <mi...@gmail.com>.
Vyshali,
Another trick you can do that works very well if you have a lot of
flowfiles to process is use session.get(int) to grab a batch. Obviously,
keep the volume you grab tuned to reasonable memory limits and all that,
but you can use that to make the script do a lot more work in one single
run. I have a flow that ends up processing millions of JSON files, and
grabbing 500 of them at once and merging them into a JSON array to create a
record batch really helps.
On Mon, Nov 13, 2017 at 12:23 PM, Vyshali <vy...@honeywell.com> wrote:
> Thank you so much Matt.
> I will try the solutions provided and come back in case of questions.
>
> Thanks,
> Vyshali
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
Re: Nifi ExecuteScript slow performance
Posted by Vyshali <vy...@honeywell.com>.
Thank you so much Matt.
I will try the solutions provided and come back in case of questions.
Thanks,
Vyshali
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Nifi ExecuteScript slow performance
Posted by Matt Burgess <ma...@apache.org>.
Vyshali,
Jython itself is known to be relatively slow, so using the scripting
processors you'll always be up against that. I see a couple of options
to improve performance:
1) Use InvokeScriptedProcessor (ISP) instead of ExecuteScript. ISP is
faster because it only loads the script once, then invokes methods on
it, rather than ExecuteScript which evaluates the script each time. I
have an ISP template in Jython [1] which should make porting your
ExecuteScript code easier.
2) Use ExecuteStreamCommand with command-line Python instead. You
won't have the flexibility of accessing attributes, processor state,
etc. but if you're just transforming content you should find
ExecuteStreamCommand with Python faster.
Regards,
Matt
[1] http://funnifi.blogspot.com/2017/11/invokescriptedprocessor-template.html
On Sun, Nov 12, 2017 at 7:31 PM, Vyshali <vy...@honeywell.com> wrote:
> Hi Matt,
>
> I'm using Jython in executescript because of my requirement.I cant switch to
> groovy because I'm using packages supported by Python.Is there any way to
> increase the speed of the executescript processor.Please help me with your
> ideas.
>
> Thanks,
> Vyshali
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Nifi ExecuteScript slow performance
Posted by Vyshali <vy...@honeywell.com>.
Hi Matt,
I'm using Jython in executescript because of my requirement.I cant switch to
groovy because I'm using packages supported by Python.Is there any way to
increase the speed of the executescript processor.Please help me with your
ideas.
Thanks,
Vyshali
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Nifi ExecuteScript slow performance
Posted by Matt Burgess <ma...@apache.org>.
Bala,
Jython in ExecuteScript is noticably slower than other languages like
Javascript and Groovy, but it shouldn't be that slow. Can you share
your script code? Also, is this just to investigate before writing a
more complex script in ExecuteScript, or do you just want to add an
attribute to a flowfile? If the latter, UpdateAttribute can do that,
but I suspect there is more to it than that (custom logic, e.g.).
Regards,
Matt
On Thu, Oct 27, 2016 at 8:08 AM, Joe Witt <jo...@gmail.com> wrote:
> Bala,
>
> Are you saying that step 2 (executing the script) is taking three
> seconds? Is that per message? Can you show the log or screenshot of
> how you're tracking that?
>
> Thanks
> Joe
>
> On Thu, Oct 27, 2016 at 6:02 AM, balacode63 <ba...@gmail.com> wrote:
>> Dear All,
>>
>> I've added a ExecuteScript in python. This script is a simple script which
>> will just add one attribute to the flowfile. The flow is definded as below,
>>
>> 1. Listen from mqtt (ConsumeMQTT)
>> 2. Add attribute to nifi flow (Execute script)
>> 3. Write to a file/post/ or some custom logic
>>
>> But its taking 3 seconds to process. Please guide me.
>>
>> Thanks,
>> Bala
>>
>>
>>
>> --
>> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html
>> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.