Re: Nifi ExecuteScript slow performance

[Vyshali <vy...@honeywell.com>]

Hi Matt,

I'm using Jython in executescript because of my requirement.I cant switch to
groovy because I'm using packages supported by Python.Is there any way to
increase the speed of the executescript processor.Please help me with your
ideas.

Thanks,
Vyshali



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Nifi ExecuteScript slow performance

[Mike Thomsen <mi...@gmail.com>]

Vyshali,

Another trick you can do that works very well if you have a lot of
flowfiles to process is use session.get(int) to grab a batch. Obviously,
keep the volume you grab tuned to reasonable memory limits and all that,
but you can use that to make the script do a lot more work in one single
run. I have a flow that ends up processing millions of JSON files, and
grabbing 500 of them at once and merging them into a JSON array to create a
record batch really helps.

On Mon, Nov 13, 2017 at 12:23 PM, Vyshali <vy...@honeywell.com> wrote:

> Thank you so much Matt.
> I will try the solutions provided and come back in case of questions.
>
> Thanks,
> Vyshali
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: Nifi ExecuteScript slow performance

[Vyshali <vy...@honeywell.com>]

Thank you so much Matt.
I will try the solutions provided and come back in case of questions.

Thanks,
Vyshali



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Nifi ExecuteScript slow performance

[Matt Burgess <ma...@apache.org>]

Vyshali,

Jython itself is known to be relatively slow, so using the scripting
processors you'll always be up against that. I see a couple of options
to improve performance:

1) Use InvokeScriptedProcessor (ISP) instead of ExecuteScript. ISP is
faster because it only loads the script once, then invokes methods on
it, rather than ExecuteScript which evaluates the script each time.  I
have an ISP template in Jython [1] which should make porting your
ExecuteScript code easier.
2) Use ExecuteStreamCommand with command-line Python instead. You
won't have the flexibility of accessing attributes, processor state,
etc. but if you're just transforming content you should find
ExecuteStreamCommand with Python faster.

Regards,
Matt

[1] http://funnifi.blogspot.com/2017/11/invokescriptedprocessor-template.html


On Sun, Nov 12, 2017 at 7:31 PM, Vyshali <vy...@honeywell.com> wrote:
> Hi Matt,
>
> I'm using Jython in executescript because of my requirement.I cant switch to
> groovy because I'm using packages supported by Python.Is there any way to
> increase the speed of the executescript processor.Please help me with your
> ideas.
>
> Thanks,
> Vyshali
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/