[jira] [Commented] (NIFI-3406) Multiple Improvements needed to ScriptedProcessor's ScriptEngines configuration/information

["Junegunn Choi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415215#comment-16415215 ] 

Junegunn Choi commented on NIFI-3406:
-------------------------------------

NiFi fails to properly initialize JRuby engines on startup if the data flow has multiple processors using JRuby engine. With {{ExecuteScript}} and {{InvokeScriptedProcessor}} both configured to use ruby engine, I kept running into some strange errors like so:

 
{noformat}
ERROR [NiFi Web Server-68] o.a.n.p.script.InvokeScriptedProcessor InvokeScriptedProcessor[id=50d5a7ea-0162-1000-8d5d-dd233c9a9ff5] Error adding script engine JSR 223 JRuby Engine
WARN [NiFi Web Server-68] o.a.n.controller.StandardProcessorNode Failed during validation
org.apache.nifi.processor.exception.ProcessException: No script engine available!

ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=519f9c79-0162-1000-00b7-f99c3dceaccd] ExecuteScript[id=519f9c79-0162-1000-00b7-f99c3dceaccd] failed to process due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: org.jruby.embed.EvalFailedException: (NameError) uninitialized constant REL_SUCCESS; rolling back session: {}
org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: org.jruby.embed.EvalFailedException: (NameError) uninitialized constant REL_SUCCESS

ERROR [Timer-Driven Process Thread-9] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=519f9c79-0162-1000-00b7-f99c3dceaccd] ExecuteScript[id=519f9c79-0162-1000-00b7-f99c3dceaccd] failed to process due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: org.jruby.embed.EvalFailedException: java.lang.NullPointerException; rolling back session: {}
{noformat}
Turned out it was due to the non-thread-safe nature of the default JRuby context, and I was able to fix the issue by adding the following configuration to {{bootstrap.conf}}.
{noformat}
# Use thread-safe JRuby context
java.arg.jruby=-Dorg.jruby.embed.localcontext.scope=threadsafe
{noformat}
Although it's trivial to workaround the issue as above, I strongly believe that the configuration should be the default and I'm doubtful that anyone would need to override the default. We can consider implementing {{ScriptEngineConfigurator}} for JRuby, but a configurator is only used after its engine is created, so we would have to add a new method to the interface, such as {{prepare()}}, so we can call it before creating an engine. Or we can simply add the above configuration to the default {{bootstrap.conf}}. What do you think?

 

 
----
Related:
 * [https://github.com/jruby/jruby/issues/3731]
 * [https://github.com/jruby/jruby/wiki/RedBridge#threadsafe]

> Multiple Improvements needed to ScriptedProcessor's ScriptEngines configuration/information
> -------------------------------------------------------------------------------------------
>
>                 Key: NIFI-3406
>                 URL: https://issues.apache.org/jira/browse/NIFI-3406
>             Project: Apache NiFi
>          Issue Type: Improvement
>    Affects Versions: 1.1.1
>            Reporter: Joseph Percivall
>            Priority: Major
>
> The issue prompting this is that JRuby's default configuration uses a Singleton scope[1]. So all threads will see the same runtime and variables. This means if a FlowFile is currently running within the body of a script, and a new one comes in, the first thread will use the new variables (session, flowfile, etc.). This is obviously a problem and can lead "flowfile not known in this session" and livelocks in the FF repo due to multiple sessions updating the same flowfile[2].
> The Scripted processors should: 
>    1: give more information about the script engine being run
>    2: provide more configuration options for the script engine it self
>    3: warn the user about the nuances of script engines and how they are not maintained by Apache NiFi so may not follow our best practices.
> [1] http://jruby.org/apidocs/org/jruby/embed/ScriptingContainer.html
> Specifically the line: "However, the default, singleton local context scope behave slightly different. If Ruby runtime has been already instantiated by another ScriptingContainer, application, etc, the same runtime will be used."
> [2] https://github.com/JPercivall/nifi/blob/1be08714731f01347ac1f98e18047fe7d9ab8afd/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/MinimalLockingWriteAheadLog.java#L69-L69



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)