[jira] [Commented] (NIFI-6229) support GraalVM

["Dirk Arends (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17728556#comment-17728556 ] 

Dirk Arends commented on NIFI-6229:
-----------------------------------

I’ve been attempting to get GraalVM working with the latest snapshot (2.0.0) of NiFi (or any version for that matter) with limited success. The following is a summary of the progress made in the hope anyone else is able to assist.

*Versions:*
 - graalvm-ce-java17-22.3.2
 - NiFi 2.0.0-SNAPSHOT (fork)
 - Maven 3.9.2 (mvn) [1] 

h2. Test setup steps:

1. Download GraalVM from their github released page [2]
2. Install js into the GrallVM [3]
{code:java}
/<path>/<to>/graalvm-ce-java17-22.3.1/bin/gu install js
{code}
3. Clone my fork of NiFi 2.0.0-SNAPSHOT [4]
4. Build NiFi. In the root dir of the source tree:
{code:java}
export JAVA_HOME=/<path>/<to>/graalvm-ce-java17-22.3.1
mvn -T 2 clean install -Pinclude-grpc -DskipTests
{code}
5. Add {{java.arg.jsengine}} to {{./nifi-assembly/target/nifi-2.0.0-SNAPSHOT-bin/nifi-2.0.0-SNAPSHOT/conf.bootstrap.conf}}
{code:java}
java.arg.jsengine=-Dpolyglot.js.nashorn-compat=true
{code}
6. Run NiFi
{code:java}
export JAVA_HOME=/<path>/<to>/graalvm-ce-java17-22.3.1
./nifi-assembly/target/nifi-2.0.0-SNAPSHOT-bin/nifi-2.0.0-SNAPSHOT/bin/nifi.sh run
{code}
7. Import the attached flow definition [^Graal_Testing_Examples.json]
h2. Troubleshooting:

If ExecuteScript or InvokeScriptedProcessor processors throw errors saying there is no script engine, this means either:

1. {{JAVA_HOME}} is not set to GraalVM so NiFi is not running on GraalVM (NiFi lists the {{{{JAVA_HOME}}}} target in the terminal when the run command is used)
2. JavaScript is not installed on GraalVM. (see Test setup steps #2)
h2. Test issue 1 - Unsupported operation identifier 'iterator' and object '[object Array]':
 # Complete steps from above
 # The sample includes two process groups
 ## NiFi example scripts - scripts originally from NiFis test scripts [5]
 --- test_reader.js
 --- test_onTrigger.js
 --- testScriptROutesToFailure.js
 --- testInvokeScriptCausesException.js
 ## Minimal sample scripts
 --- Basic ExecuteScript
 --- Basic InvokeScriptedProcessor
 --- Basic InvokeScriptedProcessor (Built with Rollup to allow us to use TypeScript and newer ES features)
 # Each process group loosely has
 ## Imported scripts with with no changes
 ## Modified scripts to include a workaround for the following error. I had to change all occurrences of initialising an array with {{[]}} to {{{}new Java.type('java.util.ArrayList')(){}}}:

{code:java}
16:14:27 AEST ERROR
InvokeScriptedProcessor[id=053180d8-05e2-311d-ebd7-1665877a86e7] Unable to get relationships from scripted Processor: org.graalvm.polyglot.PolyglotException: Unsupported operation identifier 'iterator' and object '[object Array]'(language: JavaScript, type: Array). Identifier is not executable or instantiable.
{code}
{code:java}
16:14:26 AEST ERROR
InvokeScriptedProcessor[id=053180d8-05e2-311d-ebd7-1665877a86e7] Unable to validate the script Processor: java.lang.UnsupportedOperationException: Unsupported operation identifier 'size' and object '[object Array]'(language: JavaScript, type: Array). Identifier is not executable or instantiable.: java.lang.UnsupportedOperationException: Unsupported operation identifier 'size' and object '[object Array]'(language: JavaScript, type: Array). Identifier is not executable or instantiable.
 - Caused by: Attached Guest Language Frames (1)
{code}
I’ve found references to issues with casting GraalVM variables incorrectly between JavaScript and Java [6]. I was not able to replicate this issue using a standalone GraalVM script
{code:java}
./graalvm-ce-17/bin/javac Standalone_Graal_Test.java -truffle
./graalvm-ce-17/bin/java Standalone_Graal_Test -Dpolyglot.js.nashorn-compat=true
{code}
[^Standalone_Graal_Test.java]

In the long term, having to use this workaround is not ideal because part of the appeal to using GraalVM is the newer versions of supported Javascript.
h2. Test issue 2 - Multi threaded access requested by thread but is not allowed for language(s) js:

Following on from Test Issue #1.3. All scripts with the array fix are now able to run. Unfortunately they will sometimes (fairly regularly) complain about multithreading not being allowed.
{code:java}
16:16:14 AEST ERROR
InvokeScriptedProcessor[id=cd0d8770-801f-3790-ac2b-33f47dd0059c] Unable to get relationships from scripted Processor: java.lang.IllegalStateException: Multi threaded access requested by thread Thread[NiFi Web Server-152,5,main] but is not allowed for language(s) js.
{code}
It seems that different NiFi threads are trying to access the script object simultaneously which seems to indicate a larger underlying issue with the differences between Nashorn and GraalVM. While some solutions are documented I'm not sure how they can apply to NiFi.
{quote}the main reason why we do not allow two concurrent threads to access the same Context at the same time is thread safety. In Nashorn, you could share any object between threads (e.g., the global object) leading to unexpected data races. We don't have concrete plans to support concurrent access to the same context at the moment, but we are investigating alternative solutions that might allow multi-threaded JS code (like in your examples) to run on Graal.js. We cannot estimate yet when such support will land.
[7]
{quote}
Below is some relevant background information on the issue:

[https://medium.com/graalvm/multi-threaded-java-javascript-language-interoperability-in-graalvm-2f19c1f9c37b]
[https://medium.com/swlh/porting-from-nashorn-how-to-handle-multi-threading-in-graal-js-957e359b7df5]
[https://github.com/iitsoftware/graaljs-concurrency-problem]

[1] [https://maven.apache.org/install.html]
[2] [https://github.com/graalvm/graalvm-ce-builds/releases/tag/vm-22.3.2]
[3] [https://www.graalvm.org/22.3/reference-manual/js/]
[4] [https://github.com/Aerilym/nifi/tree/graal] 
[5] [https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/test/resources/javascript]
[6] [https://github.com/oracle/graaljs/issues/3#issuecomment-555661056]
[7] [https://github.com/oracle/graaljs/issues/59#issuecomment-423183994]

> support GraalVM
> ---------------
>
>                 Key: NIFI-6229
>                 URL: https://issues.apache.org/jira/browse/NIFI-6229
>             Project: Apache NiFi
>          Issue Type: Wish
>            Reporter: Behrouz
>            Priority: Minor
>              Labels: features
>         Attachments: Graal_Testing_Examples.json, Standalone_Graal_Test.java
>
>
> supporting GraalVM ([https://www.graalvm.org/]) would be very helpful for Nodejs, R, C++, ... programmer to work with Nifi.
> "GraalVM is a universal virtual machine for running applications written in JavaScript, Python, Ruby, R, JVM-based languages like Java, Scala, Kotlin, Clojure, and LLVM-based languages such as C and C++.
> GraalVM removes the isolation between programming languages and enables interoperability in a shared runtime. It can run either standalone or in the context of OpenJDK, Node.js, Oracle Database, or MySQL"



--
This message was sent by Atlassian Jira
(v8.20.10#820010)