You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@groovy.apache.org by "K Adithyan (tech)" <ad...@gmail.com> on 2018/02/07 15:14:43 UTC
Executing Groovy DSL scripts concurrently
Team,
Our application is a Root Cause Analysis Computation Engine for Telecom
networks. We have been implementing the logics in java only so far.
We are now planning to write DSLs for each feature and write the logics
using the DSL. We have implemented a framework for plugging in any number
of DSLs and integrated that DSL Framework with our application.
Each of the logics written over the DSL are executed concurrently using 40
or 50 or 60 threads based on the rate of the incoming events. Due to the
concurrency, we are now creating `GroovyShell` object for every execution
and calling `shell.evaluate(ourScriptFile)` to run the DSL scripts with our
own Binding object and delegates.
This prevents the system to run the logic at its own speed. Everytime
parsing and running takes significant time which reduces the event handling
capacity of the application.
I have attempted with `GroovyShell` and `GroovyScriptEngine`. But both are
offering same performance levels. GroovyShell is slightly better than
GroovyScriptEngine.
Under these circumstances, what is the best way to wrap our script logic,
that is based on our own DSLs, and run it concurrently without any overhead
for parsing, compilation, etc?????????
Pls advice
Reg,
Adithyan K
Re: Executing Groovy DSL scripts concurrently
Posted by Daniel Sun <re...@hotmail.com>.
Hi Adithyan,
How about caching the class instance and reuse it? Here are some
scratch code for your reference:
```
static LRUCache<String, Class> lruCache = new LRUCache<String, Class>(100);
def dslSrc = '''
your DSL source code
'''
def md5 = md5(dslSrc)
def resultClass = lruCache.getAndPut(md5, k -> new GroovyShell().parse
dslSrc)
```
Cheers,
Daniel.Sun
--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Users-f329450.html
RE: Executing Groovy DSL scripts concurrently
Posted by Korbee Reinout <Re...@snb.ch>.
Hi,
According to: http://docs.groovy-lang.org/latest/html/documentation/guide-integrating.html
“You must be very careful when using shared data in a multithreaded environment. The Binding instance that you pass to GroovyShell is not thread safe, and shared by all scripts.”
There are some guidelines regarding multi-threading and thread-safety. Before attempting something like this, ask yourself the question, “can I guarantee that my code is thread safe?”, which is very different from running a successful test without concurrency issues.
You could use the GroovyClassLoader directly and keep the compiled scripts in a cache, see the mentioned docs on how to obtain thread-safety with the Binding object.
From: K Adithyan (tech) [mailto:adi.k.dev@gmail.com]
Sent: Wednesday, February 7, 2018 4:15 PM
To: users@groovy.apache.org
Subject: Executing Groovy DSL scripts concurrently
Team,
Our application is a Root Cause Analysis Computation Engine for Telecom networks. We have been implementing the logics in java only so far.
We are now planning to write DSLs for each feature and write the logics using the DSL. We have implemented a framework for plugging in any number of DSLs and integrated that DSL Framework with our application.
Each of the logics written over the DSL are executed concurrently using 40 or 50 or 60 threads based on the rate of the incoming events. Due to the concurrency, we are now creating `GroovyShell` object for every execution and calling `shell.evaluate(ourScriptFile)` to run the DSL scripts with our own Binding object and delegates.
This prevents the system to run the logic at its own speed. Everytime parsing and running takes significant time which reduces the event handling capacity of the application.
I have attempted with `GroovyShell` and `GroovyScriptEngine`. But both are offering same performance levels. GroovyShell is slightly better than GroovyScriptEngine.
Under these circumstances, what is the best way to wrap our script logic, that is based on our own DSLs, and run it concurrently without any overhead for parsing, compilation, etc?????????
Pls advice
Reg,
Adithyan K
This e-mail message contains confidential information and is intended only for the named recipient(s). If you are not an intended recipient, any disclosure, copying or distribution is prohibited. Please notify the sender immediately by e-mail if you have received this message in error and delete this message from your system. As internet communications are not secure, the Swiss National Bank accepts no liability for any errors or omissions in the contents of this message. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of the Swiss National Bank.
RE: Executing Groovy DSL scripts concurrently
Posted by Korbee Reinout <Re...@snb.ch>.
I use the Groovy Script as a sort of function that returns a specific type and call it repeatedly with different bindings, the following is an excerpt of my code:
private static final GroovyClassLoader GROOVY_CLASSLOADER = new GroovyClassLoader();
private static final Map<String, Script> functionCache = new HashMap<>();
public static MyReturnType evaluate(String functionString, Object myDataToBind) {
Script script = null;
try {
script = resolveFunction(functionString);
} catch (Exception e) {
throw new IllegalArgumentException("The function could not be resolved: " + functionString, e);
}
script.getBinding().setVariable(VAR, myDataToBind);
Object o = null;
try {
o = script.run();
} catch (Exception e) {
throw new IllegalArgumentException("The function is not valid: " + functionString, e);
}
MyReturnType ts = null;
if(o instanceof MyReturnType) {
ts = (MyReturnType) o;
} else {
throw new IllegalArgumentException("The function doesn't evaluate to a MyReturnType: " + functionString);
}
return ts;
}
Script resolveFunction(String f) throws InstantiationException, IllegalAccessException {
if(functionCache.containsKey(f)) { // functions are likely repeated many times in one chunk
return functionCache.get(f);
}
// fiddle a bit with the string representation of the script
Class<?> clazz = GROOVY_CLASSLOADER.parseClass(f);
Object cc = clazz.newInstance();
Script script = (Script) cc;
functionCache.put(f, script);
return script;
}
From: Ralph Johnson [mailto:rjohnson.uiuc@gmail.com]
Sent: Tuesday, February 13, 2018 3:41 AM
To: users@groovy.apache.org
Subject: Re: Executing Groovy DSL scripts concurrently
That guide to integrating is *very* interesting! I had never seen GroovyClassLoader before.
I work with a large application that uses GroovyShell a lot. It has thousands of scripts. Sometimes they call each other recursively, which runs into the same problem with shared Bindings that concurrency has. I wondered why a script couldn't be a class so that I could instantiate it each time it ran, and so not have to share bindings. I am pretty sure that with GroovyClassLoader, I can do it. Thanks for pointing it out!
-Ralph Johnson
On Mon, Feb 12, 2018 at 2:10 AM, Korbee Reinout <Re...@snb.ch>> wrote:
[Boxbe]<https://www.boxbe.com/overview>[http://www.boxbe.com/stfopen?tc_serial=36637963201&tc_rand=1084693400&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001]This message is eligible for Automatic Cleanup! (Reinout.Korbee@snb.ch<ma...@snb.ch>) Add cleanup rule<https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3DIACL3oJKfUFUI8o%252BV%252BSvhRz7CVhfgVAef%252BSiCMBkkH0%253D%26token%3DmOLRlI1UuZw8SVNtchg7i08qve4GprNHZPJ%252FuFCSHStD%252F7QwDC7CiD1J%252FXpeem8HBaBmBiSO7lwH0XUlOXqrrSd4Lxlv%252BVtlgDCgdh9yWPHzz1U9J3EMThUh%252B8pDNwMq4JUlD72gGHFJICg5jMje9A%253D%253D&tc_serial=36637963201&tc_rand=1084693400&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> | More info<http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=36637963201&tc_rand=1084693400&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
Hi,
According to: http://docs.groovy-lang.org/latest/html/documentation/guide-integrating.html
“You must be very careful when using shared data in a multithreaded environment. The Binding instance that you pass to GroovyShell is not thread safe, and shared by all scripts.”
There are some guidelines regarding multi-threading and thread-safety. Before attempting something like this, ask yourself the question, “can I guarantee that my code is thread safe?”, which is very different from running a successful test without concurrency issues.
You could use the GroovyClassLoader directly and keep the compiled scripts in a cache, see the mentioned docs on how to obtain thread-safety with the Binding object.
From: K Adithyan (tech) [mailto:adi.k.dev@gmail.com<ma...@gmail.com>]
Sent: Wednesday, February 7, 2018 4:15 PM
To: users@groovy.apache.org<ma...@groovy.apache.org>
Subject: Executing Groovy DSL scripts concurrently
Team,
Our application is a Root Cause Analysis Computation Engine for Telecom networks. We have been implementing the logics in java only so far.
We are now planning to write DSLs for each feature and write the logics using the DSL. We have implemented a framework for plugging in any number of DSLs and integrated that DSL Framework with our application.
Each of the logics written over the DSL are executed concurrently using 40 or 50 or 60 threads based on the rate of the incoming events. Due to the concurrency, we are now creating `GroovyShell` object for every execution and calling `shell.evaluate(ourScriptFile)` to run the DSL scripts with our own Binding object and delegates.
This prevents the system to run the logic at its own speed. Everytime parsing and running takes significant time which reduces the event handling capacity of the application.
I have attempted with `GroovyShell` and `GroovyScriptEngine`. But both are offering same performance levels. GroovyShell is slightly better than GroovyScriptEngine.
Under these circumstances, what is the best way to wrap our script logic, that is based on our own DSLs, and run it concurrently without any overhead for parsing, compilation, etc?????????
Pls advice
Reg,
Adithyan K
This e-mail message contains confidential information and is intended only for the named recipient(s). If you are not an intended recipient, any disclosure, copying or distribution is prohibited. Please notify the sender immediately by e-mail if you have received this message in error and delete this message from your system. As internet communications are not secure, the Swiss National Bank accepts no liability for any errors or omissions in the contents of this message. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of the Swiss National Bank.
Re: Executing Groovy DSL scripts concurrently
Posted by Ralph Johnson <rj...@gmail.com>.
That guide to integrating is *very* interesting! I had never seen
GroovyClassLoader before.
I work with a large application that uses GroovyShell a lot. It has
thousands of scripts. Sometimes they call each other recursively, which
runs into the same problem with shared Bindings that concurrency has. I
wondered why a script couldn't be a class so that I could instantiate it
each time it ran, and so not have to share bindings. I am pretty sure
that with GroovyClassLoader, I can do it. Thanks for pointing it out!
-Ralph Johnson
On Mon, Feb 12, 2018 at 2:10 AM, Korbee Reinout <Re...@snb.ch>
wrote:
> [image: Boxbe] <https://www.boxbe.com/overview> This message is eligible
> for Automatic Cleanup! (Reinout.Korbee@snb.ch) Add cleanup rule
> <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3DIACL3oJKfUFUI8o%252BV%252BSvhRz7CVhfgVAef%252BSiCMBkkH0%253D%26token%3DmOLRlI1UuZw8SVNtchg7i08qve4GprNHZPJ%252FuFCSHStD%252F7QwDC7CiD1J%252FXpeem8HBaBmBiSO7lwH0XUlOXqrrSd4Lxlv%252BVtlgDCgdh9yWPHzz1U9J3EMThUh%252B8pDNwMq4JUlD72gGHFJICg5jMje9A%253D%253D&tc_serial=36637963201&tc_rand=1084693400&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
> | More info
> <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=36637963201&tc_rand=1084693400&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
>
> Hi,
>
>
>
> According to: http://docs.groovy-lang.org/latest/html/documentation/
> guide-integrating.html
>
>
>
> “You must be very careful when using shared data in a multithreaded
> environment. The Binding instance that you pass to GroovyShell is *not*
> thread safe, and shared by all scripts.”
>
>
>
> There are some guidelines regarding multi-threading and thread-safety.
> Before attempting something like this, ask yourself the question, “can I
> guarantee that my code is thread safe?”, which is very different from
> running a successful test without concurrency issues.
>
>
>
> You could use the GroovyClassLoader directly and keep the compiled scripts
> in a cache, see the mentioned docs on how to obtain thread-safety with the
> Binding object.
>
>
>
> *From:* K Adithyan (tech) [mailto:adi.k.dev@gmail.com]
> *Sent:* Wednesday, February 7, 2018 4:15 PM
> *To:* users@groovy.apache.org
> *Subject:* Executing Groovy DSL scripts concurrently
>
>
>
> Team,
>
>
>
> Our application is a Root Cause Analysis Computation Engine for Telecom
> networks. We have been implementing the logics in java only so far.
>
>
>
> We are now planning to write DSLs for each feature and write the logics
> using the DSL. We have implemented a framework for plugging in any number
> of DSLs and integrated that DSL Framework with our application.
>
>
>
> Each of the logics written over the DSL are executed concurrently using 40
> or 50 or 60 threads based on the rate of the incoming events. Due to the
> concurrency, we are now creating `GroovyShell` object for every execution
> and calling `shell.evaluate(ourScriptFile)` to run the DSL scripts with
> our own Binding object and delegates.
>
>
>
> This prevents the system to run the logic at its own speed. Everytime
> parsing and running takes significant time which reduces the event handling
> capacity of the application.
>
>
> I have attempted with `GroovyShell` and `GroovyScriptEngine`. But both are
> offering same performance levels. GroovyShell is slightly better than
> GroovyScriptEngine.
>
>
>
> Under these circumstances, what is the best way to wrap our script logic,
> that is based on our own DSLs, and run it concurrently without any overhead
> for parsing, compilation, etc?????????
>
>
>
>
>
> Pls advice
>
>
> Reg,
>
> Adithyan K
>
> This e-mail message contains confidential information and is intended only
> for the named recipient(s). If you are not an intended recipient, any
> disclosure, copying or distribution is prohibited. Please notify the sender
> immediately by e-mail if you have received this message in error and delete
> this message from your system. As internet communications are not secure,
> the Swiss National Bank accepts no liability for any errors or omissions in
> the contents of this message. Any views expressed in this message are those
> of the individual sender, except where the sender specifically states them
> to be the views of the Swiss National Bank.
>
>