[scxml] profiling scxml commons

["R.C. Hoekstra" <r....@erasmusmc.nl>]

Hi all @ scxml commons. 

We had some profiling done on our scxml multi agent simulation project. Our issue is that scxml is too slow. We need about 100,000 instances of scxml engines for a multi agent simulation, and tests show that the approach with scxml is over 10 times slower than an alternative but less flexible approach. Still, the flexibility of the scxml commons project serves us very well, so we need to look for ways to improve the performance. 

For our project setup see my mailing to this list on Fri, 30 May, 20:45.

We have the following tips and improvements: 

1) Now, there's a logger applied for each instance, which is inefficient. 

Make default Log in SimpleContext static. In SCXMLExecutionContext/SCXMLExecutor, it can be static final even.
public class SimpleContext implements Context, Serializable {
   /** Implementation independent log category. */
   private static final Log DEFAULT_LOG = LogFactory.getLog(Context.class);
   private  Log log = DEFAULT_LOG;


2)  SimpleContext containsKey() + get() to get()

you can rewrite SimpleContext to:
   public Object get(final String name) {
    Object localValue = getVars().get(name);
       if (localValue!=null) {
return localValue;
       } else if (parent != null) {
           return parent.get(name);
       } else {
           return null;
       }
   }


We think item 1) & 2) will lead to a 30-50% improvement in our scenario, and they are very simple to apply. 


3) Other issues we are not sure about: 

- Status.getAllStates can probably be cached: Difficult to determine if this can be done since the states member is writable from outside and not sure when this getAllStates() cache should be invalidated. But it takes up quite some time.

- SCXMLSemanticsImpl.isLegalConfig is probably a paranoia check, can we disable it?


4) Some questions about further speed improvements: 

4a) scripting:

In the configuration as given, about 10% is spend in JEXL scripting. Depending on how much we want to do with that, we could switch to another scripting implementation (would require some benchmarking) or maybe move some of the scripts into the Java domain.

Do you guys have any clues about which of the provided scripting implementations is expected to be the fastest? Are there other possibilities to improve the scripting performance?


4b) initializing / common stuff:

What happens now is that each StateMachine engine (SCXMLExecutor) is instantiated a thousand times, once for each agent. Maybe it's possible to move some of the stuff to some common initialization where it is now done for each agent. Do you see any opportunities here?


Thanks, 
kind regards, 

Rinke Hoekstra.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org

Re: [scxml] profiling scxml commons

[Ate Douma <at...@douma.nu>]

On 18-06-14 09:42, R.C. Hoekstra wrote:
> Hi all @ scxml commons.
>
> We had some profiling done on our scxml multi agent simulation project. Our issue is that scxml is too slow. We need about 100,000 instances of scxml engines for a multi agent simulation, and tests show that the approach with scxml is over 10 times slower than an alternative but less flexible approach. Still, the flexibility of the scxml commons project serves us very well, so we need to look for ways to improve the performance.
>

Great, and already thanks a lot for the valuable feedback and testing.

I think there is definitely room for improvements, and I'd be happy to help out 
and already can take care of some of your suggestions below.

More inline comments below.


> For our project setup see my mailing to this list on Fri, 30 May, 20:45.
>
> We have the following tips and improvements:
>
> 1) Now, there's a logger applied for each instance, which is inefficient.
>
> Make default Log in SimpleContext static. In SCXMLExecutionContext/SCXMLExecutor, it can be static final even.
> public class SimpleContext implements Context, Serializable {
>     /** Implementation independent log category. */
>     private static final Log DEFAULT_LOG = LogFactory.getLog(Context.class);
>     private  Log log = DEFAULT_LOG;

Sure, will take care of this.

>
>
> 2)  SimpleContext containsKey() + get() to get()
>
> you can rewrite SimpleContext to:
>     public Object get(final String name) {
>      Object localValue = getVars().get(name);
>         if (localValue!=null) {
> return localValue;
>         } else if (parent != null) {
>             return parent.get(name);
>         } else {
>             return null;
>         }
>     }

+1, also a trivial improvement, thanks.

>
>
> We think item 1) & 2) will lead to a 30-50% improvement in our scenario, and they are very simple to apply.
>
>
> 3) Other issues we are not sure about:
>
> - Status.getAllStates can probably be cached: Difficult to determine if this can be done since the states member is writable from outside and not sure when this getAllStates() cache should be invalidated. But it takes up quite some time.
Yeah, I think we can add some caching here, but as you already indicate, the 
underlying states map is writable so it really depends on how often the 
#getAllStates() method is invoked (extra) between updates against the states map 
how much gain you'll receive.
But it is something I'll look into and try to add some caching.

>
> - SCXMLSemanticsImpl.isLegalConfig is probably a paranoia check, can we disable it?
It is a paranoia check yes, so if you are sure all your possible statemachine 
transitions will end up legal anyhow, this check could be disabled.
I can add an option to configure this, but will keep it enabled by default.
Of course, you already could customize (extend) the current SCXMLSemanticsImpl 
right now and overwrite this method to do nothing right now.
But adding a configurable option for this is a simply improvement I'm happy to add.

>
>
> 4) Some questions about further speed improvements:
>
> 4a) scripting:
>
> In the configuration as given, about 10% is spend in JEXL scripting. Depending on how much we want to do with that, we could switch to another scripting implementation (would require some benchmarking) or maybe move some of the scripts into the Java domain.
>
> Do you guys have any clues about which of the provided scripting implementations is expected to be the fastest? Are there other possibilities to improve the scripting performance?

I suggest trying out and testing the Groovy scripting instead. For *our* 
use-cases it turned out to be faster (about 30%), but it expect it depends on 
the specific evaluations you need to do.
If you try out the Groovy scripting it would be great if you can report back 
your results here.

>
>
> 4b) initializing / common stuff:
>
> What happens now is that each StateMachine engine (SCXMLExecutor) is instantiated a thousand times, once for each agent. Maybe it's possible to move some of the stuff to some common initialization where it is now done for each agent. Do you see any opportunities here?
>
This'll take some more time to look into.
It's probably possible to restructure the setup a bit to be more optimal for 
your use-case, but I need to think about the consequences for other use-cases as 
well. And take into account other/future functional changes which are still 
needed to bring the Commons SCXML further in line with the specification.

To be honest about this, I haven't had much time the last two months to work on 
Commons SCXML, too busy with other obligations.
But I expect to have more time in a few weeks time.

The easy improvements you proposed above, I will pick up shortly though within 
the coming week or so.

Thank again for the feedback and keep it coming :)

Regards,

Ate

>
> Thanks,
> kind regards,
>
> Rinke Hoekstra.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org