You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@groovy.apache.org by MG <mg...@arscreat.com> on 2022/05/09 20:21:17 UTC

Re: Groovy 3 -> 4 Performance Degradation - Clojure/Jython/JRuby/... ?

Another question would be, whether anyone here knows whether other 
dynamic JVM languages such as Clojure/Jython/JRuby/... are
a) using Indy calls, and
b) are or have been facing similar performance problems... ?

On 09/05/2022 19:53, Jochen Theodorou wrote:
> On 09.05.22 17:41, MG wrote:
>> Hi Jochen,
>>
>> since I am not feeling well right now, just some quick answers to some
>> of your questions:
>>
>>  1. I am not at liberty to release the source code from our project, not
>>     even partially, so these JARs are currently all I can provide. I
>>     felt when I reported this problem initially, there were doubts that
>>     it even exists, so after I could not pinpoint the problem on my
>>     side, I invested the time to extract these project parts, clean them
>>     from any historical references to other modules etc and supply them
>>     in compiled form.
>
> I think I can test if it is really my assumption or not. I made myself a
> little program, that will create a class with many methods and one more
> method, that is calling all of them, one-by-one. The methods are empty
> and nothing takes parameters. I then put this in the compiler and
> measure times for creating the class and running the methods.
>
> For 1000 Methods I get on my machine these results for Groovy-3.0.9
> non-Indy:
> 1 Class: 36ms
> 10 Classes: 114ms
> 100 Classes: 985ms
> 1000 Classes: 8227ms
>
> the last case means, that 1+ million methods are executed inside of the
> test class
>
> Then I repeat the test with Groovy-4.0.2 Indy:
> 1  Class: 137ms        (* 3,80)
> 10 Classes: 670ms      (* 5,88)
> 100 Classes: 3711ms    (* 3,77)
> 1000 Classes: 108235ms (* 13,16)
>
> I would not classify these numbers as a proper test, but it does hint at
> what I had assumed... creating the initial callsites in indy is too
> expensive
>
> the script I used:
>
>> def CLASSES = args[0] as Integer
>> def METHODS = args[1] as Integer
>>
>>
>> def methods = ""
>> def calls   = ""
>> METHODS.times { i ->
>>   methods += "\ndef foo$i(){}"
>>   calls   += "\nfoo$i()"
>> }
>>
>> def gcl = new GroovyClassLoader();
>> def testClasses = (1..CLASSES).collect{
>>     def testClass = """
>>     class MyPerfTest$it{
>>       $methods
>>       def run() {
>>         $calls
>>       }
>>     }
>>     """
>>     gcl.parseClass(testClass)
>> }
>>
>>
>> def time1 = System.nanoTime()
>> for (clazz :testClasses) {
>>   clazz.newInstance().run()
>> }
>> def time2 = System.nanoTime()
>> def timeDiff = (time2-time1)/1000000
>> println "Time = $timeDiff ms"
>
> While this is surely not "real world" my goal was testing the callsite
> creation, and I think the test does this well enough. Why exactly this
> scenario? Because I know indy can perform well in micro-benchmarks,
> where you crunch a number in tight loops. and if there is no problem
> there, then it must be with the many callsites you tend to visit only
> once - like here.
>
>>  2. As I have said in previous posts, the performance degradation occurs
>>     with non-Indy vs Indy. In fact this is how I could pinpoint that
>>     Indy was to blame: After realizing that our test suite ran for 3h
>>     under Groovy 4, instead of 1h under Groovy 3, and that this was not
>>     restricted to a few tests becoming really slow, but was a more or
>>     less general phenomenon (which was puzzling), I ran the test suite
>>     under Groovy 3 Indy (for the first time), and got basically the same
>>     reduced performance.
>
> I had a suspicion for a long time already actually. Bad since I sadly
> have barely any time for Groovy these days... and checking this can take
> very very long.
>
> [...]
>>  4. Throwing away the first test loop: It is a bit of an acdemic
>>     discusssion, since Groovy 4 performance is always 2 to 3 times
>>     slower in any case. You can execute the test script interleaved
>>     between Grooyv 3 / 4 multiple times to see how measurements develop,
>>     but I would say it is clear that Groovy 4 is always way slower than
>>     Groovy 3 for this test, and that is all that matters :-)
>
> in my little test factor 2-3 times is for me factor 3-4 and that was my
> best case.
>
> bye Jochen