You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Jacques Nadeau <ja...@apache.org> on 2013/04/27 05:16:32 UTC
Janino performance versus javax.tools.JavaCompiler
I'm working on some of the bytecode stuff for Drill and found some
interesting random testing data:
Java6 adds a new JavaCompiler API. Janino supports utilizing that
instead of its internal parsing. The benefit of this is that all the
JDK 5 stuff like generics are fully supported. Since that seemed
nice, i figured we could use that.
In initial testing, I've actually found it substantially slower than
the Janino compiler. I ran some crappy microbenchmarks just to see
why my stuff was going so slow. Running a basic comparison test of
the simple ExpressionEvaluator example utilizing Janino versus JDK
showed an average compile and single invocation time of ~18000 micros
on JDK versus ~950 micros on Janino.
Has anyone else experimented with this/experienced this?
Example block below (straight from Janino docs).
ExpressionEvaluator ee = new ExpressionEvaluator("c > d ? c : d",
// expression
int.class, // expressionType
new String[] { "c", "d" }, // parameterNames
new Class[] { int.class, int.class } // parameterTypes
);
// Evaluate it with varying parameter values; very fast.
return (Integer) ee.evaluate(new Object[] { // parameterValues
new Integer(10), new Integer(11), });
Re: Janino performance versus javax.tools.JavaCompiler
Posted by Lisen Mu <im...@gmail.com>.
ok, here is what I've got so far:
Evaluate Janino: 882 micros. JDK: 1675 micros. Direct: 39 micros. DBody:
39 micros.
Compile Janino: 1122 micros. JDK: 28131 micros. Direct: 4074 micros.
DBody: 1457 micros.
please see the ClassBodyEvaluatorBuilder for 'DBody' implementation at
https://github.com/immars/incubator-drill/commit/79f7cbc7cb8e7c0b116032bba902343a47e53aa4
On Tue, May 21, 2013 at 10:58 PM, Lisen Mu <im...@gmail.com> wrote:
> If testing compile time alone, my result is like this:
>
> Janino: 1077 micros. JDK: 23533 micros. Direct: 3587 micros.
>
> ExpressionEvaluator approach is faster than SimpleCompiler approach. I
> think it's because ExpressionEvaluator parses only the expression string
> and puts together the rest of the class with janino objects. Maybe it's
> possible to get both better compile time and better evaluation time. I
> would try this later some time.
>
>
>
>
>
>
> On Tue, May 21, 2013 at 2:23 AM, Jacques Nadeau <ja...@apache.org>wrote:
>
>> Yeah, my test was entirely focused on compile time. In the case of
>> simple queries, the compile time differences would be important. In
>> larger queries, this would be less of an issue.
>>
>> j
>>
>>
>> On Mon, May 20, 2013 at 8:56 AM, Lisen Mu <im...@gmail.com> wrote:
>> > Jacques,
>> >
>> > I've tried Janino now, with your
>> > org.apache.drill.exec.compile.TestClassCompilationTypes,
>> >
>> > One thing is that the compilation time dominates the benchmark. If we
>> > compile once and benchmark evaluate() only, the performance of class
>> > compiled by JDK compiler is comparable with the class compiled by
>> Janino's
>> > embed compiler. JDK compiler seems to invoke javac which is more
>> > heavyweight than Janino.
>> >
>> > Another thing is that evalute() of ExpressionEvaluator by both Janino
>> and
>> > JDK implementation need to do reflect invocation once. I assume that in
>> the
>> > future utilization in drill, we would have several limited type of
>> > evaluate() method signature; so by using SimpleCompiler to compile the
>> > expression directly into a subclass of our Evaluator interface, we can
>> > fully avoid reflection.
>> >
>> > Here is my benchmark result:
>> >
>> > Janino: 882 micros. JDK: 1641 micros. Direct: 41 micros.
>> >
>> > Please see the
>> >
>> https://github.com/immars/incubator-drill/commit/34c0c3db7287904dbfdef9751e41e252b264dc88
>> >
>> >
>> >
>> >
>> > On Sun, Apr 28, 2013 at 10:25 AM, Ted Dunning <te...@gmail.com>
>> wrote:
>> >
>> >> Is the actual execution time after warming up the JIT the same?
>> >>
>> >>
>> >> On Fri, Apr 26, 2013 at 8:16 PM, Jacques Nadeau <ja...@apache.org>
>> >> wrote:
>> >>
>> >> > I'm working on some of the bytecode stuff for Drill and found some
>> >> > interesting random testing data:
>> >> >
>> >> > Java6 adds a new JavaCompiler API. Janino supports utilizing that
>> >> > instead of its internal parsing. The benefit of this is that all the
>> >> > JDK 5 stuff like generics are fully supported. Since that seemed
>> >> > nice, i figured we could use that.
>> >> >
>> >> > In initial testing, I've actually found it substantially slower than
>> >> > the Janino compiler. I ran some crappy microbenchmarks just to see
>> >> > why my stuff was going so slow. Running a basic comparison test of
>> >> > the simple ExpressionEvaluator example utilizing Janino versus JDK
>> >> > showed an average compile and single invocation time of ~18000 micros
>> >> > on JDK versus ~950 micros on Janino.
>> >> >
>> >> > Has anyone else experimented with this/experienced this?
>> >> >
>> >> > Example block below (straight from Janino docs).
>> >> >
>> >> > ExpressionEvaluator ee = new ExpressionEvaluator("c > d ? c :
>> d",
>> >> > // expression
>> >> > int.class, // expressionType
>> >> > new String[] { "c", "d" }, // parameterNames
>> >> > new Class[] { int.class, int.class } // parameterTypes
>> >> > );
>> >> >
>> >> > // Evaluate it with varying parameter values; very fast.
>> >> > return (Integer) ee.evaluate(new Object[] { // parameterValues
>> >> > new Integer(10), new Integer(11), });
>> >> >
>> >>
>>
>
>
Re: Janino performance versus javax.tools.JavaCompiler
Posted by Lisen Mu <im...@gmail.com>.
If testing compile time alone, my result is like this:
Janino: 1077 micros. JDK: 23533 micros. Direct: 3587 micros.
ExpressionEvaluator approach is faster than SimpleCompiler approach. I
think it's because ExpressionEvaluator parses only the expression string
and puts together the rest of the class with janino objects. Maybe it's
possible to get both better compile time and better evaluation time. I
would try this later some time.
On Tue, May 21, 2013 at 2:23 AM, Jacques Nadeau <ja...@apache.org> wrote:
> Yeah, my test was entirely focused on compile time. In the case of
> simple queries, the compile time differences would be important. In
> larger queries, this would be less of an issue.
>
> j
>
>
> On Mon, May 20, 2013 at 8:56 AM, Lisen Mu <im...@gmail.com> wrote:
> > Jacques,
> >
> > I've tried Janino now, with your
> > org.apache.drill.exec.compile.TestClassCompilationTypes,
> >
> > One thing is that the compilation time dominates the benchmark. If we
> > compile once and benchmark evaluate() only, the performance of class
> > compiled by JDK compiler is comparable with the class compiled by
> Janino's
> > embed compiler. JDK compiler seems to invoke javac which is more
> > heavyweight than Janino.
> >
> > Another thing is that evalute() of ExpressionEvaluator by both Janino and
> > JDK implementation need to do reflect invocation once. I assume that in
> the
> > future utilization in drill, we would have several limited type of
> > evaluate() method signature; so by using SimpleCompiler to compile the
> > expression directly into a subclass of our Evaluator interface, we can
> > fully avoid reflection.
> >
> > Here is my benchmark result:
> >
> > Janino: 882 micros. JDK: 1641 micros. Direct: 41 micros.
> >
> > Please see the
> >
> https://github.com/immars/incubator-drill/commit/34c0c3db7287904dbfdef9751e41e252b264dc88
> >
> >
> >
> >
> > On Sun, Apr 28, 2013 at 10:25 AM, Ted Dunning <te...@gmail.com>
> wrote:
> >
> >> Is the actual execution time after warming up the JIT the same?
> >>
> >>
> >> On Fri, Apr 26, 2013 at 8:16 PM, Jacques Nadeau <ja...@apache.org>
> >> wrote:
> >>
> >> > I'm working on some of the bytecode stuff for Drill and found some
> >> > interesting random testing data:
> >> >
> >> > Java6 adds a new JavaCompiler API. Janino supports utilizing that
> >> > instead of its internal parsing. The benefit of this is that all the
> >> > JDK 5 stuff like generics are fully supported. Since that seemed
> >> > nice, i figured we could use that.
> >> >
> >> > In initial testing, I've actually found it substantially slower than
> >> > the Janino compiler. I ran some crappy microbenchmarks just to see
> >> > why my stuff was going so slow. Running a basic comparison test of
> >> > the simple ExpressionEvaluator example utilizing Janino versus JDK
> >> > showed an average compile and single invocation time of ~18000 micros
> >> > on JDK versus ~950 micros on Janino.
> >> >
> >> > Has anyone else experimented with this/experienced this?
> >> >
> >> > Example block below (straight from Janino docs).
> >> >
> >> > ExpressionEvaluator ee = new ExpressionEvaluator("c > d ? c : d",
> >> > // expression
> >> > int.class, // expressionType
> >> > new String[] { "c", "d" }, // parameterNames
> >> > new Class[] { int.class, int.class } // parameterTypes
> >> > );
> >> >
> >> > // Evaluate it with varying parameter values; very fast.
> >> > return (Integer) ee.evaluate(new Object[] { // parameterValues
> >> > new Integer(10), new Integer(11), });
> >> >
> >>
>
Re: Janino performance versus javax.tools.JavaCompiler
Posted by Jacques Nadeau <ja...@apache.org>.
Yeah, my test was entirely focused on compile time. In the case of
simple queries, the compile time differences would be important. In
larger queries, this would be less of an issue.
j
On Mon, May 20, 2013 at 8:56 AM, Lisen Mu <im...@gmail.com> wrote:
> Jacques,
>
> I've tried Janino now, with your
> org.apache.drill.exec.compile.TestClassCompilationTypes,
>
> One thing is that the compilation time dominates the benchmark. If we
> compile once and benchmark evaluate() only, the performance of class
> compiled by JDK compiler is comparable with the class compiled by Janino's
> embed compiler. JDK compiler seems to invoke javac which is more
> heavyweight than Janino.
>
> Another thing is that evalute() of ExpressionEvaluator by both Janino and
> JDK implementation need to do reflect invocation once. I assume that in the
> future utilization in drill, we would have several limited type of
> evaluate() method signature; so by using SimpleCompiler to compile the
> expression directly into a subclass of our Evaluator interface, we can
> fully avoid reflection.
>
> Here is my benchmark result:
>
> Janino: 882 micros. JDK: 1641 micros. Direct: 41 micros.
>
> Please see the
> https://github.com/immars/incubator-drill/commit/34c0c3db7287904dbfdef9751e41e252b264dc88
>
>
>
>
> On Sun, Apr 28, 2013 at 10:25 AM, Ted Dunning <te...@gmail.com> wrote:
>
>> Is the actual execution time after warming up the JIT the same?
>>
>>
>> On Fri, Apr 26, 2013 at 8:16 PM, Jacques Nadeau <ja...@apache.org>
>> wrote:
>>
>> > I'm working on some of the bytecode stuff for Drill and found some
>> > interesting random testing data:
>> >
>> > Java6 adds a new JavaCompiler API. Janino supports utilizing that
>> > instead of its internal parsing. The benefit of this is that all the
>> > JDK 5 stuff like generics are fully supported. Since that seemed
>> > nice, i figured we could use that.
>> >
>> > In initial testing, I've actually found it substantially slower than
>> > the Janino compiler. I ran some crappy microbenchmarks just to see
>> > why my stuff was going so slow. Running a basic comparison test of
>> > the simple ExpressionEvaluator example utilizing Janino versus JDK
>> > showed an average compile and single invocation time of ~18000 micros
>> > on JDK versus ~950 micros on Janino.
>> >
>> > Has anyone else experimented with this/experienced this?
>> >
>> > Example block below (straight from Janino docs).
>> >
>> > ExpressionEvaluator ee = new ExpressionEvaluator("c > d ? c : d",
>> > // expression
>> > int.class, // expressionType
>> > new String[] { "c", "d" }, // parameterNames
>> > new Class[] { int.class, int.class } // parameterTypes
>> > );
>> >
>> > // Evaluate it with varying parameter values; very fast.
>> > return (Integer) ee.evaluate(new Object[] { // parameterValues
>> > new Integer(10), new Integer(11), });
>> >
>>
Re: Janino performance versus javax.tools.JavaCompiler
Posted by Lisen Mu <im...@gmail.com>.
Jacques,
I've tried Janino now, with your
org.apache.drill.exec.compile.TestClassCompilationTypes,
One thing is that the compilation time dominates the benchmark. If we
compile once and benchmark evaluate() only, the performance of class
compiled by JDK compiler is comparable with the class compiled by Janino's
embed compiler. JDK compiler seems to invoke javac which is more
heavyweight than Janino.
Another thing is that evalute() of ExpressionEvaluator by both Janino and
JDK implementation need to do reflect invocation once. I assume that in the
future utilization in drill, we would have several limited type of
evaluate() method signature; so by using SimpleCompiler to compile the
expression directly into a subclass of our Evaluator interface, we can
fully avoid reflection.
Here is my benchmark result:
Janino: 882 micros. JDK: 1641 micros. Direct: 41 micros.
Please see the
https://github.com/immars/incubator-drill/commit/34c0c3db7287904dbfdef9751e41e252b264dc88
On Sun, Apr 28, 2013 at 10:25 AM, Ted Dunning <te...@gmail.com> wrote:
> Is the actual execution time after warming up the JIT the same?
>
>
> On Fri, Apr 26, 2013 at 8:16 PM, Jacques Nadeau <ja...@apache.org>
> wrote:
>
> > I'm working on some of the bytecode stuff for Drill and found some
> > interesting random testing data:
> >
> > Java6 adds a new JavaCompiler API. Janino supports utilizing that
> > instead of its internal parsing. The benefit of this is that all the
> > JDK 5 stuff like generics are fully supported. Since that seemed
> > nice, i figured we could use that.
> >
> > In initial testing, I've actually found it substantially slower than
> > the Janino compiler. I ran some crappy microbenchmarks just to see
> > why my stuff was going so slow. Running a basic comparison test of
> > the simple ExpressionEvaluator example utilizing Janino versus JDK
> > showed an average compile and single invocation time of ~18000 micros
> > on JDK versus ~950 micros on Janino.
> >
> > Has anyone else experimented with this/experienced this?
> >
> > Example block below (straight from Janino docs).
> >
> > ExpressionEvaluator ee = new ExpressionEvaluator("c > d ? c : d",
> > // expression
> > int.class, // expressionType
> > new String[] { "c", "d" }, // parameterNames
> > new Class[] { int.class, int.class } // parameterTypes
> > );
> >
> > // Evaluate it with varying parameter values; very fast.
> > return (Integer) ee.evaluate(new Object[] { // parameterValues
> > new Integer(10), new Integer(11), });
> >
>
Re: Janino performance versus javax.tools.JavaCompiler
Posted by Ted Dunning <te...@gmail.com>.
Is the actual execution time after warming up the JIT the same?
On Fri, Apr 26, 2013 at 8:16 PM, Jacques Nadeau <ja...@apache.org> wrote:
> I'm working on some of the bytecode stuff for Drill and found some
> interesting random testing data:
>
> Java6 adds a new JavaCompiler API. Janino supports utilizing that
> instead of its internal parsing. The benefit of this is that all the
> JDK 5 stuff like generics are fully supported. Since that seemed
> nice, i figured we could use that.
>
> In initial testing, I've actually found it substantially slower than
> the Janino compiler. I ran some crappy microbenchmarks just to see
> why my stuff was going so slow. Running a basic comparison test of
> the simple ExpressionEvaluator example utilizing Janino versus JDK
> showed an average compile and single invocation time of ~18000 micros
> on JDK versus ~950 micros on Janino.
>
> Has anyone else experimented with this/experienced this?
>
> Example block below (straight from Janino docs).
>
> ExpressionEvaluator ee = new ExpressionEvaluator("c > d ? c : d",
> // expression
> int.class, // expressionType
> new String[] { "c", "d" }, // parameterNames
> new Class[] { int.class, int.class } // parameterTypes
> );
>
> // Evaluate it with varying parameter values; very fast.
> return (Integer) ee.evaluate(new Object[] { // parameterValues
> new Integer(10), new Integer(11), });
>