You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@groovy.apache.org by "Daniel.Sun" <su...@apache.org> on 2017/12/22 09:48:32 UTC
About the native lambda
Hi all,
I tried to generate bytecode for native lambda. Currently some basic
tests pass[1]
The problem I am trying to resolve is how to get the method's name
and descriptor? [2] It seems the type flow analysis have to rely on the
existing static compilation...
Any help is appreciated!
[1]
https://github.com/danielsun1106/groovy/blob/native-lambda/subprojects/parser-antlr4/src/test/resources/core/Lambda_02x.groovy
[2]
https://github.com/danielsun1106/groovy/blob/native-lambda/src/main/java/org/codehaus/groovy/classgen/asm/LambdaWriter.java#L75-L76
Cheers,
Daniel.Sun
--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
RE: About the native lambda
Posted by "Daniel.Sun" <su...@apache.org>.
Groovy community is so warm!
Thank you all for your help, I'll set aside more time to investigate further
:-)
Cheers,
Daniel.Sun
--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
RE: About the native lambda
Posted by Uwe Schindler <us...@apache.org>.
I forgot to mention the entry point to Painless’s main bootstrapper for everything involvig its DEF type (method calls, operators, array accesses, list accesses, load/store,…:
https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/DefBootstrap.java
Uwe
-----
Uwe Schindler
uschindler@apache.org
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/
From: Uwe Schindler [mailto:uschindler@apache.org]
Sent: Friday, December 22, 2017 2:53 PM
To: dev@groovy.apache.org
Cc: forax@univ-mlv.fr
Subject: RE: About the native lambda
Hi,
you can have a look at Elasticsearch’s Painless scripting language. It has both variants: Static Lambdas and dynamic Lambdas on DEF type (code calls a method on the def type that takes a lambda/function ref). In the latter case the actual call to the method taking the lambda is delegated through the standard invokedynamic we use for DEF (the DefBootstrap.java is the main bootstrap entry point for all dynamic stuff involving DEF type, it creates several caches like a polymorpic for most cases, a megamporphic using classvalue and for 2 types operators it uses a monomorphic cache for scaling reasons).
If you call a method on DEF where you give it a lambda or method reference, nothing is known. In that case it goes through the above “master bootstrap” where it resolves the correct method according to types and the just calls the LambdaBootstrap without invokedynamic from Java code (it is just a call to the lambda bootstrapper):
https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/Def.java#L371-L381
It just uses a different LambdaBootstrapper (LAMBDA_BOOTSTRAP_HANDLE), because starting with Java 9 the default one - shipped with Java - got more picky and did not work for all required type conversions. So we have our own LambdaMetaFactory (but that’s unrelated, in Java 10 the bugs should be fixed and we may fall back to the Java one):
https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/LambdaBootstrap.java
This one uses a cool trick to do the type conversions – a second invokedynamic inside the generated lambda functional interface method implementation (that does the type conversion for us with “as Type”), see:
* https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/LambdaBootstrap.java#L434-L440
* https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/LambdaBootstrap.java#L501-L514
If you have a method call taking a lambda where everything is known (the “Java case”), it writes a standard invokedynamic like Java does in the generated bytecode:
One place for lambda is here:
https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ELambda.java#L207-L238
(there are others for static complilation, e.g. for method refs).
That’s the “default java case”, just with our own bootstrapper!
We are currently writing a blog post about this, but it’s hard to explain for somebody with no knowledge about bytecode and how lambdas work in a single 5 page blog post!
Uwe
-----
Uwe Schindler
uschindler@apache.org <ma...@apache.org>
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/
From: Remi Forax [mailto:forax@univ-mlv.fr]
Sent: Friday, December 22, 2017 12:53 PM
To: dev <dev@groovy.apache.org <ma...@groovy.apache.org> >
Cc: dev@groovy.incubator.apache.org <ma...@groovy.incubator.apache.org>
Subject: Re: About the native lambda
_____
De: "Jesper Steen Møller" <jesper@selskabet.org <ma...@selskabet.org> >
À: "dev" <dev@groovy.apache.org <ma...@groovy.apache.org> >
Cc: dev@groovy.incubator.apache.org <ma...@groovy.incubator.apache.org>
Envoyé: Vendredi 22 Décembre 2017 11:26:06
Objet: Re: About the native lambda
On 22 Dec 2017, at 10.48, Daniel.Sun <sunlan@apache.org <ma...@apache.org> > wrote:
The problem I am trying to resolve is how to get the method's name
and descriptor? [2] It seems the type flow analysis have to rely on the
existing static compilation...
There is no short cut possible here: You need full static analysis to make lambdas Java-compatible and emit the right invokedynamic calls. And the type inference is really complicated w.r.t overloads.
You can read about it here: http://cr.openjdk.java.net/~dlsmith/jsr335-0.6.1/F.html
-Jesper
There is a "shortcut" possible, like usually with a dynamic language, you can postpone the creation until you have enough data.
So you create a lambda object but not by emiting the invokedynamic on the LambdaMetaFactory, but by creating a new Lambda object (it can be the same object as a Closure or not) and when you need to coerce it to a functional interface at that time you call the invokedynamic on the LambdaMetaFactory.
To be able to call the LambdaMetaFactory, the Lambda object need to store a MethodHandle on a static method (a constant method handle) and the Lookup object used to create the method handle.
cheers,
Rémi
RE: About the native lambda
Posted by Uwe Schindler <us...@apache.org>.
Hi,
you can have a look at Elasticsearch’s Painless scripting language. It has both variants: Static Lambdas and dynamic Lambdas on DEF type (code calls a method on the def type that takes a lambda/function ref). In the latter case the actual call to the method taking the lambda is delegated through the standard invokedynamic we use for DEF (the DefBootstrap.java is the main bootstrap entry point for all dynamic stuff involving DEF type, it creates several caches like a polymorpic for most cases, a megamporphic using classvalue and for 2 types operators it uses a monomorphic cache for scaling reasons).
If you call a method on DEF where you give it a lambda or method reference, nothing is known. In that case it goes through the above “master bootstrap” where it resolves the correct method according to types and the just calls the LambdaBootstrap without invokedynamic from Java code (it is just a call to the lambda bootstrapper):
https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/Def.java#L371-L381
It just uses a different LambdaBootstrapper (LAMBDA_BOOTSTRAP_HANDLE), because starting with Java 9 the default one - shipped with Java - got more picky and did not work for all required type conversions. So we have our own LambdaMetaFactory (but that’s unrelated, in Java 10 the bugs should be fixed and we may fall back to the Java one):
https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/LambdaBootstrap.java
This one uses a cool trick to do the type conversions – a second invokedynamic inside the generated lambda functional interface method implementation (that does the type conversion for us with “as Type”), see:
* https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/LambdaBootstrap.java#L434-L440
* https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/LambdaBootstrap.java#L501-L514
If you have a method call taking a lambda where everything is known (the “Java case”), it writes a standard invokedynamic like Java does in the generated bytecode:
One place for lambda is here:
https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ELambda.java#L207-L238
(there are others for static complilation, e.g. for method refs).
That’s the “default java case”, just with our own bootstrapper!
We are currently writing a blog post about this, but it’s hard to explain for somebody with no knowledge about bytecode and how lambdas work in a single 5 page blog post!
Uwe
-----
Uwe Schindler
uschindler@apache.org
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/
From: Remi Forax [mailto:forax@univ-mlv.fr]
Sent: Friday, December 22, 2017 12:53 PM
To: dev <de...@groovy.apache.org>
Cc: dev@groovy.incubator.apache.org
Subject: Re: About the native lambda
_____
De: "Jesper Steen Møller" <jesper@selskabet.org <ma...@selskabet.org> >
À: "dev" <dev@groovy.apache.org <ma...@groovy.apache.org> >
Cc: dev@groovy.incubator.apache.org <ma...@groovy.incubator.apache.org>
Envoyé: Vendredi 22 Décembre 2017 11:26:06
Objet: Re: About the native lambda
On 22 Dec 2017, at 10.48, Daniel.Sun <sunlan@apache.org <ma...@apache.org> > wrote:
The problem I am trying to resolve is how to get the method's name
and descriptor? [2] It seems the type flow analysis have to rely on the
existing static compilation...
There is no short cut possible here: You need full static analysis to make lambdas Java-compatible and emit the right invokedynamic calls. And the type inference is really complicated w.r.t overloads.
You can read about it here: http://cr.openjdk.java.net/~dlsmith/jsr335-0.6.1/F.html
-Jesper
There is a "shortcut" possible, like usually with a dynamic language, you can postpone the creation until you have enough data.
So you create a lambda object but not by emiting the invokedynamic on the LambdaMetaFactory, but by creating a new Lambda object (it can be the same object as a Closure or not) and when you need to coerce it to a functional interface at that time you call the invokedynamic on the LambdaMetaFactory.
To be able to call the LambdaMetaFactory, the Lambda object need to store a MethodHandle on a static method (a constant method handle) and the Lookup object used to create the method handle.
cheers,
Rémi
Re: About the native lambda
Posted by Remi Forax <fo...@univ-mlv.fr>.
> De: "Jesper Steen Møller" <je...@selskabet.org>
> À: "dev" <de...@groovy.apache.org>
> Cc: dev@groovy.incubator.apache.org
> Envoyé: Vendredi 22 Décembre 2017 11:26:06
> Objet: Re: About the native lambda
>> On 22 Dec 2017, at 10.48, Daniel.Sun < [ mailto:sunlan@apache.org |
>> sunlan@apache.org ] > wrote:
>> The problem I am trying to resolve is how to get the method's name
>> and descriptor? [2] It seems the type flow analysis have to rely on the
>> existing static compilation...
> There is no short cut possible here: You need full static analysis to make
> lambdas Java-compatible and emit the right invokedynamic calls. And the type
> inference is really complicated w.r.t overloads.
> You can read about it here: [
> http://cr.openjdk.java.net/~dlsmith/jsr335-0.6.1/F.html |
> http://cr.openjdk.java.net/~dlsmith/jsr335-0.6.1/F.html ]
> -Jesper
There is a "shortcut" possible, like usually with a dynamic language, you can postpone the creation until you have enough data.
So you create a lambda object but not by emiting the invokedynamic on the LambdaMetaFactory, but by creating a new Lambda object (it can be the same object as a Closure or not) and when you need to coerce it to a functional interface at that time you call the invokedynamic on the LambdaMetaFactory.
To be able to call the LambdaMetaFactory, the Lambda object need to store a MethodHandle on a static method (a constant method handle) and the Lookup object used to create the method handle.
cheers,
Rémi
Re: About the native lambda
Posted by Jesper Steen Møller <je...@selskabet.org>.
> On 22 Dec 2017, at 10.48, Daniel.Sun <su...@apache.org> wrote:
>
> The problem I am trying to resolve is how to get the method's name
> and descriptor? [2] It seems the type flow analysis have to rely on the
> existing static compilation...
>
There is no short cut possible here: You need full static analysis to make lambdas Java-compatible and emit the right invokedynamic calls. And the type inference is really complicated w.r.t overloads.
You can read about it here: http://cr.openjdk.java.net/~dlsmith/jsr335-0.6.1/F.html
-Jesper