You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by Arif Kasim <ar...@google.com> on 2018/10/05 17:09:17 UTC

Java > 8 support

Hello,
What's the status of java version > 8 support for beam? Thanks.

-Arif.

Re: Java > 8 support

Posted by Andrew Pilloud <ap...@google.com>.

+1 to adding Jenkins jobs that test with Java 11. Might it also be possible
to add a second java precommit that runs a subset of the tests on Java 11?

Andrew

On Thu, Oct 18, 2018 at 8:52 AM Scott Wegner <sc...@apache.org> wrote:

> If we just want to validate our produced artifacts work with Java11, I
> believe it would be easy to start running some of our Jenkins jobs on Java
> 11. beam_PostRelease_NightlyCandidate [1] is a good candidate: it runs the
> quickstart examples across all runners from the SNAPSHOT published maven
> architype [2].
>
> [1] https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/
> [2]
> https://github.com/apache/beam/blob/master/release/src/main/groovy/QuickstartArchetype.groovy
>
> On Thu, Oct 18, 2018 at 1:26 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Yes, Kenn I was talking about building artifacts (because this is the
>> strictest way to validate the compatiblity of our code base and find issues
>> with some of the classpath/code generation we use or with the
>> dependencies). I agree 100% with you, the real deal is to validate that we
>> can use Java 8 built jars with more recent Java versions without issues and
>> examples could be a good target for this.
>>
>> Supporting all non LTS java versions seems like a lot of work for a small
>> return: (1) most users will probably stay in LTS versions, (2) not any
>> single execution system has talked about supporting intermediary versions
>> (some not even LTS) (3) this will augment the combination of tests we
>> should execute to validate a release. Of course if we do care only about
>> direct runner this could make sense.
>>
>> On Thu, Oct 18, 2018 at 9:30 AM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> On Thu, Oct 18, 2018 at 4:55 AM Kenneth Knowles <ke...@apache.org> wrote:
>>>
>>>> Just to add to what Luke said: The reason we had those Java 8-only
>>>> modules was because some underlying tech (example: Gearpump) required Java
>>>> 8. If an engine requires something then it is OK for a user who chooses the
>>>> runner for that engine to also be subject to that requirement. Otherwise I
>>>> would push back a bit on specialized modules.
>>>>
>>>
>>> I, too, would avoid getting back into the situation where we have
>>> different modules with different build/runtime requirements. (Java 8 was
>>> special both because of Gearpump and because it introduced features such as
>>> lambdas, type inference, and other functional libraries that were
>>> particularly well suited to our programming model).
>>>
>>>
>>>> Ismaël: Do we need to be thinking in terms of building artifacts? I
>>>> think the important thing is if a user has a project and just pulls our
>>>> jars from maven central that they can use JRE 11 to run it. I think what we
>>>> need is a build matrix of user pipelines compiled and run w/ Java 8, 9, 10,
>>>> 11 (or some subset). This can be obtained via the examples maven archetype,
>>>> for example, having no relation at all to our own build tooling. I know
>>>> Jenkins has a plugin for this, so we don't run into the "mega-Jenkins-job"
>>>> problem.
>>>>
>>>
>>> +1 for rephrasing it this way. In general one would expect our jars of
>>> classfiles to work with later JREs. The magic we do with annotations and
>>> bytecode generation may, however, require more support and testing.
>>>
>>>
>>>> On Wed, Oct 17, 2018 at 2:03 PM Lukasz Cwik <lc...@google.com> wrote:
>>>>
>>>>> When we were supporting Java 7, there were Java 8 based modules in
>>>>> Maven. We can follow a similar pattern where Java11 (or any other version)
>>>>> have their own projects and as the base Java version moves up, we can merge
>>>>> those modules back in to simplify the set of modules being developed.
>>>>>
>>>>> On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <ie...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I want to give more context on the previous work towards Java 11
>>>>>> support. The issue was not necessarily that gradle was not ok to support
>>>>>> Java 11, but this is also a good point Lukasz.
>>>>>>
>>>>>> Months ago, we built a maven profile that basically set up Java to be
>>>>>> strictly compatible with the ‘minimal’ java.base profile and used Java
>>>>>> version (9/10) at the time. We found and fixed some missing dependencies
>>>>>> (because Java 11 removed some of the Java EE stuff from the base profile),
>>>>>> not sure if these changes were migrated to the gradle build.
>>>>>>
>>>>>> We also dealt with a good chunk of updates at the time of
>>>>>> plugins/bytebuddy and other deps to guarantee that the execution was done
>>>>>> strictly with the latest version of Java and its generated bytecode (the
>>>>>> origin of the docker build images was to guarantee this isolation). At the
>>>>>> time I run the tests of most IOs and most extensions with the existing
>>>>>> (java 8 compiled) direct runner in JRE 10 with success but of course this
>>>>>> was not the ultimate goal but a good way to evaluate the state of the code
>>>>>> base (I remember some GCP tests were broken and the ApiSurfaceTests too).
>>>>>>
>>>>>> One important thing to remember is that the goal on this work was
>>>>>> ‘limited’ to guarantee that a user that had ONLY Java 11 installed could
>>>>>> build and generate Beam artifacts and run Beam programs in the direct
>>>>>> runner. The limited scope was in part because of some issues of a full
>>>>>> migration to Java 11:
>>>>>>
>>>>>> - Support for Java modules, this is a too much radical change and
>>>>>> probably far from the scope of Beam at the moment.
>>>>>> - Support for other runners, given that most execution systems do not
>>>>>> support yet Java 11 this could imply a lot of extra work. In the meantime,
>>>>>> this could be easier now due to the portability work, anyway, still not
>>>>>> sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon.
>>>>>>
>>>>>> Let’s not forget that we still are and probably will be for a long
>>>>>> time backwards compatible with Java 8 so we need to ensure that Java 11
>>>>>> only things don’t get into the code base.
>>>>>>
>>>>>> If you Lukasz or someone else is up to take this task, I think the
>>>>>> immediate thing to do is to get back to the same state we were before the
>>>>>> move to gradle, build the equivalent of the maven profile for the gradle
>>>>>> build and tackle code generation and classpath issues, then try to fix
>>>>>> missing parts and push this into the CI too (similar to the recent python 3
>>>>>> work) to guarantee that new changes don’t break Java 11 compatibility. I
>>>>>> sadly lack of bandwidth to lead this task but I will gladly help with
>>>>>> reviews, or feedback if someone else is up to the task.
>>>>>>
>>>>>> Not sure about more recent details, but probably worth evaluating at
>>>>>> some moment is if we there is some interest on supporting multi-release
>>>>>> jars with Java 11 classes or the dream scenario of having a AOT build of
>>>>>> the direct runner.
>>>>>>
>>>>>> On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy <
>>>>>> lukasz.gajowy@gmail.com> wrote:
>>>>>>
>>>>>>> FYI: If I'm correct, migrating to Java 11 may require Migrating to
>>>>>>> Gradle 5, which to me is ok, but there are some issues on the way (such as
>>>>>>> [1] or maybe more that I'm not aware of).
>>>>>>>
>>>>>>> I did some research and it seems that Groovy 2.4.15 (the default
>>>>>>> version of Groovy for Gradle 4.10.2) is unable to work with Java 11 due to
>>>>>>> Nest based access control problem [2]. The very same error happened when I
>>>>>>> tried to compile "core" module with java11. This issue is solved in Groovy
>>>>>>> 2.5.3 [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3
>>>>>>> (NoClassDefError) and found out that the support for Groovy 2.5.3 will be
>>>>>>> added in Gradle 5.0RC1 (milestone) [4].
>>>>>>>
>>>>>>> Other than that I had some issues with spottless and findbugs (I
>>>>>>> turned them off for the sake of my experiment).
>>>>>>>
>>>>>>> Łukasz
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-4042
>>>>>>> [2]
>>>>>>> https://github.com/gradle/gradle/issues/5120#issuecomment-424610114
>>>>>>> [3] https://issues.apache.org/jira/browse/GROOVY-8727
>>>>>>> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw>
>>>>>>> [4] https://github.com/gradle/gradle/pull/7376
>>>>>>> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q>
>>>>>>>
>>>>>>>
>>>>>>> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lu...@gmail.com>
>>>>>>> napisał(a):
>>>>>>>
>>>>>>>> FWIW, the issue regarding java 11 support to gradle might be
>>>>>>>> helpful here [1]. Quoting: "there are only some minor corner cases that
>>>>>>>> don't work. Overall Java 11 support is pretty solid already" but some users
>>>>>>>> reported problems in comments with Checkstyle plugin and MacOS +
>>>>>>>> Gradle4.10.2 (this might be important for us).
>>>>>>>>
>>>>>>>> Łukasz
>>>>>>>>
>>>>>>>> [1] https://github.com/gradle/gradle/issues/5120
>>>>>>>>
>>>>>>>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pa...@google.com>
>>>>>>>> napisał(a):
>>>>>>>>
>>>>>>>>> Hello all,
>>>>>>>>> If I understand you correctly Ismael, a good amount of
>>>>>>>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the amount
>>>>>>>>> of work necessary on the core module should be relatively small. Is this
>>>>>>>>> correct? Are there improvements that may be missing in terms of
>>>>>>>>> modularization?
>>>>>>>>>
>>>>>>>>> There is also the work necessary to build/run tests with Gradle....
>>>>>>>>>
>>>>>>>>> I am also curious... how much work do you estimate is necessary to
>>>>>>>>> support Java 11 with some of the existing sources? I understand that we
>>>>>>>>> have many, many sources, but perhaps some of the more popular ones (e.g.
>>>>>>>>> TextIO)?
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> -P.
>>>>>>>>>
>>>>>>>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <ar...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for the clarification Ismaël.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *  •  **Arif Kasim*
>>>>>>>>>> *  • * Strategic Cloud Engineer
>>>>>>>>>> *  •  *Google, Inc.
>>>>>>>>>>   •  arifkasim@google.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Just wanted to clarify, there is already a JIRA for ongoing work
>>>>>>>>>>> on
>>>>>>>>>>> Java 11 support.
>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-2530
>>>>>>>>>>>
>>>>>>>>>>> I led the initial work on supporting what at the time was Java
>>>>>>>>>>> 9/10,
>>>>>>>>>>> so far the biggest blockers were around the ApiSurface tests
>>>>>>>>>>> (not at
>>>>>>>>>>> all compatible with these versions) but at the time we were at 5
>>>>>>>>>>> tests
>>>>>>>>>>> from getting sdks/core passing. Notice also that the scope of
>>>>>>>>>>> this
>>>>>>>>>>> JIRA evolved to support only the LTS version (Java 11), and
>>>>>>>>>>> specifically to support only sdks/core + direct runner.
>>>>>>>>>>> Supporting all
>>>>>>>>>>> IOs or runners really is more a question of the dependencies
>>>>>>>>>>> working
>>>>>>>>>>> nicely with Java 11 so this will probably take long time. Also
>>>>>>>>>>> the
>>>>>>>>>>> idea so far does NOT include supporting the Java module system
>>>>>>>>>>> at all.
>>>>>>>>>>>
>>>>>>>>>>> I stopped working on this during the move to gradle because it
>>>>>>>>>>> was too
>>>>>>>>>>> hard to tackle both Java evolving and all the ongoing changes in
>>>>>>>>>>> the
>>>>>>>>>>> build system. If somebody in the community wants to contribute
>>>>>>>>>>> in this
>>>>>>>>>>> area it will be greatly appreciated, notice that all the work we
>>>>>>>>>>> did
>>>>>>>>>>> on the build system for this needs to be implemented now in
>>>>>>>>>>> gradle
>>>>>>>>>>> too.
>>>>>>>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <
>>>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to
>>>>>>>>>>> inject the proxy class is. There are other strategies you can use in
>>>>>>>>>>> bytebuddy which work.
>>>>>>>>>>> >
>>>>>>>>>>> > Romain Manni-Bucau
>>>>>>>>>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a
>>>>>>>>>>> écrit :
>>>>>>>>>>> >>
>>>>>>>>>>> >> Romain, do you have any more details on the ByteBuddy
>>>>>>>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just
>>>>>>>>>>> with new language features?
>>>>>>>>>>> >>
>>>>>>>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <
>>>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> Hi Arif,
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it
>>>>>>>>>>> runs (but it means your pipeline is very very simple since it does not have
>>>>>>>>>>> a dofn ;)) if your engine supports it. Also note that the modules not being
>>>>>>>>>>> named you can have to use some weird import names or even unstable ones if
>>>>>>>>>>> you want to use modules (but there is no real reason to do that yet in
>>>>>>>>>>> java).
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> Romain Manni-Bucau
>>>>>>>>>>> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>>>>>>> >>>
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <
>>>>>>>>>>> arifkasim@google.com> a écrit :
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> Hello,
>>>>>>>>>>> >>>> What's the status of java version > 8 support for beam?
>>>>>>>>>>> Thanks.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> -Arif.
>>>>>>>>>>>
>>>>>>>>>>
>
> --
>
>
>
>
> Got feedback? tinyurl.com/swegner-feedback
>

Re: Java > 8 support

Posted by Scott Wegner <sc...@apache.org>.

If we just want to validate our produced artifacts work with Java11, I
believe it would be easy to start running some of our Jenkins jobs on Java
11. beam_PostRelease_NightlyCandidate [1] is a good candidate: it runs the
quickstart examples across all runners from the SNAPSHOT published maven
architype [2].

[1] https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/
[2]
https://github.com/apache/beam/blob/master/release/src/main/groovy/QuickstartArchetype.groovy

On Thu, Oct 18, 2018 at 1:26 AM Ismaël Mejía <ie...@gmail.com> wrote:

> Yes, Kenn I was talking about building artifacts (because this is the
> strictest way to validate the compatiblity of our code base and find issues
> with some of the classpath/code generation we use or with the
> dependencies). I agree 100% with you, the real deal is to validate that we
> can use Java 8 built jars with more recent Java versions without issues and
> examples could be a good target for this.
>
> Supporting all non LTS java versions seems like a lot of work for a small
> return: (1) most users will probably stay in LTS versions, (2) not any
> single execution system has talked about supporting intermediary versions
> (some not even LTS) (3) this will augment the combination of tests we
> should execute to validate a release. Of course if we do care only about
> direct runner this could make sense.
>
> On Thu, Oct 18, 2018 at 9:30 AM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> On Thu, Oct 18, 2018 at 4:55 AM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> Just to add to what Luke said: The reason we had those Java 8-only
>>> modules was because some underlying tech (example: Gearpump) required Java
>>> 8. If an engine requires something then it is OK for a user who chooses the
>>> runner for that engine to also be subject to that requirement. Otherwise I
>>> would push back a bit on specialized modules.
>>>
>>
>> I, too, would avoid getting back into the situation where we have
>> different modules with different build/runtime requirements. (Java 8 was
>> special both because of Gearpump and because it introduced features such as
>> lambdas, type inference, and other functional libraries that were
>> particularly well suited to our programming model).
>>
>>
>>> Ismaël: Do we need to be thinking in terms of building artifacts? I
>>> think the important thing is if a user has a project and just pulls our
>>> jars from maven central that they can use JRE 11 to run it. I think what we
>>> need is a build matrix of user pipelines compiled and run w/ Java 8, 9, 10,
>>> 11 (or some subset). This can be obtained via the examples maven archetype,
>>> for example, having no relation at all to our own build tooling. I know
>>> Jenkins has a plugin for this, so we don't run into the "mega-Jenkins-job"
>>> problem.
>>>
>>
>> +1 for rephrasing it this way. In general one would expect our jars of
>> classfiles to work with later JREs. The magic we do with annotations and
>> bytecode generation may, however, require more support and testing.
>>
>>
>>> On Wed, Oct 17, 2018 at 2:03 PM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>>> When we were supporting Java 7, there were Java 8 based modules in
>>>> Maven. We can follow a similar pattern where Java11 (or any other version)
>>>> have their own projects and as the base Java version moves up, we can merge
>>>> those modules back in to simplify the set of modules being developed.
>>>>
>>>> On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>
>>>>> I want to give more context on the previous work towards Java 11
>>>>> support. The issue was not necessarily that gradle was not ok to support
>>>>> Java 11, but this is also a good point Lukasz.
>>>>>
>>>>> Months ago, we built a maven profile that basically set up Java to be
>>>>> strictly compatible with the ‘minimal’ java.base profile and used Java
>>>>> version (9/10) at the time. We found and fixed some missing dependencies
>>>>> (because Java 11 removed some of the Java EE stuff from the base profile),
>>>>> not sure if these changes were migrated to the gradle build.
>>>>>
>>>>> We also dealt with a good chunk of updates at the time of
>>>>> plugins/bytebuddy and other deps to guarantee that the execution was done
>>>>> strictly with the latest version of Java and its generated bytecode (the
>>>>> origin of the docker build images was to guarantee this isolation). At the
>>>>> time I run the tests of most IOs and most extensions with the existing
>>>>> (java 8 compiled) direct runner in JRE 10 with success but of course this
>>>>> was not the ultimate goal but a good way to evaluate the state of the code
>>>>> base (I remember some GCP tests were broken and the ApiSurfaceTests too).
>>>>>
>>>>> One important thing to remember is that the goal on this work was
>>>>> ‘limited’ to guarantee that a user that had ONLY Java 11 installed could
>>>>> build and generate Beam artifacts and run Beam programs in the direct
>>>>> runner. The limited scope was in part because of some issues of a full
>>>>> migration to Java 11:
>>>>>
>>>>> - Support for Java modules, this is a too much radical change and
>>>>> probably far from the scope of Beam at the moment.
>>>>> - Support for other runners, given that most execution systems do not
>>>>> support yet Java 11 this could imply a lot of extra work. In the meantime,
>>>>> this could be easier now due to the portability work, anyway, still not
>>>>> sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon.
>>>>>
>>>>> Let’s not forget that we still are and probably will be for a long
>>>>> time backwards compatible with Java 8 so we need to ensure that Java 11
>>>>> only things don’t get into the code base.
>>>>>
>>>>> If you Lukasz or someone else is up to take this task, I think the
>>>>> immediate thing to do is to get back to the same state we were before the
>>>>> move to gradle, build the equivalent of the maven profile for the gradle
>>>>> build and tackle code generation and classpath issues, then try to fix
>>>>> missing parts and push this into the CI too (similar to the recent python 3
>>>>> work) to guarantee that new changes don’t break Java 11 compatibility. I
>>>>> sadly lack of bandwidth to lead this task but I will gladly help with
>>>>> reviews, or feedback if someone else is up to the task.
>>>>>
>>>>> Not sure about more recent details, but probably worth evaluating at
>>>>> some moment is if we there is some interest on supporting multi-release
>>>>> jars with Java 11 classes or the dream scenario of having a AOT build of
>>>>> the direct runner.
>>>>>
>>>>> On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy <
>>>>> lukasz.gajowy@gmail.com> wrote:
>>>>>
>>>>>> FYI: If I'm correct, migrating to Java 11 may require Migrating to
>>>>>> Gradle 5, which to me is ok, but there are some issues on the way (such as
>>>>>> [1] or maybe more that I'm not aware of).
>>>>>>
>>>>>> I did some research and it seems that Groovy 2.4.15 (the default
>>>>>> version of Groovy for Gradle 4.10.2) is unable to work with Java 11 due to
>>>>>> Nest based access control problem [2]. The very same error happened when I
>>>>>> tried to compile "core" module with java11. This issue is solved in Groovy
>>>>>> 2.5.3 [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3
>>>>>> (NoClassDefError) and found out that the support for Groovy 2.5.3 will be
>>>>>> added in Gradle 5.0RC1 (milestone) [4].
>>>>>>
>>>>>> Other than that I had some issues with spottless and findbugs (I
>>>>>> turned them off for the sake of my experiment).
>>>>>>
>>>>>> Łukasz
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-4042
>>>>>> [2]
>>>>>> https://github.com/gradle/gradle/issues/5120#issuecomment-424610114
>>>>>> [3] https://issues.apache.org/jira/browse/GROOVY-8727
>>>>>> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw>
>>>>>> [4] https://github.com/gradle/gradle/pull/7376
>>>>>> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q>
>>>>>>
>>>>>>
>>>>>> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lu...@gmail.com>
>>>>>> napisał(a):
>>>>>>
>>>>>>> FWIW, the issue regarding java 11 support to gradle might be helpful
>>>>>>> here [1]. Quoting: "there are only some minor corner cases that don't work.
>>>>>>> Overall Java 11 support is pretty solid already" but some users reported
>>>>>>> problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 (this
>>>>>>> might be important for us).
>>>>>>>
>>>>>>> Łukasz
>>>>>>>
>>>>>>> [1] https://github.com/gradle/gradle/issues/5120
>>>>>>>
>>>>>>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pa...@google.com>
>>>>>>> napisał(a):
>>>>>>>
>>>>>>>> Hello all,
>>>>>>>> If I understand you correctly Ismael, a good amount of
>>>>>>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the amount
>>>>>>>> of work necessary on the core module should be relatively small. Is this
>>>>>>>> correct? Are there improvements that may be missing in terms of
>>>>>>>> modularization?
>>>>>>>>
>>>>>>>> There is also the work necessary to build/run tests with Gradle....
>>>>>>>>
>>>>>>>> I am also curious... how much work do you estimate is necessary to
>>>>>>>> support Java 11 with some of the existing sources? I understand that we
>>>>>>>> have many, many sources, but perhaps some of the more popular ones (e.g.
>>>>>>>> TextIO)?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> -P.
>>>>>>>>
>>>>>>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <ar...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the clarification Ismaël.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *  •  **Arif Kasim*
>>>>>>>>> *  • * Strategic Cloud Engineer
>>>>>>>>> *  •  *Google, Inc.
>>>>>>>>>   •  arifkasim@google.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Just wanted to clarify, there is already a JIRA for ongoing work
>>>>>>>>>> on
>>>>>>>>>> Java 11 support.
>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-2530
>>>>>>>>>>
>>>>>>>>>> I led the initial work on supporting what at the time was Java
>>>>>>>>>> 9/10,
>>>>>>>>>> so far the biggest blockers were around the ApiSurface tests (not
>>>>>>>>>> at
>>>>>>>>>> all compatible with these versions) but at the time we were at 5
>>>>>>>>>> tests
>>>>>>>>>> from getting sdks/core passing. Notice also that the scope of this
>>>>>>>>>> JIRA evolved to support only the LTS version (Java 11), and
>>>>>>>>>> specifically to support only sdks/core + direct runner.
>>>>>>>>>> Supporting all
>>>>>>>>>> IOs or runners really is more a question of the dependencies
>>>>>>>>>> working
>>>>>>>>>> nicely with Java 11 so this will probably take long time. Also the
>>>>>>>>>> idea so far does NOT include supporting the Java module system at
>>>>>>>>>> all.
>>>>>>>>>>
>>>>>>>>>> I stopped working on this during the move to gradle because it
>>>>>>>>>> was too
>>>>>>>>>> hard to tackle both Java evolving and all the ongoing changes in
>>>>>>>>>> the
>>>>>>>>>> build system. If somebody in the community wants to contribute in
>>>>>>>>>> this
>>>>>>>>>> area it will be greatly appreciated, notice that all the work we
>>>>>>>>>> did
>>>>>>>>>> on the build system for this needs to be implemented now in gradle
>>>>>>>>>> too.
>>>>>>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <
>>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>> >
>>>>>>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to
>>>>>>>>>> inject the proxy class is. There are other strategies you can use in
>>>>>>>>>> bytebuddy which work.
>>>>>>>>>> >
>>>>>>>>>> > Romain Manni-Bucau
>>>>>>>>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a
>>>>>>>>>> écrit :
>>>>>>>>>> >>
>>>>>>>>>> >> Romain, do you have any more details on the ByteBuddy
>>>>>>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just
>>>>>>>>>> with new language features?
>>>>>>>>>> >>
>>>>>>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <
>>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>> >>>
>>>>>>>>>> >>> Hi Arif,
>>>>>>>>>> >>>
>>>>>>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it
>>>>>>>>>> runs (but it means your pipeline is very very simple since it does not have
>>>>>>>>>> a dofn ;)) if your engine supports it. Also note that the modules not being
>>>>>>>>>> named you can have to use some weird import names or even unstable ones if
>>>>>>>>>> you want to use modules (but there is no real reason to do that yet in
>>>>>>>>>> java).
>>>>>>>>>> >>>
>>>>>>>>>> >>> Romain Manni-Bucau
>>>>>>>>>> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com>
>>>>>>>>>> a écrit :
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Hello,
>>>>>>>>>> >>>> What's the status of java version > 8 support for beam?
>>>>>>>>>> Thanks.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> -Arif.
>>>>>>>>>>
>>>>>>>>>

-- 




Got feedback? tinyurl.com/swegner-feedback

Re: Java > 8 support

Posted by Ismaël Mejía <ie...@gmail.com>.

Yes, Kenn I was talking about building artifacts (because this is the
strictest way to validate the compatiblity of our code base and find issues
with some of the classpath/code generation we use or with the
dependencies). I agree 100% with you, the real deal is to validate that we
can use Java 8 built jars with more recent Java versions without issues and
examples could be a good target for this.

Supporting all non LTS java versions seems like a lot of work for a small
return: (1) most users will probably stay in LTS versions, (2) not any
single execution system has talked about supporting intermediary versions
(some not even LTS) (3) this will augment the combination of tests we
should execute to validate a release. Of course if we do care only about
direct runner this could make sense.

On Thu, Oct 18, 2018 at 9:30 AM Robert Bradshaw <ro...@google.com> wrote:

> On Thu, Oct 18, 2018 at 4:55 AM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Just to add to what Luke said: The reason we had those Java 8-only
>> modules was because some underlying tech (example: Gearpump) required Java
>> 8. If an engine requires something then it is OK for a user who chooses the
>> runner for that engine to also be subject to that requirement. Otherwise I
>> would push back a bit on specialized modules.
>>
>
> I, too, would avoid getting back into the situation where we have
> different modules with different build/runtime requirements. (Java 8 was
> special both because of Gearpump and because it introduced features such as
> lambdas, type inference, and other functional libraries that were
> particularly well suited to our programming model).
>
>
>> Ismaël: Do we need to be thinking in terms of building artifacts? I think
>> the important thing is if a user has a project and just pulls our jars from
>> maven central that they can use JRE 11 to run it. I think what we need is a
>> build matrix of user pipelines compiled and run w/ Java 8, 9, 10, 11 (or
>> some subset). This can be obtained via the examples maven archetype, for
>> example, having no relation at all to our own build tooling. I know Jenkins
>> has a plugin for this, so we don't run into the "mega-Jenkins-job" problem.
>>
>
> +1 for rephrasing it this way. In general one would expect our jars of
> classfiles to work with later JREs. The magic we do with annotations and
> bytecode generation may, however, require more support and testing.
>
>
>> On Wed, Oct 17, 2018 at 2:03 PM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> When we were supporting Java 7, there were Java 8 based modules in
>>> Maven. We can follow a similar pattern where Java11 (or any other version)
>>> have their own projects and as the base Java version moves up, we can merge
>>> those modules back in to simplify the set of modules being developed.
>>>
>>> On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> I want to give more context on the previous work towards Java 11
>>>> support. The issue was not necessarily that gradle was not ok to support
>>>> Java 11, but this is also a good point Lukasz.
>>>>
>>>> Months ago, we built a maven profile that basically set up Java to be
>>>> strictly compatible with the ‘minimal’ java.base profile and used Java
>>>> version (9/10) at the time. We found and fixed some missing dependencies
>>>> (because Java 11 removed some of the Java EE stuff from the base profile),
>>>> not sure if these changes were migrated to the gradle build.
>>>>
>>>> We also dealt with a good chunk of updates at the time of
>>>> plugins/bytebuddy and other deps to guarantee that the execution was done
>>>> strictly with the latest version of Java and its generated bytecode (the
>>>> origin of the docker build images was to guarantee this isolation). At the
>>>> time I run the tests of most IOs and most extensions with the existing
>>>> (java 8 compiled) direct runner in JRE 10 with success but of course this
>>>> was not the ultimate goal but a good way to evaluate the state of the code
>>>> base (I remember some GCP tests were broken and the ApiSurfaceTests too).
>>>>
>>>> One important thing to remember is that the goal on this work was
>>>> ‘limited’ to guarantee that a user that had ONLY Java 11 installed could
>>>> build and generate Beam artifacts and run Beam programs in the direct
>>>> runner. The limited scope was in part because of some issues of a full
>>>> migration to Java 11:
>>>>
>>>> - Support for Java modules, this is a too much radical change and
>>>> probably far from the scope of Beam at the moment.
>>>> - Support for other runners, given that most execution systems do not
>>>> support yet Java 11 this could imply a lot of extra work. In the meantime,
>>>> this could be easier now due to the portability work, anyway, still not
>>>> sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon.
>>>>
>>>> Let’s not forget that we still are and probably will be for a long time
>>>> backwards compatible with Java 8 so we need to ensure that Java 11 only
>>>> things don’t get into the code base.
>>>>
>>>> If you Lukasz or someone else is up to take this task, I think the
>>>> immediate thing to do is to get back to the same state we were before the
>>>> move to gradle, build the equivalent of the maven profile for the gradle
>>>> build and tackle code generation and classpath issues, then try to fix
>>>> missing parts and push this into the CI too (similar to the recent python 3
>>>> work) to guarantee that new changes don’t break Java 11 compatibility. I
>>>> sadly lack of bandwidth to lead this task but I will gladly help with
>>>> reviews, or feedback if someone else is up to the task.
>>>>
>>>> Not sure about more recent details, but probably worth evaluating at
>>>> some moment is if we there is some interest on supporting multi-release
>>>> jars with Java 11 classes or the dream scenario of having a AOT build of
>>>> the direct runner.
>>>>
>>>> On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy <lu...@gmail.com>
>>>> wrote:
>>>>
>>>>> FYI: If I'm correct, migrating to Java 11 may require Migrating to
>>>>> Gradle 5, which to me is ok, but there are some issues on the way (such as
>>>>> [1] or maybe more that I'm not aware of).
>>>>>
>>>>> I did some research and it seems that Groovy 2.4.15 (the default
>>>>> version of Groovy for Gradle 4.10.2) is unable to work with Java 11 due to
>>>>> Nest based access control problem [2]. The very same error happened when I
>>>>> tried to compile "core" module with java11. This issue is solved in Groovy
>>>>> 2.5.3 [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3
>>>>> (NoClassDefError) and found out that the support for Groovy 2.5.3 will be
>>>>> added in Gradle 5.0RC1 (milestone) [4].
>>>>>
>>>>> Other than that I had some issues with spottless and findbugs (I
>>>>> turned them off for the sake of my experiment).
>>>>>
>>>>> Łukasz
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/BEAM-4042
>>>>> [2]
>>>>> https://github.com/gradle/gradle/issues/5120#issuecomment-424610114
>>>>> [3] https://issues.apache.org/jira/browse/GROOVY-8727
>>>>> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw>
>>>>> [4] https://github.com/gradle/gradle/pull/7376
>>>>> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q>
>>>>>
>>>>>
>>>>> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lu...@gmail.com>
>>>>> napisał(a):
>>>>>
>>>>>> FWIW, the issue regarding java 11 support to gradle might be helpful
>>>>>> here [1]. Quoting: "there are only some minor corner cases that don't work.
>>>>>> Overall Java 11 support is pretty solid already" but some users reported
>>>>>> problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 (this
>>>>>> might be important for us).
>>>>>>
>>>>>> Łukasz
>>>>>>
>>>>>> [1] https://github.com/gradle/gradle/issues/5120
>>>>>>
>>>>>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pa...@google.com>
>>>>>> napisał(a):
>>>>>>
>>>>>>> Hello all,
>>>>>>> If I understand you correctly Ismael, a good amount of
>>>>>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the amount
>>>>>>> of work necessary on the core module should be relatively small. Is this
>>>>>>> correct? Are there improvements that may be missing in terms of
>>>>>>> modularization?
>>>>>>>
>>>>>>> There is also the work necessary to build/run tests with Gradle....
>>>>>>>
>>>>>>> I am also curious... how much work do you estimate is necessary to
>>>>>>> support Java 11 with some of the existing sources? I understand that we
>>>>>>> have many, many sources, but perhaps some of the more popular ones (e.g.
>>>>>>> TextIO)?
>>>>>>>
>>>>>>> Thanks!
>>>>>>> -P.
>>>>>>>
>>>>>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <ar...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the clarification Ismaël.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *  •  **Arif Kasim*
>>>>>>>> *  • * Strategic Cloud Engineer
>>>>>>>> *  •  *Google, Inc.
>>>>>>>>   •  arifkasim@google.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Just wanted to clarify, there is already a JIRA for ongoing work on
>>>>>>>>> Java 11 support.
>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-2530
>>>>>>>>>
>>>>>>>>> I led the initial work on supporting what at the time was Java
>>>>>>>>> 9/10,
>>>>>>>>> so far the biggest blockers were around the ApiSurface tests (not
>>>>>>>>> at
>>>>>>>>> all compatible with these versions) but at the time we were at 5
>>>>>>>>> tests
>>>>>>>>> from getting sdks/core passing. Notice also that the scope of this
>>>>>>>>> JIRA evolved to support only the LTS version (Java 11), and
>>>>>>>>> specifically to support only sdks/core + direct runner. Supporting
>>>>>>>>> all
>>>>>>>>> IOs or runners really is more a question of the dependencies
>>>>>>>>> working
>>>>>>>>> nicely with Java 11 so this will probably take long time. Also the
>>>>>>>>> idea so far does NOT include supporting the Java module system at
>>>>>>>>> all.
>>>>>>>>>
>>>>>>>>> I stopped working on this during the move to gradle because it was
>>>>>>>>> too
>>>>>>>>> hard to tackle both Java evolving and all the ongoing changes in
>>>>>>>>> the
>>>>>>>>> build system. If somebody in the community wants to contribute in
>>>>>>>>> this
>>>>>>>>> area it will be greatly appreciated, notice that all the work we
>>>>>>>>> did
>>>>>>>>> on the build system for this needs to be implemented now in gradle
>>>>>>>>> too.
>>>>>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <
>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>> >
>>>>>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to inject
>>>>>>>>> the proxy class is. There are other strategies you can use in bytebuddy
>>>>>>>>> which work.
>>>>>>>>> >
>>>>>>>>> > Romain Manni-Bucau
>>>>>>>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a
>>>>>>>>> écrit :
>>>>>>>>> >>
>>>>>>>>> >> Romain, do you have any more details on the ByteBuddy
>>>>>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just
>>>>>>>>> with new language features?
>>>>>>>>> >>
>>>>>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <
>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> Hi Arif,
>>>>>>>>> >>>
>>>>>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs
>>>>>>>>> (but it means your pipeline is very very simple since it does not have a
>>>>>>>>> dofn ;)) if your engine supports it. Also note that the modules not being
>>>>>>>>> named you can have to use some weird import names or even unstable ones if
>>>>>>>>> you want to use modules (but there is no real reason to do that yet in
>>>>>>>>> java).
>>>>>>>>> >>>
>>>>>>>>> >>> Romain Manni-Bucau
>>>>>>>>> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com>
>>>>>>>>> a écrit :
>>>>>>>>> >>>>
>>>>>>>>> >>>> Hello,
>>>>>>>>> >>>> What's the status of java version > 8 support for beam?
>>>>>>>>> Thanks.
>>>>>>>>> >>>>
>>>>>>>>> >>>> -Arif.
>>>>>>>>>
>>>>>>>>

Re: Java > 8 support

Posted by Robert Bradshaw <ro...@google.com>.

On Thu, Oct 18, 2018 at 4:55 AM Kenneth Knowles <ke...@apache.org> wrote:

> Just to add to what Luke said: The reason we had those Java 8-only modules
> was because some underlying tech (example: Gearpump) required Java 8. If an
> engine requires something then it is OK for a user who chooses the runner
> for that engine to also be subject to that requirement. Otherwise I would
> push back a bit on specialized modules.
>

I, too, would avoid getting back into the situation where we have different
modules with different build/runtime requirements. (Java 8 was special both
because of Gearpump and because it introduced features such as lambdas,
type inference, and other functional libraries that were particularly well
suited to our programming model).


> Ismaël: Do we need to be thinking in terms of building artifacts? I think
> the important thing is if a user has a project and just pulls our jars from
> maven central that they can use JRE 11 to run it. I think what we need is a
> build matrix of user pipelines compiled and run w/ Java 8, 9, 10, 11 (or
> some subset). This can be obtained via the examples maven archetype, for
> example, having no relation at all to our own build tooling. I know Jenkins
> has a plugin for this, so we don't run into the "mega-Jenkins-job" problem.
>

+1 for rephrasing it this way. In general one would expect our jars of
classfiles to work with later JREs. The magic we do with annotations and
bytecode generation may, however, require more support and testing.


> On Wed, Oct 17, 2018 at 2:03 PM Lukasz Cwik <lc...@google.com> wrote:
>
>> When we were supporting Java 7, there were Java 8 based modules in Maven.
>> We can follow a similar pattern where Java11 (or any other version) have
>> their own projects and as the base Java version moves up, we can merge
>> those modules back in to simplify the set of modules being developed.
>>
>> On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> I want to give more context on the previous work towards Java 11
>>> support. The issue was not necessarily that gradle was not ok to support
>>> Java 11, but this is also a good point Lukasz.
>>>
>>> Months ago, we built a maven profile that basically set up Java to be
>>> strictly compatible with the ‘minimal’ java.base profile and used Java
>>> version (9/10) at the time. We found and fixed some missing dependencies
>>> (because Java 11 removed some of the Java EE stuff from the base profile),
>>> not sure if these changes were migrated to the gradle build.
>>>
>>> We also dealt with a good chunk of updates at the time of
>>> plugins/bytebuddy and other deps to guarantee that the execution was done
>>> strictly with the latest version of Java and its generated bytecode (the
>>> origin of the docker build images was to guarantee this isolation). At the
>>> time I run the tests of most IOs and most extensions with the existing
>>> (java 8 compiled) direct runner in JRE 10 with success but of course this
>>> was not the ultimate goal but a good way to evaluate the state of the code
>>> base (I remember some GCP tests were broken and the ApiSurfaceTests too).
>>>
>>> One important thing to remember is that the goal on this work was
>>> ‘limited’ to guarantee that a user that had ONLY Java 11 installed could
>>> build and generate Beam artifacts and run Beam programs in the direct
>>> runner. The limited scope was in part because of some issues of a full
>>> migration to Java 11:
>>>
>>> - Support for Java modules, this is a too much radical change and
>>> probably far from the scope of Beam at the moment.
>>> - Support for other runners, given that most execution systems do not
>>> support yet Java 11 this could imply a lot of extra work. In the meantime,
>>> this could be easier now due to the portability work, anyway, still not
>>> sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon.
>>>
>>> Let’s not forget that we still are and probably will be for a long time
>>> backwards compatible with Java 8 so we need to ensure that Java 11 only
>>> things don’t get into the code base.
>>>
>>> If you Lukasz or someone else is up to take this task, I think the
>>> immediate thing to do is to get back to the same state we were before the
>>> move to gradle, build the equivalent of the maven profile for the gradle
>>> build and tackle code generation and classpath issues, then try to fix
>>> missing parts and push this into the CI too (similar to the recent python 3
>>> work) to guarantee that new changes don’t break Java 11 compatibility. I
>>> sadly lack of bandwidth to lead this task but I will gladly help with
>>> reviews, or feedback if someone else is up to the task.
>>>
>>> Not sure about more recent details, but probably worth evaluating at
>>> some moment is if we there is some interest on supporting multi-release
>>> jars with Java 11 classes or the dream scenario of having a AOT build of
>>> the direct runner.
>>>
>>> On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy <lu...@gmail.com>
>>> wrote:
>>>
>>>> FYI: If I'm correct, migrating to Java 11 may require Migrating to
>>>> Gradle 5, which to me is ok, but there are some issues on the way (such as
>>>> [1] or maybe more that I'm not aware of).
>>>>
>>>> I did some research and it seems that Groovy 2.4.15 (the default
>>>> version of Groovy for Gradle 4.10.2) is unable to work with Java 11 due to
>>>> Nest based access control problem [2]. The very same error happened when I
>>>> tried to compile "core" module with java11. This issue is solved in Groovy
>>>> 2.5.3 [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3
>>>> (NoClassDefError) and found out that the support for Groovy 2.5.3 will be
>>>> added in Gradle 5.0RC1 (milestone) [4].
>>>>
>>>> Other than that I had some issues with spottless and findbugs (I turned
>>>> them off for the sake of my experiment).
>>>>
>>>> Łukasz
>>>>
>>>> [1] https://issues.apache.org/jira/browse/BEAM-4042
>>>> [2] https://github.com/gradle/gradle/issues/5120#issuecomment-424610114
>>>>
>>>> [3] https://issues.apache.org/jira/browse/GROOVY-8727
>>>> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw>
>>>> [4] https://github.com/gradle/gradle/pull/7376
>>>> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q>
>>>>
>>>>
>>>> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lu...@gmail.com>
>>>> napisał(a):
>>>>
>>>>> FWIW, the issue regarding java 11 support to gradle might be helpful
>>>>> here [1]. Quoting: "there are only some minor corner cases that don't work.
>>>>> Overall Java 11 support is pretty solid already" but some users reported
>>>>> problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 (this
>>>>> might be important for us).
>>>>>
>>>>> Łukasz
>>>>>
>>>>> [1] https://github.com/gradle/gradle/issues/5120
>>>>>
>>>>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pa...@google.com>
>>>>> napisał(a):
>>>>>
>>>>>> Hello all,
>>>>>> If I understand you correctly Ismael, a good amount of
>>>>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the amount
>>>>>> of work necessary on the core module should be relatively small. Is this
>>>>>> correct? Are there improvements that may be missing in terms of
>>>>>> modularization?
>>>>>>
>>>>>> There is also the work necessary to build/run tests with Gradle....
>>>>>>
>>>>>> I am also curious... how much work do you estimate is necessary to
>>>>>> support Java 11 with some of the existing sources? I understand that we
>>>>>> have many, many sources, but perhaps some of the more popular ones (e.g.
>>>>>> TextIO)?
>>>>>>
>>>>>> Thanks!
>>>>>> -P.
>>>>>>
>>>>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <ar...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the clarification Ismaël.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *  •  **Arif Kasim*
>>>>>>> *  • * Strategic Cloud Engineer
>>>>>>> *  •  *Google, Inc.
>>>>>>>   •  arifkasim@google.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Just wanted to clarify, there is already a JIRA for ongoing work on
>>>>>>>> Java 11 support.
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-2530
>>>>>>>>
>>>>>>>> I led the initial work on supporting what at the time was Java 9/10,
>>>>>>>> so far the biggest blockers were around the ApiSurface tests (not at
>>>>>>>> all compatible with these versions) but at the time we were at 5
>>>>>>>> tests
>>>>>>>> from getting sdks/core passing. Notice also that the scope of this
>>>>>>>> JIRA evolved to support only the LTS version (Java 11), and
>>>>>>>> specifically to support only sdks/core + direct runner. Supporting
>>>>>>>> all
>>>>>>>> IOs or runners really is more a question of the dependencies working
>>>>>>>> nicely with Java 11 so this will probably take long time. Also the
>>>>>>>> idea so far does NOT include supporting the Java module system at
>>>>>>>> all.
>>>>>>>>
>>>>>>>> I stopped working on this during the move to gradle because it was
>>>>>>>> too
>>>>>>>> hard to tackle both Java evolving and all the ongoing changes in the
>>>>>>>> build system. If somebody in the community wants to contribute in
>>>>>>>> this
>>>>>>>> area it will be greatly appreciated, notice that all the work we did
>>>>>>>> on the build system for this needs to be implemented now in gradle
>>>>>>>> too.
>>>>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>> >
>>>>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to inject
>>>>>>>> the proxy class is. There are other strategies you can use in bytebuddy
>>>>>>>> which work.
>>>>>>>> >
>>>>>>>> > Romain Manni-Bucau
>>>>>>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a
>>>>>>>> écrit :
>>>>>>>> >>
>>>>>>>> >> Romain, do you have any more details on the ByteBuddy
>>>>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just
>>>>>>>> with new language features?
>>>>>>>> >>
>>>>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>> >>>
>>>>>>>> >>> Hi Arif,
>>>>>>>> >>>
>>>>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs
>>>>>>>> (but it means your pipeline is very very simple since it does not have a
>>>>>>>> dofn ;)) if your engine supports it. Also note that the modules not being
>>>>>>>> named you can have to use some weird import names or even unstable ones if
>>>>>>>> you want to use modules (but there is no real reason to do that yet in
>>>>>>>> java).
>>>>>>>> >>>
>>>>>>>> >>> Romain Manni-Bucau
>>>>>>>> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com>
>>>>>>>> a écrit :
>>>>>>>> >>>>
>>>>>>>> >>>> Hello,
>>>>>>>> >>>> What's the status of java version > 8 support for beam? Thanks.
>>>>>>>> >>>>
>>>>>>>> >>>> -Arif.
>>>>>>>>
>>>>>>>

Re: Java > 8 support

Posted by Kenneth Knowles <ke...@apache.org>.

Just to add to what Luke said: The reason we had those Java 8-only modules
was because some underlying tech (example: Gearpump) required Java 8. If an
engine requires something then it is OK for a user who chooses the runner
for that engine to also be subject to that requirement. Otherwise I would
push back a bit on specialized modules.

Ismaël: Do we need to be thinking in terms of building artifacts? I think
the important thing is if a user has a project and just pulls our jars from
maven central that they can use JRE 11 to run it. I think what we need is a
build matrix of user pipelines compiled and run w/ Java 8, 9, 10, 11 (or
some subset). This can be obtained via the examples maven archetype, for
example, having no relation at all to our own build tooling. I know Jenkins
has a plugin for this, so we don't run into the "mega-Jenkins-job" problem.

Kenn

On Wed, Oct 17, 2018 at 2:03 PM Lukasz Cwik <lc...@google.com> wrote:

> When we were supporting Java 7, there were Java 8 based modules in Maven.
> We can follow a similar pattern where Java11 (or any other version) have
> their own projects and as the base Java version moves up, we can merge
> those modules back in to simplify the set of modules being developed.
>
> On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> I want to give more context on the previous work towards Java 11 support.
>> The issue was not necessarily that gradle was not ok to support Java 11,
>> but this is also a good point Lukasz.
>>
>> Months ago, we built a maven profile that basically set up Java to be
>> strictly compatible with the ‘minimal’ java.base profile and used Java
>> version (9/10) at the time. We found and fixed some missing dependencies
>> (because Java 11 removed some of the Java EE stuff from the base profile),
>> not sure if these changes were migrated to the gradle build.
>>
>> We also dealt with a good chunk of updates at the time of
>> plugins/bytebuddy and other deps to guarantee that the execution was done
>> strictly with the latest version of Java and its generated bytecode (the
>> origin of the docker build images was to guarantee this isolation). At the
>> time I run the tests of most IOs and most extensions with the existing
>> (java 8 compiled) direct runner in JRE 10 with success but of course this
>> was not the ultimate goal but a good way to evaluate the state of the code
>> base (I remember some GCP tests were broken and the ApiSurfaceTests too).
>>
>> One important thing to remember is that the goal on this work was
>> ‘limited’ to guarantee that a user that had ONLY Java 11 installed could
>> build and generate Beam artifacts and run Beam programs in the direct
>> runner. The limited scope was in part because of some issues of a full
>> migration to Java 11:
>>
>> - Support for Java modules, this is a too much radical change and
>> probably far from the scope of Beam at the moment.
>> - Support for other runners, given that most execution systems do not
>> support yet Java 11 this could imply a lot of extra work. In the meantime,
>> this could be easier now due to the portability work, anyway, still not
>> sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon.
>>
>> Let’s not forget that we still are and probably will be for a long time
>> backwards compatible with Java 8 so we need to ensure that Java 11 only
>> things don’t get into the code base.
>>
>> If you Lukasz or someone else is up to take this task, I think the
>> immediate thing to do is to get back to the same state we were before the
>> move to gradle, build the equivalent of the maven profile for the gradle
>> build and tackle code generation and classpath issues, then try to fix
>> missing parts and push this into the CI too (similar to the recent python 3
>> work) to guarantee that new changes don’t break Java 11 compatibility. I
>> sadly lack of bandwidth to lead this task but I will gladly help with
>> reviews, or feedback if someone else is up to the task.
>>
>> Not sure about more recent details, but probably worth evaluating at some
>> moment is if we there is some interest on supporting multi-release jars
>> with Java 11 classes or the dream scenario of having a AOT build of the
>> direct runner.
>>
>> On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy <lu...@gmail.com>
>> wrote:
>>
>>> FYI: If I'm correct, migrating to Java 11 may require Migrating to
>>> Gradle 5, which to me is ok, but there are some issues on the way (such as
>>> [1] or maybe more that I'm not aware of).
>>>
>>> I did some research and it seems that Groovy 2.4.15 (the default version
>>> of Groovy for Gradle 4.10.2) is unable to work with Java 11 due to Nest
>>> based access control problem [2]. The very same error happened when I tried
>>> to compile "core" module with java11. This issue is solved in Groovy 2.5.3
>>> [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3
>>> (NoClassDefError) and found out that the support for Groovy 2.5.3 will be
>>> added in Gradle 5.0RC1 (milestone) [4].
>>>
>>> Other than that I had some issues with spottless and findbugs (I turned
>>> them off for the sake of my experiment).
>>>
>>> Łukasz
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-4042
>>> [2] https://github.com/gradle/gradle/issues/5120#issuecomment-424610114
>>> [3] https://issues.apache.org/jira/browse/GROOVY-8727
>>> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw>
>>> [4] https://github.com/gradle/gradle/pull/7376
>>> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q>
>>>
>>>
>>> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lu...@gmail.com>
>>> napisał(a):
>>>
>>>> FWIW, the issue regarding java 11 support to gradle might be helpful
>>>> here [1]. Quoting: "there are only some minor corner cases that don't work.
>>>> Overall Java 11 support is pretty solid already" but some users reported
>>>> problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 (this
>>>> might be important for us).
>>>>
>>>> Łukasz
>>>>
>>>> [1] https://github.com/gradle/gradle/issues/5120
>>>>
>>>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pa...@google.com>
>>>> napisał(a):
>>>>
>>>>> Hello all,
>>>>> If I understand you correctly Ismael, a good amount of
>>>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the amount
>>>>> of work necessary on the core module should be relatively small. Is this
>>>>> correct? Are there improvements that may be missing in terms of
>>>>> modularization?
>>>>>
>>>>> There is also the work necessary to build/run tests with Gradle....
>>>>>
>>>>> I am also curious... how much work do you estimate is necessary to
>>>>> support Java 11 with some of the existing sources? I understand that we
>>>>> have many, many sources, but perhaps some of the more popular ones (e.g.
>>>>> TextIO)?
>>>>>
>>>>> Thanks!
>>>>> -P.
>>>>>
>>>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <ar...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks for the clarification Ismaël.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *  •  **Arif Kasim*
>>>>>> *  • * Strategic Cloud Engineer
>>>>>> *  •  *Google, Inc.
>>>>>>   •  arifkasim@google.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ie...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Just wanted to clarify, there is already a JIRA for ongoing work on
>>>>>>> Java 11 support.
>>>>>>> https://issues.apache.org/jira/browse/BEAM-2530
>>>>>>>
>>>>>>> I led the initial work on supporting what at the time was Java 9/10,
>>>>>>> so far the biggest blockers were around the ApiSurface tests (not at
>>>>>>> all compatible with these versions) but at the time we were at 5
>>>>>>> tests
>>>>>>> from getting sdks/core passing. Notice also that the scope of this
>>>>>>> JIRA evolved to support only the LTS version (Java 11), and
>>>>>>> specifically to support only sdks/core + direct runner. Supporting
>>>>>>> all
>>>>>>> IOs or runners really is more a question of the dependencies working
>>>>>>> nicely with Java 11 so this will probably take long time. Also the
>>>>>>> idea so far does NOT include supporting the Java module system at
>>>>>>> all.
>>>>>>>
>>>>>>> I stopped working on this during the move to gradle because it was
>>>>>>> too
>>>>>>> hard to tackle both Java evolving and all the ongoing changes in the
>>>>>>> build system. If somebody in the community wants to contribute in
>>>>>>> this
>>>>>>> area it will be greatly appreciated, notice that all the work we did
>>>>>>> on the build system for this needs to be implemented now in gradle
>>>>>>> too.
>>>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>> >
>>>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to inject
>>>>>>> the proxy class is. There are other strategies you can use in bytebuddy
>>>>>>> which work.
>>>>>>> >
>>>>>>> > Romain Manni-Bucau
>>>>>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>>> >
>>>>>>> >
>>>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a
>>>>>>> écrit :
>>>>>>> >>
>>>>>>> >> Romain, do you have any more details on the ByteBuddy
>>>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just
>>>>>>> with new language features?
>>>>>>> >>
>>>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>> >>>
>>>>>>> >>> Hi Arif,
>>>>>>> >>>
>>>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs
>>>>>>> (but it means your pipeline is very very simple since it does not have a
>>>>>>> dofn ;)) if your engine supports it. Also note that the modules not being
>>>>>>> named you can have to use some weird import names or even unstable ones if
>>>>>>> you want to use modules (but there is no real reason to do that yet in
>>>>>>> java).
>>>>>>> >>>
>>>>>>> >>> Romain Manni-Bucau
>>>>>>> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com>
>>>>>>> a écrit :
>>>>>>> >>>>
>>>>>>> >>>> Hello,
>>>>>>> >>>> What's the status of java version > 8 support for beam? Thanks.
>>>>>>> >>>>
>>>>>>> >>>> -Arif.
>>>>>>>
>>>>>>

Re: Java > 8 support

Posted by Lukasz Cwik <lc...@google.com>.

When we were supporting Java 7, there were Java 8 based modules in Maven.
We can follow a similar pattern where Java11 (or any other version) have
their own projects and as the base Java version moves up, we can merge
those modules back in to simplify the set of modules being developed.

On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <ie...@gmail.com> wrote:

> I want to give more context on the previous work towards Java 11 support.
> The issue was not necessarily that gradle was not ok to support Java 11,
> but this is also a good point Lukasz.
>
> Months ago, we built a maven profile that basically set up Java to be
> strictly compatible with the ‘minimal’ java.base profile and used Java
> version (9/10) at the time. We found and fixed some missing dependencies
> (because Java 11 removed some of the Java EE stuff from the base profile),
> not sure if these changes were migrated to the gradle build.
>
> We also dealt with a good chunk of updates at the time of
> plugins/bytebuddy and other deps to guarantee that the execution was done
> strictly with the latest version of Java and its generated bytecode (the
> origin of the docker build images was to guarantee this isolation). At the
> time I run the tests of most IOs and most extensions with the existing
> (java 8 compiled) direct runner in JRE 10 with success but of course this
> was not the ultimate goal but a good way to evaluate the state of the code
> base (I remember some GCP tests were broken and the ApiSurfaceTests too).
>
> One important thing to remember is that the goal on this work was
> ‘limited’ to guarantee that a user that had ONLY Java 11 installed could
> build and generate Beam artifacts and run Beam programs in the direct
> runner. The limited scope was in part because of some issues of a full
> migration to Java 11:
>
> - Support for Java modules, this is a too much radical change and probably
> far from the scope of Beam at the moment.
> - Support for other runners, given that most execution systems do not
> support yet Java 11 this could imply a lot of extra work. In the meantime,
> this could be easier now due to the portability work, anyway, still not
> sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon.
>
> Let’s not forget that we still are and probably will be for a long time
> backwards compatible with Java 8 so we need to ensure that Java 11 only
> things don’t get into the code base.
>
> If you Lukasz or someone else is up to take this task, I think the
> immediate thing to do is to get back to the same state we were before the
> move to gradle, build the equivalent of the maven profile for the gradle
> build and tackle code generation and classpath issues, then try to fix
> missing parts and push this into the CI too (similar to the recent python 3
> work) to guarantee that new changes don’t break Java 11 compatibility. I
> sadly lack of bandwidth to lead this task but I will gladly help with
> reviews, or feedback if someone else is up to the task.
>
> Not sure about more recent details, but probably worth evaluating at some
> moment is if we there is some interest on supporting multi-release jars
> with Java 11 classes or the dream scenario of having a AOT build of the
> direct runner.
>
> On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy <lu...@gmail.com>
> wrote:
>
>> FYI: If I'm correct, migrating to Java 11 may require Migrating to Gradle
>> 5, which to me is ok, but there are some issues on the way (such as [1] or
>> maybe more that I'm not aware of).
>>
>> I did some research and it seems that Groovy 2.4.15 (the default version
>> of Groovy for Gradle 4.10.2) is unable to work with Java 11 due to Nest
>> based access control problem [2]. The very same error happened when I tried
>> to compile "core" module with java11. This issue is solved in Groovy 2.5.3
>> [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3
>> (NoClassDefError) and found out that the support for Groovy 2.5.3 will be
>> added in Gradle 5.0RC1 (milestone) [4].
>>
>> Other than that I had some issues with spottless and findbugs (I turned
>> them off for the sake of my experiment).
>>
>> Łukasz
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-4042
>> [2] https://github.com/gradle/gradle/issues/5120#issuecomment-424610114
>> [3] https://issues.apache.org/jira/browse/GROOVY-8727
>> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw>
>> [4] https://github.com/gradle/gradle/pull/7376
>> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q>
>>
>>
>> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lu...@gmail.com>
>> napisał(a):
>>
>>> FWIW, the issue regarding java 11 support to gradle might be helpful
>>> here [1]. Quoting: "there are only some minor corner cases that don't work.
>>> Overall Java 11 support is pretty solid already" but some users reported
>>> problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 (this
>>> might be important for us).
>>>
>>> Łukasz
>>>
>>> [1] https://github.com/gradle/gradle/issues/5120
>>>
>>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pa...@google.com> napisał(a):
>>>
>>>> Hello all,
>>>> If I understand you correctly Ismael, a good amount of
>>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the amount
>>>> of work necessary on the core module should be relatively small. Is this
>>>> correct? Are there improvements that may be missing in terms of
>>>> modularization?
>>>>
>>>> There is also the work necessary to build/run tests with Gradle....
>>>>
>>>> I am also curious... how much work do you estimate is necessary to
>>>> support Java 11 with some of the existing sources? I understand that we
>>>> have many, many sources, but perhaps some of the more popular ones (e.g.
>>>> TextIO)?
>>>>
>>>> Thanks!
>>>> -P.
>>>>
>>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <ar...@google.com>
>>>> wrote:
>>>>
>>>>> Thanks for the clarification Ismaël.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *  •  **Arif Kasim*
>>>>> *  • * Strategic Cloud Engineer
>>>>> *  •  *Google, Inc.
>>>>>   •  arifkasim@google.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ie...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Just wanted to clarify, there is already a JIRA for ongoing work on
>>>>>> Java 11 support.
>>>>>> https://issues.apache.org/jira/browse/BEAM-2530
>>>>>>
>>>>>> I led the initial work on supporting what at the time was Java 9/10,
>>>>>> so far the biggest blockers were around the ApiSurface tests (not at
>>>>>> all compatible with these versions) but at the time we were at 5 tests
>>>>>> from getting sdks/core passing. Notice also that the scope of this
>>>>>> JIRA evolved to support only the LTS version (Java 11), and
>>>>>> specifically to support only sdks/core + direct runner. Supporting all
>>>>>> IOs or runners really is more a question of the dependencies working
>>>>>> nicely with Java 11 so this will probably take long time. Also the
>>>>>> idea so far does NOT include supporting the Java module system at all.
>>>>>>
>>>>>> I stopped working on this during the move to gradle because it was too
>>>>>> hard to tackle both Java evolving and all the ongoing changes in the
>>>>>> build system. If somebody in the community wants to contribute in this
>>>>>> area it will be greatly appreciated, notice that all the work we did
>>>>>> on the build system for this needs to be implemented now in gradle
>>>>>> too.
>>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>> >
>>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to inject
>>>>>> the proxy class is. There are other strategies you can use in bytebuddy
>>>>>> which work.
>>>>>> >
>>>>>> > Romain Manni-Bucau
>>>>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>> >
>>>>>> >
>>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a écrit
>>>>>> :
>>>>>> >>
>>>>>> >> Romain, do you have any more details on the ByteBuddy
>>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just
>>>>>> with new language features?
>>>>>> >>
>>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>> >>>
>>>>>> >>> Hi Arif,
>>>>>> >>>
>>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs
>>>>>> (but it means your pipeline is very very simple since it does not have a
>>>>>> dofn ;)) if your engine supports it. Also note that the modules not being
>>>>>> named you can have to use some weird import names or even unstable ones if
>>>>>> you want to use modules (but there is no real reason to do that yet in
>>>>>> java).
>>>>>> >>>
>>>>>> >>> Romain Manni-Bucau
>>>>>> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com> a
>>>>>> écrit :
>>>>>> >>>>
>>>>>> >>>> Hello,
>>>>>> >>>> What's the status of java version > 8 support for beam? Thanks.
>>>>>> >>>>
>>>>>> >>>> -Arif.
>>>>>>
>>>>>

Re: Java > 8 support

Posted by Ismaël Mejía <ie...@gmail.com>.

I want to give more context on the previous work towards Java 11 support.
The issue was not necessarily that gradle was not ok to support Java 11,
but this is also a good point Lukasz.

Months ago, we built a maven profile that basically set up Java to be
strictly compatible with the ‘minimal’ java.base profile and used Java
version (9/10) at the time. We found and fixed some missing dependencies
(because Java 11 removed some of the Java EE stuff from the base profile),
not sure if these changes were migrated to the gradle build.

We also dealt with a good chunk of updates at the time of plugins/bytebuddy
and other deps to guarantee that the execution was done strictly with the
latest version of Java and its generated bytecode (the origin of the docker
build images was to guarantee this isolation). At the time I run the tests
of most IOs and most extensions with the existing (java 8 compiled) direct
runner in JRE 10 with success but of course this was not the ultimate goal
but a good way to evaluate the state of the code base (I remember some GCP
tests were broken and the ApiSurfaceTests too).

One important thing to remember is that the goal on this work was ‘limited’
to guarantee that a user that had ONLY Java 11 installed could build and
generate Beam artifacts and run Beam programs in the direct runner. The
limited scope was in part because of some issues of a full migration to
Java 11:

- Support for Java modules, this is a too much radical change and probably
far from the scope of Beam at the moment.
- Support for other runners, given that most execution systems do not
support yet Java 11 this could imply a lot of extra work. In the meantime,
this could be easier now due to the portability work, anyway, still not
sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon.

Let’s not forget that we still are and probably will be for a long time
backwards compatible with Java 8 so we need to ensure that Java 11 only
things don’t get into the code base.

If you Lukasz or someone else is up to take this task, I think the
immediate thing to do is to get back to the same state we were before the
move to gradle, build the equivalent of the maven profile for the gradle
build and tackle code generation and classpath issues, then try to fix
missing parts and push this into the CI too (similar to the recent python 3
work) to guarantee that new changes don’t break Java 11 compatibility. I
sadly lack of bandwidth to lead this task but I will gladly help with
reviews, or feedback if someone else is up to the task.

Not sure about more recent details, but probably worth evaluating at some
moment is if we there is some interest on supporting multi-release jars
with Java 11 classes or the dream scenario of having a AOT build of the
direct runner.

On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy <lu...@gmail.com>
wrote:

> FYI: If I'm correct, migrating to Java 11 may require Migrating to Gradle
> 5, which to me is ok, but there are some issues on the way (such as [1] or
> maybe more that I'm not aware of).
>
> I did some research and it seems that Groovy 2.4.15 (the default version
> of Groovy for Gradle 4.10.2) is unable to work with Java 11 due to Nest
> based access control problem [2]. The very same error happened when I tried
> to compile "core" module with java11. This issue is solved in Groovy 2.5.3
> [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3
> (NoClassDefError) and found out that the support for Groovy 2.5.3 will be
> added in Gradle 5.0RC1 (milestone) [4].
>
> Other than that I had some issues with spottless and findbugs (I turned
> them off for the sake of my experiment).
>
> Łukasz
>
> [1] https://issues.apache.org/jira/browse/BEAM-4042
> [2] https://github.com/gradle/gradle/issues/5120#issuecomment-424610114
> [3] https://issues.apache.org/jira/browse/GROOVY-8727
> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw>
> [4] https://github.com/gradle/gradle/pull/7376
> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q>
>
>
> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lu...@gmail.com>
> napisał(a):
>
>> FWIW, the issue regarding java 11 support to gradle might be helpful here
>> [1]. Quoting: "there are only some minor corner cases that don't work.
>> Overall Java 11 support is pretty solid already" but some users reported
>> problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 (this
>> might be important for us).
>>
>> Łukasz
>>
>> [1] https://github.com/gradle/gradle/issues/5120
>>
>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pa...@google.com> napisał(a):
>>
>>> Hello all,
>>> If I understand you correctly Ismael, a good amount of
>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the amount
>>> of work necessary on the core module should be relatively small. Is this
>>> correct? Are there improvements that may be missing in terms of
>>> modularization?
>>>
>>> There is also the work necessary to build/run tests with Gradle....
>>>
>>> I am also curious... how much work do you estimate is necessary to
>>> support Java 11 with some of the existing sources? I understand that we
>>> have many, many sources, but perhaps some of the more popular ones (e.g.
>>> TextIO)?
>>>
>>> Thanks!
>>> -P.
>>>
>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <ar...@google.com>
>>> wrote:
>>>
>>>> Thanks for the clarification Ismaël.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *  •  **Arif Kasim*
>>>> *  • * Strategic Cloud Engineer
>>>> *  •  *Google, Inc.
>>>>   •  arifkasim@google.com
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>
>>>>> Just wanted to clarify, there is already a JIRA for ongoing work on
>>>>> Java 11 support.
>>>>> https://issues.apache.org/jira/browse/BEAM-2530
>>>>>
>>>>> I led the initial work on supporting what at the time was Java 9/10,
>>>>> so far the biggest blockers were around the ApiSurface tests (not at
>>>>> all compatible with these versions) but at the time we were at 5 tests
>>>>> from getting sdks/core passing. Notice also that the scope of this
>>>>> JIRA evolved to support only the LTS version (Java 11), and
>>>>> specifically to support only sdks/core + direct runner. Supporting all
>>>>> IOs or runners really is more a question of the dependencies working
>>>>> nicely with Java 11 so this will probably take long time. Also the
>>>>> idea so far does NOT include supporting the Java module system at all.
>>>>>
>>>>> I stopped working on this during the move to gradle because it was too
>>>>> hard to tackle both Java evolving and all the ongoing changes in the
>>>>> build system. If somebody in the community wants to contribute in this
>>>>> area it will be greatly appreciated, notice that all the work we did
>>>>> on the build system for this needs to be implemented now in gradle
>>>>> too.
>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>> >
>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to inject the
>>>>> proxy class is. There are other strategies you can use in bytebuddy which
>>>>> work.
>>>>> >
>>>>> > Romain Manni-Bucau
>>>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>> >
>>>>> >
>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a écrit :
>>>>> >>
>>>>> >> Romain, do you have any more details on the ByteBuddy
>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just
>>>>> with new language features?
>>>>> >>
>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>> >>>
>>>>> >>> Hi Arif,
>>>>> >>>
>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs
>>>>> (but it means your pipeline is very very simple since it does not have a
>>>>> dofn ;)) if your engine supports it. Also note that the modules not being
>>>>> named you can have to use some weird import names or even unstable ones if
>>>>> you want to use modules (but there is no real reason to do that yet in
>>>>> java).
>>>>> >>>
>>>>> >>> Romain Manni-Bucau
>>>>> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>>> >>>
>>>>> >>>
>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com> a
>>>>> écrit :
>>>>> >>>>
>>>>> >>>> Hello,
>>>>> >>>> What's the status of java version > 8 support for beam? Thanks.
>>>>> >>>>
>>>>> >>>> -Arif.
>>>>>
>>>>

Re: Java > 8 support

Posted by Łukasz Gajowy <lu...@gmail.com>.

FYI: If I'm correct, migrating to Java 11 may require Migrating to Gradle
5, which to me is ok, but there are some issues on the way (such as [1] or
maybe more that I'm not aware of).

I did some research and it seems that Groovy 2.4.15 (the default version of
Groovy for Gradle 4.10.2) is unable to work with Java 11 due to Nest based
access control problem [2]. The very same error happened when I tried to
compile "core" module with java11. This issue is solved in Groovy 2.5.3 [3]
. I was unable to run Gradle 4.10.2 with Groovy 2.5.3 (NoClassDefError) and
found out that the support for Groovy 2.5.3 will be added in Gradle 5.0RC1
(milestone) [4].

Other than that I had some issues with spottless and findbugs (I turned
them off for the sake of my experiment).

Łukasz

[1] https://issues.apache.org/jira/browse/BEAM-4042
[2] https://github.com/gradle/gradle/issues/5120#issuecomment-424610114
[3] https://issues.apache.org/jira/browse/GROOVY-8727
<https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw>
[4] https://github.com/gradle/gradle/pull/7376
<https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q>


czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lu...@gmail.com>
napisał(a):

> FWIW, the issue regarding java 11 support to gradle might be helpful here
> [1]. Quoting: "there are only some minor corner cases that don't work.
> Overall Java 11 support is pretty solid already" but some users reported
> problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 (this
> might be important for us).
>
> Łukasz
>
> [1] https://github.com/gradle/gradle/issues/5120
>
> czw., 11 paź 2018 o 02:06 Pablo Estrada <pa...@google.com> napisał(a):
>
>> Hello all,
>> If I understand you correctly Ismael, a good amount of
>> 'beam-sdks-java-core' tests are already passing with Java 11, so the amount
>> of work necessary on the core module should be relatively small. Is this
>> correct? Are there improvements that may be missing in terms of
>> modularization?
>>
>> There is also the work necessary to build/run tests with Gradle....
>>
>> I am also curious... how much work do you estimate is necessary to
>> support Java 11 with some of the existing sources? I understand that we
>> have many, many sources, but perhaps some of the more popular ones (e.g.
>> TextIO)?
>>
>> Thanks!
>> -P.
>>
>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <ar...@google.com> wrote:
>>
>>> Thanks for the clarification Ismaël.
>>>
>>>
>>>
>>>
>>>
>>> *  •  **Arif Kasim*
>>> *  • * Strategic Cloud Engineer
>>> *  •  *Google, Inc.
>>>   •  arifkasim@google.com
>>>
>>>
>>>
>>>
>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> Just wanted to clarify, there is already a JIRA for ongoing work on
>>>> Java 11 support.
>>>> https://issues.apache.org/jira/browse/BEAM-2530
>>>>
>>>> I led the initial work on supporting what at the time was Java 9/10,
>>>> so far the biggest blockers were around the ApiSurface tests (not at
>>>> all compatible with these versions) but at the time we were at 5 tests
>>>> from getting sdks/core passing. Notice also that the scope of this
>>>> JIRA evolved to support only the LTS version (Java 11), and
>>>> specifically to support only sdks/core + direct runner. Supporting all
>>>> IOs or runners really is more a question of the dependencies working
>>>> nicely with Java 11 so this will probably take long time. Also the
>>>> idea so far does NOT include supporting the Java module system at all.
>>>>
>>>> I stopped working on this during the move to gradle because it was too
>>>> hard to tackle both Java evolving and all the ongoing changes in the
>>>> build system. If somebody in the community wants to contribute in this
>>>> area it will be greatly appreciated, notice that all the work we did
>>>> on the build system for this needs to be implemented now in gradle
>>>> too.
>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>> >
>>>> > @Reuven: bytebuddy by itself no but the way beam tries to inject the
>>>> proxy class is. There are other strategies you can use in bytebuddy which
>>>> work.
>>>> >
>>>> > Romain Manni-Bucau
>>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>> >
>>>> >
>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a écrit :
>>>> >>
>>>> >> Romain, do you have any more details on the ByteBuddy
>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just
>>>> with new language features?
>>>> >>
>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>> >>>
>>>> >>> Hi Arif,
>>>> >>>
>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but
>>>> it means your pipeline is very very simple since it does not have a dofn
>>>> ;)) if your engine supports it. Also note that the modules not being named
>>>> you can have to use some weird import names or even unstable ones if you
>>>> want to use modules (but there is no real reason to do that yet in java).
>>>> >>>
>>>> >>> Romain Manni-Bucau
>>>> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>> >>>
>>>> >>>
>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com> a
>>>> écrit :
>>>> >>>>
>>>> >>>> Hello,
>>>> >>>> What's the status of java version > 8 support for beam? Thanks.
>>>> >>>>
>>>> >>>> -Arif.
>>>>
>>>

Re: Java > 8 support

Posted by Łukasz Gajowy <lu...@gmail.com>.

FWIW, the issue regarding java 11 support to gradle might be helpful here
[1]. Quoting: "there are only some minor corner cases that don't work.
Overall Java 11 support is pretty solid already" but some users reported
problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 (this
might be important for us).

Łukasz

[1] https://github.com/gradle/gradle/issues/5120

czw., 11 paź 2018 o 02:06 Pablo Estrada <pa...@google.com> napisał(a):

> Hello all,
> If I understand you correctly Ismael, a good amount of
> 'beam-sdks-java-core' tests are already passing with Java 11, so the amount
> of work necessary on the core module should be relatively small. Is this
> correct? Are there improvements that may be missing in terms of
> modularization?
>
> There is also the work necessary to build/run tests with Gradle....
>
> I am also curious... how much work do you estimate is necessary to support
> Java 11 with some of the existing sources? I understand that we have many,
> many sources, but perhaps some of the more popular ones (e.g. TextIO)?
>
> Thanks!
> -P.
>
> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <ar...@google.com> wrote:
>
>> Thanks for the clarification Ismaël.
>>
>>
>>
>>
>>
>> *  •  **Arif Kasim*
>> *  • * Strategic Cloud Engineer
>> *  •  *Google, Inc.
>>   •  arifkasim@google.com
>>
>>
>>
>>
>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> Just wanted to clarify, there is already a JIRA for ongoing work on
>>> Java 11 support.
>>> https://issues.apache.org/jira/browse/BEAM-2530
>>>
>>> I led the initial work on supporting what at the time was Java 9/10,
>>> so far the biggest blockers were around the ApiSurface tests (not at
>>> all compatible with these versions) but at the time we were at 5 tests
>>> from getting sdks/core passing. Notice also that the scope of this
>>> JIRA evolved to support only the LTS version (Java 11), and
>>> specifically to support only sdks/core + direct runner. Supporting all
>>> IOs or runners really is more a question of the dependencies working
>>> nicely with Java 11 so this will probably take long time. Also the
>>> idea so far does NOT include supporting the Java module system at all.
>>>
>>> I stopped working on this during the move to gradle because it was too
>>> hard to tackle both Java evolving and all the ongoing changes in the
>>> build system. If somebody in the community wants to contribute in this
>>> area it will be greatly appreciated, notice that all the work we did
>>> on the build system for this needs to be implemented now in gradle
>>> too.
>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <rm...@gmail.com>
>>> wrote:
>>> >
>>> > @Reuven: bytebuddy by itself no but the way beam tries to inject the
>>> proxy class is. There are other strategies you can use in bytebuddy which
>>> work.
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> >
>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a écrit :
>>> >>
>>> >> Romain, do you have any more details on the ByteBuddy
>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just
>>> with new language features?
>>> >>
>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>> >>>
>>> >>> Hi Arif,
>>> >>>
>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but
>>> it means your pipeline is very very simple since it does not have a dofn
>>> ;)) if your engine supports it. Also note that the modules not being named
>>> you can have to use some weird import names or even unstable ones if you
>>> want to use modules (but there is no real reason to do that yet in java).
>>> >>>
>>> >>> Romain Manni-Bucau
>>> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >>>
>>> >>>
>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com> a
>>> écrit :
>>> >>>>
>>> >>>> Hello,
>>> >>>> What's the status of java version > 8 support for beam? Thanks.
>>> >>>>
>>> >>>> -Arif.
>>>
>>

Re: Java > 8 support

Posted by Pablo Estrada <pa...@google.com>.

Hello all,
If I understand you correctly Ismael, a good amount of
'beam-sdks-java-core' tests are already passing with Java 11, so the amount
of work necessary on the core module should be relatively small. Is this
correct? Are there improvements that may be missing in terms of
modularization?

There is also the work necessary to build/run tests with Gradle....

I am also curious... how much work do you estimate is necessary to support
Java 11 with some of the existing sources? I understand that we have many,
many sources, but perhaps some of the more popular ones (e.g. TextIO)?

Thanks!
-P.

On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <ar...@google.com> wrote:

> Thanks for the clarification Ismaël.
>
>
>
>
>
> *  •  **Arif Kasim*
> *  • * Strategic Cloud Engineer
> *  •  *Google, Inc.
>   •  arifkasim@google.com
>
>
>
>
> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Just wanted to clarify, there is already a JIRA for ongoing work on
>> Java 11 support.
>> https://issues.apache.org/jira/browse/BEAM-2530
>>
>> I led the initial work on supporting what at the time was Java 9/10,
>> so far the biggest blockers were around the ApiSurface tests (not at
>> all compatible with these versions) but at the time we were at 5 tests
>> from getting sdks/core passing. Notice also that the scope of this
>> JIRA evolved to support only the LTS version (Java 11), and
>> specifically to support only sdks/core + direct runner. Supporting all
>> IOs or runners really is more a question of the dependencies working
>> nicely with Java 11 so this will probably take long time. Also the
>> idea so far does NOT include supporting the Java module system at all.
>>
>> I stopped working on this during the move to gradle because it was too
>> hard to tackle both Java evolving and all the ongoing changes in the
>> build system. If somebody in the community wants to contribute in this
>> area it will be greatly appreciated, notice that all the work we did
>> on the build system for this needs to be implemented now in gradle
>> too.
>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <rm...@gmail.com>
>> wrote:
>> >
>> > @Reuven: bytebuddy by itself no but the way beam tries to inject the
>> proxy class is. There are other strategies you can use in bytebuddy which
>> work.
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >
>> >
>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a écrit :
>> >>
>> >> Romain, do you have any more details on the ByteBuddy incompatibility?
>> Is ByteBuddy incompatible with the Java 11 JRE, or just with new language
>> features?
>> >>
>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <
>> rmannibucau@gmail.com> wrote:
>> >>>
>> >>> Hi Arif,
>> >>>
>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but
>> it means your pipeline is very very simple since it does not have a dofn
>> ;)) if your engine supports it. Also note that the modules not being named
>> you can have to use some weird import names or even unstable ones if you
>> want to use modules (but there is no real reason to do that yet in java).
>> >>>
>> >>> Romain Manni-Bucau
>> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >>>
>> >>>
>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com> a
>> écrit :
>> >>>>
>> >>>> Hello,
>> >>>> What's the status of java version > 8 support for beam? Thanks.
>> >>>>
>> >>>> -Arif.
>>
>

Re: Java > 8 support

Posted by Arif Kasim <ar...@google.com>.

Thanks for the clarification Ismaël.





*  •  **Arif Kasim*
*  • * Strategic Cloud Engineer
*  •  *Google, Inc.
  •  arifkasim@google.com




On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ie...@gmail.com> wrote:

> Just wanted to clarify, there is already a JIRA for ongoing work on
> Java 11 support.
> https://issues.apache.org/jira/browse/BEAM-2530
>
> I led the initial work on supporting what at the time was Java 9/10,
> so far the biggest blockers were around the ApiSurface tests (not at
> all compatible with these versions) but at the time we were at 5 tests
> from getting sdks/core passing. Notice also that the scope of this
> JIRA evolved to support only the LTS version (Java 11), and
> specifically to support only sdks/core + direct runner. Supporting all
> IOs or runners really is more a question of the dependencies working
> nicely with Java 11 so this will probably take long time. Also the
> idea so far does NOT include supporting the Java module system at all.
>
> I stopped working on this during the move to gradle because it was too
> hard to tackle both Java evolving and all the ongoing changes in the
> build system. If somebody in the community wants to contribute in this
> area it will be greatly appreciated, notice that all the work we did
> on the build system for this needs to be implemented now in gradle
> too.
> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
> >
> > @Reuven: bytebuddy by itself no but the way beam tries to inject the
> proxy class is. There are other strategies you can use in bytebuddy which
> work.
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> >
> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a écrit :
> >>
> >> Romain, do you have any more details on the ByteBuddy incompatibility?
> Is ByteBuddy incompatible with the Java 11 JRE, or just with new language
> features?
> >>
> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <
> rmannibucau@gmail.com> wrote:
> >>>
> >>> Hi Arif,
> >>>
> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but it
> means your pipeline is very very simple since it does not have a dofn ;))
> if your engine supports it. Also note that the modules not being named you
> can have to use some weird import names or even unstable ones if you want
> to use modules (but there is no real reason to do that yet in java).
> >>>
> >>> Romain Manni-Bucau
> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>>
> >>>
> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com> a
> écrit :
> >>>>
> >>>> Hello,
> >>>> What's the status of java version > 8 support for beam? Thanks.
> >>>>
> >>>> -Arif.
>

Re: Java > 8 support

Posted by Ismaël Mejía <ie...@gmail.com>.

Just wanted to clarify, there is already a JIRA for ongoing work on
Java 11 support.
https://issues.apache.org/jira/browse/BEAM-2530

I led the initial work on supporting what at the time was Java 9/10,
so far the biggest blockers were around the ApiSurface tests (not at
all compatible with these versions) but at the time we were at 5 tests
from getting sdks/core passing. Notice also that the scope of this
JIRA evolved to support only the LTS version (Java 11), and
specifically to support only sdks/core + direct runner. Supporting all
IOs or runners really is more a question of the dependencies working
nicely with Java 11 so this will probably take long time. Also the
idea so far does NOT include supporting the Java module system at all.

I stopped working on this during the move to gradle because it was too
hard to tackle both Java evolving and all the ongoing changes in the
build system. If somebody in the community wants to contribute in this
area it will be greatly appreciated, notice that all the work we did
on the build system for this needs to be implemented now in gradle
too.
On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <rm...@gmail.com> wrote:
>
> @Reuven: bytebuddy by itself no but the way beam tries to inject the proxy class is. There are other strategies you can use in bytebuddy which work.
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a écrit :
>>
>> Romain, do you have any more details on the ByteBuddy incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just with new language features?
>>
>> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <rm...@gmail.com> wrote:
>>>
>>> Hi Arif,
>>>
>>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but it means your pipeline is very very simple since it does not have a dofn ;)) if your engine supports it. Also note that the modules not being named you can have to use some weird import names or even unstable ones if you want to use modules (but there is no real reason to do that yet in java).
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>
>>>
>>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com> a écrit :
>>>>
>>>> Hello,
>>>> What's the status of java version > 8 support for beam? Thanks.
>>>>
>>>> -Arif.

Re: Java > 8 support

Posted by Romain Manni-Bucau <rm...@gmail.com>.

@Reuven: bytebuddy by itself no but the way beam tries to inject the proxy
class is. There are other strategies you can use in bytebuddy which work.

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a écrit :

> Romain, do you have any more details on the ByteBuddy incompatibility? Is
> ByteBuddy incompatible with the Java 11 JRE, or just with new language
> features?
>
> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> Hi Arif,
>>
>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but it
>> means your pipeline is very very simple since it does not have a dofn ;))
>> if your engine supports it. Also note that the modules not being named you
>> can have to use some weird import names or even unstable ones if you want
>> to use modules (but there is no real reason to do that yet in java).
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>>
>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com> a écrit :
>>
>>> Hello,
>>> What's the status of java version > 8 support for beam? Thanks.
>>>
>>> -Arif.
>>>
>>

Re: Java > 8 support

Posted by Reuven Lax <re...@google.com>.

Romain, do you have any more details on the ByteBuddy incompatibility? Is
ByteBuddy incompatible with the Java 11 JRE, or just with new language
features?

On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:

> Hi Arif,
>
> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but it
> means your pipeline is very very simple since it does not have a dofn ;))
> if your engine supports it. Also note that the modules not being named you
> can have to use some weird import names or even unstable ones if you want
> to use modules (but there is no real reason to do that yet in java).
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com> a écrit :
>
>> Hello,
>> What's the status of java version > 8 support for beam? Thanks.
>>
>> -Arif.
>>
>

Re: Java > 8 support

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.

Hi,

I mean that some IO dependency, especially for the tests should be
updated for the build with J9.

The runtime should work with J9, but not the build.

Regards
JB

On 06/10/2018 17:45, Romain Manni-Bucau wrote:
> I dont think so JB - until you want to use jlink but it is out of scope
> of beam - and since j9 didnt break unsafe it should work.
> 
> Le sam. 6 oct. 2018 17:23, Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> a écrit :
> 
>     Same thing, it requires some change in the dependency, especially for
>     the Jigsaw modules.
> 
>     Regards
>     JB
> 
>     On 06/10/2018 15:01, Arif Kasim wrote:
>     > Thanks all. What about Java 9 support?
>     >
>     >
>     >       
>     >       
>     >       
>     >       
>     >
>     >       
>     >
>     >     *  •  **Arif Kasim**
>     >     **  • * Strategic Cloud Engineer*
>     >     **  •  *Google, Inc.*
>     >     *  •  arifkasim@google.com <ma...@google.com>
>     <mailto:arifkasim@google.com <ma...@google.com>>  
>     >
>     >
>     >
>     >
>     > On Fri, Oct 5, 2018 at 7:20 PM Romain Manni-Bucau
>     <rmannibucau@gmail.com <ma...@gmail.com>
>     > <mailto:rmannibucau@gmail.com <ma...@gmail.com>>> wrote:
>     >
>     >     Hi Arif,
>     >
>     >     AFAIK bytebuddy code is not java 11 friendly otherwise it runs
>     (but
>     >     it means your pipeline is very very simple since it does not
>     have a
>     >     dofn ;)) if your engine supports it. Also note that the
>     modules not
>     >     being named you can have to use some weird import names or even
>     >     unstable ones if you want to use modules (but there is no real
>     >     reason to do that yet in java).
>     >
>     >     Romain Manni-Bucau
>     >     @rmannibucau <https://twitter.com/rmannibucau> |  Blog
>     >     <https://rmannibucau.metawerx.net/> | Old Blog
>     >     <http://rmannibucau.wordpress.com> | Github
>     >     <https://github.com/rmannibucau> | LinkedIn
>     >     <https://www.linkedin.com/in/rmannibucau> | Book
>     >   
>      <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>     >
>     >
>     >     Le ven. 5 oct. 2018 à 19:10, Arif Kasim <arifkasim@google.com
>     <ma...@google.com>
>     >     <mailto:arifkasim@google.com <ma...@google.com>>> a
>     écrit :
>     >
>     >         Hello,
>     >         What's the status of java version > 8 support for beam?
>     Thanks.
>     >
>     >         -Arif.
>     >
> 
>     -- 
>     Jean-Baptiste Onofré
>     jbonofre@apache.org <ma...@apache.org>
>     http://blog.nanthrax.net
>     Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Java > 8 support

Posted by Romain Manni-Bucau <rm...@gmail.com>.

I dont think so JB - until you want to use jlink but it is out of scope of
beam - and since j9 didnt break unsafe it should work.

Le sam. 6 oct. 2018 17:23, Jean-Baptiste Onofré <jb...@nanthrax.net> a écrit :

> Same thing, it requires some change in the dependency, especially for
> the Jigsaw modules.
>
> Regards
> JB
>
> On 06/10/2018 15:01, Arif Kasim wrote:
> > Thanks all. What about Java 9 support?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >     *  •  **Arif Kasim**
> >     **  • * Strategic Cloud Engineer*
> >     **  •  *Google, Inc.*
> >     *  •  arifkasim@google.com <ma...@google.com>
> >
> >
> >
> >
> > On Fri, Oct 5, 2018 at 7:20 PM Romain Manni-Bucau <rmannibucau@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> >     Hi Arif,
> >
> >     AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but
> >     it means your pipeline is very very simple since it does not have a
> >     dofn ;)) if your engine supports it. Also note that the modules not
> >     being named you can have to use some weird import names or even
> >     unstable ones if you want to use modules (but there is no real
> >     reason to do that yet in java).
> >
> >     Romain Manni-Bucau
> >     @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> >     <https://rmannibucau.metawerx.net/> | Old Blog
> >     <http://rmannibucau.wordpress.com> | Github
> >     <https://github.com/rmannibucau> | LinkedIn
> >     <https://www.linkedin.com/in/rmannibucau> | Book
> >     <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> >
> >
> >     Le ven. 5 oct. 2018 à 19:10, Arif Kasim <arifkasim@google.com
> >     <ma...@google.com>> a écrit :
> >
> >         Hello,
> >         What's the status of java version > 8 support for beam? Thanks.
> >
> >         -Arif.
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Java > 8 support

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.

Same thing, it requires some change in the dependency, especially for
the Jigsaw modules.

Regards
JB

On 06/10/2018 15:01, Arif Kasim wrote:
> Thanks all. What about Java 9 support?
> 
> 
> 	
> 	
> 	
> 	
> 
> 	
> 
>     *  •  **Arif Kasim**
>     **  • * Strategic Cloud Engineer*
>     **  •  *Google, Inc.*
>     *  •  arifkasim@google.com <ma...@google.com>  
> 
> 
> 
> 
> On Fri, Oct 5, 2018 at 7:20 PM Romain Manni-Bucau <rmannibucau@gmail.com
> <ma...@gmail.com>> wrote:
> 
>     Hi Arif,
> 
>     AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but
>     it means your pipeline is very very simple since it does not have a
>     dofn ;)) if your engine supports it. Also note that the modules not
>     being named you can have to use some weird import names or even
>     unstable ones if you want to use modules (but there is no real
>     reason to do that yet in java).
> 
>     Romain Manni-Bucau
>     @rmannibucau <https://twitter.com/rmannibucau> |  Blog
>     <https://rmannibucau.metawerx.net/> | Old Blog
>     <http://rmannibucau.wordpress.com> | Github
>     <https://github.com/rmannibucau> | LinkedIn
>     <https://www.linkedin.com/in/rmannibucau> | Book
>     <https://www.packtpub.com/application-development/java-ee-8-high-performance>
> 
> 
>     Le ven. 5 oct. 2018 à 19:10, Arif Kasim <arifkasim@google.com
>     <ma...@google.com>> a écrit :
> 
>         Hello,
>         What's the status of java version > 8 support for beam? Thanks.
> 
>         -Arif.
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Java > 8 support

Posted by Arif Kasim <ar...@google.com>.

Thanks all. What about Java 9 support?




*  •  **Arif Kasim*
*  • * Strategic Cloud Engineer
*  •  *Google, Inc.
  •  arifkasim@google.com




On Fri, Oct 5, 2018 at 7:20 PM Romain Manni-Bucau <rm...@gmail.com>
wrote:

> Hi Arif,
>
> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but it
> means your pipeline is very very simple since it does not have a dofn ;))
> if your engine supports it. Also note that the modules not being named you
> can have to use some weird import names or even unstable ones if you want
> to use modules (but there is no real reason to do that yet in java).
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com> a écrit :
>
>> Hello,
>> What's the status of java version > 8 support for beam? Thanks.
>>
>> -Arif.
>>
>

Re: Java > 8 support

Posted by Romain Manni-Bucau <rm...@gmail.com>.

Hi Arif,

AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but it
means your pipeline is very very simple since it does not have a dofn ;))
if your engine supports it. Also note that the modules not being named you
can have to use some weird import names or even unstable ones if you want
to use modules (but there is no real reason to do that yet in java).

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le ven. 5 oct. 2018 à 19:10, Arif Kasim <ar...@google.com> a écrit :

> Hello,
> What's the status of java version > 8 support for beam? Thanks.
>
> -Arif.
>

Re: Java > 8 support

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.

Hi Arif,

Java 8 is supported but we also plan some update in the core (like
removing joda-time to use "native" Java 8 lib).

I guess you are mostly asking for Java 11, I just try a rough build and
indeed, we have lot of changes/updates to do.

I did the update to Java 11 in Apache Karaf, and it required several
changes (some with important impact).

I created a Jira to track progress on this one:
https://issues.apache.org/jira/browse/BEAM-5666

Regards
JB

On 05/10/2018 19:09, Arif Kasim wrote:
> Hello,
> What's the status of java version > 8 support for beam? Thanks.
> 
> -Arif.

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com