Re: Java 11 compatibility question

[Łukasz Gajowy <lg...@apache.org>]

Hi all,

I want to contribute more actively to this and push Beam as close as
currently possible towards Java11 both in terms of running and compiling
the project with it.

I needed a bigger picture so I created a spreadsheet to have a clear
roadmap for the whole process. It starts with testing existing java 8
artifacts (part of this is already done) and continues with providing
compile support and later JPMS support for the project. I figured that
before I storm JIRA with some new subtasks of BEAM-2530 it's good to have
something like this thought through. I hope this is also helpful for others
if they want to help to migrate the project to Java 11. Here's the
spreadsheet:

https://s.apache.org/java11-support-roadmap

Any comments highly appreciated. :)

FWIW, grpc devs ’’will be looking into options" for resolving the
above-mentioned gprc issue "this quarter":
https://github.com/grpc/grpc-java/issues/3522

Thanks!
Łukasz

śr., 21 sie 2019 o 20:46 Kenneth Knowles <ke...@apache.org> napisał(a):

>
>
> On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <el...@ibiblio.org>
> wrote:
>
>>
>>
>> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> a per case approach (the exception could be portable runners not based
>>> on Java).
>>>
>>> Of course other definitions of being Java 11 compatible are interesting
>>> but probably not part of our current scope. Actions like change the
>>> codebase to use Java 11 specific APIs / idioms, publish Java 11 specific
>>> artifacts or use Java Platform Modules (JPM). All of these may be nice to
>>> have but are probably less important for end users who may just want to be
>>> able to use Beam in its current form in Java 11 VMs.
>>>
>>> What do others think? Is this enough to announce Java 11 compatibility
>>> and add the documentation to the webpage?
>>>
>>
>> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam really
>> does need to be compatible with JPMS-using apps. The bare minimum here is
>> avoiding split packages, and that needs to include all transitive
>> dependencies, not just Beam itself. I don't think we meet that bar now.
>>
>
> We definitely don't meet the basic bar ourselves, unless someone has done
> a lot of clean up. We've had classes shuffled from jar to jar quite a lot
> without changing their namespace appropriately. It may be mostly limited to
> runner-facing pieces, but I expect for a number of runners (notably the
> Direct Runner) that is enough to bite users.
>
> Kenn
>
>
>>
>> --
>> Elliotte Rusty Harold
>> elharo@ibiblio.org
>>
>

Re: Java 11 compatibility question

[Łukasz Gajowy <lu...@gmail.com>]

Thank you for mentioning all that.

I'm aware of the possible issues. There's a lot of other issues (smaller or
greater) like missing APIs, dependencies being not compatible with jdk11
itself (even when used without JPMS) and so on. The idea is to face all the
problems, create problem-specific jiras and try to mitigate them while
completing consequent steps of the roadmap. I also see big value in testing
already accomplished steps in java11_preCommit suite and preventing
regressions this way.

Regarding JPMS - we need to wait at least for grpc and google-cloud-java to
deal with the split packages problem as it's stated above. I suspect that
other dependencies also may have this problem (e.g. I couldn't find a
ticket for JPMS support in Spark's jira). This is why JPMS compatibility is
located in a later phase of the roadmap.

Thanks!

pt., 18 paź 2019 o 23:39 Luke Cwik <lc...@google.com> napisał(a):

> There are some changes with Java where the system class loader is no
> longer a URL class loader[1].
>
> Also reflection is changing such that non-public fields/methods aren't
> accessible which we (or our dependencies) may be doing. Not sure how our
> usage of bytecode generation/proxies will need to change.
>
> Finally, for JPMS support to actually be usable, you'll need to update our
> deps to JPMS compatible as well.
>
> 1: https://issues.apache.org/jira/browse/BEAM-5495
>
> On Fri, Oct 18, 2019 at 6:14 AM Łukasz Gajowy <lg...@apache.org> wrote:
>
>> Hi all,
>>
>> I want to contribute more actively to this and push Beam as close as
>> currently possible towards Java11 both in terms of running and compiling
>> the project with it.
>>
>> I needed a bigger picture so I created a spreadsheet to have a clear
>> roadmap for the whole process. It starts with testing existing java 8
>> artifacts (part of this is already done) and continues with providing
>> compile support and later JPMS support for the project. I figured that
>> before I storm JIRA with some new subtasks of BEAM-2530 it's good to
>> have something like this thought through. I hope this is also helpful for
>> others if they want to help to migrate the project to Java 11. Here's the
>> spreadsheet:
>>
>> https://s.apache.org/java11-support-roadmap
>>
>> Any comments highly appreciated. :)
>>
>> FWIW, grpc devs ’’will be looking into options" for resolving the
>> above-mentioned gprc issue "this quarter":
>> https://github.com/grpc/grpc-java/issues/3522
>>
>> Thanks!
>> Łukasz
>>
>> śr., 21 sie 2019 o 20:46 Kenneth Knowles <ke...@apache.org> napisał(a):
>>
>>>
>>>
>>> On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <
>>> elharo@ibiblio.org> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>
>>>>> a per case approach (the exception could be portable runners not based
>>>>> on Java).
>>>>>
>>>>> Of course other definitions of being Java 11 compatible are
>>>>> interesting but probably not part of our current scope. Actions like change
>>>>> the codebase to use Java 11 specific APIs / idioms, publish Java 11
>>>>> specific artifacts or use Java Platform Modules (JPM). All of these may be
>>>>> nice to have but are probably less important for end users who may just
>>>>> want to be able to use Beam in its current form in Java 11 VMs.
>>>>>
>>>>> What do others think? Is this enough to announce Java 11 compatibility
>>>>> and add the documentation to the webpage?
>>>>>
>>>>
>>>> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam
>>>> really does need to be compatible with JPMS-using apps. The bare minimum
>>>> here is avoiding split packages, and that needs to include all transitive
>>>> dependencies, not just Beam itself. I don't think we meet that bar now.
>>>>
>>>
>>> We definitely don't meet the basic bar ourselves, unless someone has
>>> done a lot of clean up. We've had classes shuffled from jar to jar quite a
>>> lot without changing their namespace appropriately. It may be mostly
>>> limited to runner-facing pieces, but I expect for a number of runners
>>> (notably the Direct Runner) that is enough to bite users.
>>>
>>> Kenn
>>>
>>>
>>>>
>>>> --
>>>> Elliotte Rusty Harold
>>>> elharo@ibiblio.org
>>>>
>>>

Re: Java 11 compatibility question

[Luke Cwik <lc...@google.com>]

There are some changes with Java where the system class loader is no longer
a URL class loader[1].

Also reflection is changing such that non-public fields/methods aren't
accessible which we (or our dependencies) may be doing. Not sure how our
usage of bytecode generation/proxies will need to change.

Finally, for JPMS support to actually be usable, you'll need to update our
deps to JPMS compatible as well.

1: https://issues.apache.org/jira/browse/BEAM-5495

On Fri, Oct 18, 2019 at 6:14 AM Łukasz Gajowy <lg...@apache.org> wrote:

> Hi all,
>
> I want to contribute more actively to this and push Beam as close as
> currently possible towards Java11 both in terms of running and compiling
> the project with it.
>
> I needed a bigger picture so I created a spreadsheet to have a clear
> roadmap for the whole process. It starts with testing existing java 8
> artifacts (part of this is already done) and continues with providing
> compile support and later JPMS support for the project. I figured that
> before I storm JIRA with some new subtasks of BEAM-2530 it's good to have
> something like this thought through. I hope this is also helpful for others
> if they want to help to migrate the project to Java 11. Here's the
> spreadsheet:
>
> https://s.apache.org/java11-support-roadmap
>
> Any comments highly appreciated. :)
>
> FWIW, grpc devs ’’will be looking into options" for resolving the
> above-mentioned gprc issue "this quarter":
> https://github.com/grpc/grpc-java/issues/3522
>
> Thanks!
> Łukasz
>
> śr., 21 sie 2019 o 20:46 Kenneth Knowles <ke...@apache.org> napisał(a):
>
>>
>>
>> On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <el...@ibiblio.org>
>> wrote:
>>
>>>
>>>
>>> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> a per case approach (the exception could be portable runners not based
>>>> on Java).
>>>>
>>>> Of course other definitions of being Java 11 compatible are interesting
>>>> but probably not part of our current scope. Actions like change the
>>>> codebase to use Java 11 specific APIs / idioms, publish Java 11 specific
>>>> artifacts or use Java Platform Modules (JPM). All of these may be nice to
>>>> have but are probably less important for end users who may just want to be
>>>> able to use Beam in its current form in Java 11 VMs.
>>>>
>>>> What do others think? Is this enough to announce Java 11 compatibility
>>>> and add the documentation to the webpage?
>>>>
>>>
>>> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam really
>>> does need to be compatible with JPMS-using apps. The bare minimum here is
>>> avoiding split packages, and that needs to include all transitive
>>> dependencies, not just Beam itself. I don't think we meet that bar now.
>>>
>>
>> We definitely don't meet the basic bar ourselves, unless someone has done
>> a lot of clean up. We've had classes shuffled from jar to jar quite a lot
>> without changing their namespace appropriately. It may be mostly limited to
>> runner-facing pieces, but I expect for a number of runners (notably the
>> Direct Runner) that is enough to bite users.
>>
>> Kenn
>>
>>
>>>
>>> --
>>> Elliotte Rusty Harold
>>> elharo@ibiblio.org
>>>
>>