You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Michał Walenia <mi...@polidea.com> on 2019/08/07 13:41:34 UTC
Java 11 compatibility question
Hi everyone,
I want to gather the collective knowledge here about Java 11 compatibility
and ask about the tests needed to deem Beam compatible with JDK 11.
Right now concerning testing JDK 11 compatibility I implemented:
- Jenkins jobs for running ValidatesRunner test sets in both Direct and
Dataflow runners, [1], [2]
- ValidatesRunner portability API tests for Dataflow [3],
- examples in normal and portable mode for the Dataflow runner [4], [5].
Are these tests sufficient to say that we’re java 11 compatible? What other
aspects do we need to test to be able to say that?
Regards,
Michał
[1]
https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Direct/
[2]
https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/
[3]
https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow/
[4] https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow/
[5]
https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow_Portability/
--
Michał Walenia
Polidea <https://www.polidea.com/> | Software Engineer
M: +48 791 432 002 <+48791432002>
E: michal.walenia@polidea.com
Unique Tech
Check out our projects! <https://www.polidea.com/our-work>
Re: Java 11 compatibility question
Posted by Łukasz Gajowy <lu...@gmail.com>.
Thank you for mentioning all that.
I'm aware of the possible issues. There's a lot of other issues (smaller or
greater) like missing APIs, dependencies being not compatible with jdk11
itself (even when used without JPMS) and so on. The idea is to face all the
problems, create problem-specific jiras and try to mitigate them while
completing consequent steps of the roadmap. I also see big value in testing
already accomplished steps in java11_preCommit suite and preventing
regressions this way.
Regarding JPMS - we need to wait at least for grpc and google-cloud-java to
deal with the split packages problem as it's stated above. I suspect that
other dependencies also may have this problem (e.g. I couldn't find a
ticket for JPMS support in Spark's jira). This is why JPMS compatibility is
located in a later phase of the roadmap.
Thanks!
pt., 18 paź 2019 o 23:39 Luke Cwik <lc...@google.com> napisał(a):
> There are some changes with Java where the system class loader is no
> longer a URL class loader[1].
>
> Also reflection is changing such that non-public fields/methods aren't
> accessible which we (or our dependencies) may be doing. Not sure how our
> usage of bytecode generation/proxies will need to change.
>
> Finally, for JPMS support to actually be usable, you'll need to update our
> deps to JPMS compatible as well.
>
> 1: https://issues.apache.org/jira/browse/BEAM-5495
>
> On Fri, Oct 18, 2019 at 6:14 AM Łukasz Gajowy <lg...@apache.org> wrote:
>
>> Hi all,
>>
>> I want to contribute more actively to this and push Beam as close as
>> currently possible towards Java11 both in terms of running and compiling
>> the project with it.
>>
>> I needed a bigger picture so I created a spreadsheet to have a clear
>> roadmap for the whole process. It starts with testing existing java 8
>> artifacts (part of this is already done) and continues with providing
>> compile support and later JPMS support for the project. I figured that
>> before I storm JIRA with some new subtasks of BEAM-2530 it's good to
>> have something like this thought through. I hope this is also helpful for
>> others if they want to help to migrate the project to Java 11. Here's the
>> spreadsheet:
>>
>> https://s.apache.org/java11-support-roadmap
>>
>> Any comments highly appreciated. :)
>>
>> FWIW, grpc devs ’’will be looking into options" for resolving the
>> above-mentioned gprc issue "this quarter":
>> https://github.com/grpc/grpc-java/issues/3522
>>
>> Thanks!
>> Łukasz
>>
>> śr., 21 sie 2019 o 20:46 Kenneth Knowles <ke...@apache.org> napisał(a):
>>
>>>
>>>
>>> On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <
>>> elharo@ibiblio.org> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>
>>>>> a per case approach (the exception could be portable runners not based
>>>>> on Java).
>>>>>
>>>>> Of course other definitions of being Java 11 compatible are
>>>>> interesting but probably not part of our current scope. Actions like change
>>>>> the codebase to use Java 11 specific APIs / idioms, publish Java 11
>>>>> specific artifacts or use Java Platform Modules (JPM). All of these may be
>>>>> nice to have but are probably less important for end users who may just
>>>>> want to be able to use Beam in its current form in Java 11 VMs.
>>>>>
>>>>> What do others think? Is this enough to announce Java 11 compatibility
>>>>> and add the documentation to the webpage?
>>>>>
>>>>
>>>> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam
>>>> really does need to be compatible with JPMS-using apps. The bare minimum
>>>> here is avoiding split packages, and that needs to include all transitive
>>>> dependencies, not just Beam itself. I don't think we meet that bar now.
>>>>
>>>
>>> We definitely don't meet the basic bar ourselves, unless someone has
>>> done a lot of clean up. We've had classes shuffled from jar to jar quite a
>>> lot without changing their namespace appropriately. It may be mostly
>>> limited to runner-facing pieces, but I expect for a number of runners
>>> (notably the Direct Runner) that is enough to bite users.
>>>
>>> Kenn
>>>
>>>
>>>>
>>>> --
>>>> Elliotte Rusty Harold
>>>> elharo@ibiblio.org
>>>>
>>>
Re: Java 11 compatibility question
Posted by Luke Cwik <lc...@google.com>.
There are some changes with Java where the system class loader is no longer
a URL class loader[1].
Also reflection is changing such that non-public fields/methods aren't
accessible which we (or our dependencies) may be doing. Not sure how our
usage of bytecode generation/proxies will need to change.
Finally, for JPMS support to actually be usable, you'll need to update our
deps to JPMS compatible as well.
1: https://issues.apache.org/jira/browse/BEAM-5495
On Fri, Oct 18, 2019 at 6:14 AM Łukasz Gajowy <lg...@apache.org> wrote:
> Hi all,
>
> I want to contribute more actively to this and push Beam as close as
> currently possible towards Java11 both in terms of running and compiling
> the project with it.
>
> I needed a bigger picture so I created a spreadsheet to have a clear
> roadmap for the whole process. It starts with testing existing java 8
> artifacts (part of this is already done) and continues with providing
> compile support and later JPMS support for the project. I figured that
> before I storm JIRA with some new subtasks of BEAM-2530 it's good to have
> something like this thought through. I hope this is also helpful for others
> if they want to help to migrate the project to Java 11. Here's the
> spreadsheet:
>
> https://s.apache.org/java11-support-roadmap
>
> Any comments highly appreciated. :)
>
> FWIW, grpc devs ’’will be looking into options" for resolving the
> above-mentioned gprc issue "this quarter":
> https://github.com/grpc/grpc-java/issues/3522
>
> Thanks!
> Łukasz
>
> śr., 21 sie 2019 o 20:46 Kenneth Knowles <ke...@apache.org> napisał(a):
>
>>
>>
>> On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <el...@ibiblio.org>
>> wrote:
>>
>>>
>>>
>>> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> a per case approach (the exception could be portable runners not based
>>>> on Java).
>>>>
>>>> Of course other definitions of being Java 11 compatible are interesting
>>>> but probably not part of our current scope. Actions like change the
>>>> codebase to use Java 11 specific APIs / idioms, publish Java 11 specific
>>>> artifacts or use Java Platform Modules (JPM). All of these may be nice to
>>>> have but are probably less important for end users who may just want to be
>>>> able to use Beam in its current form in Java 11 VMs.
>>>>
>>>> What do others think? Is this enough to announce Java 11 compatibility
>>>> and add the documentation to the webpage?
>>>>
>>>
>>> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam really
>>> does need to be compatible with JPMS-using apps. The bare minimum here is
>>> avoiding split packages, and that needs to include all transitive
>>> dependencies, not just Beam itself. I don't think we meet that bar now.
>>>
>>
>> We definitely don't meet the basic bar ourselves, unless someone has done
>> a lot of clean up. We've had classes shuffled from jar to jar quite a lot
>> without changing their namespace appropriately. It may be mostly limited to
>> runner-facing pieces, but I expect for a number of runners (notably the
>> Direct Runner) that is enough to bite users.
>>
>> Kenn
>>
>>
>>>
>>> --
>>> Elliotte Rusty Harold
>>> elharo@ibiblio.org
>>>
>>
Re: Java 11 compatibility question
Posted by Łukasz Gajowy <lg...@apache.org>.
Hi all,
I want to contribute more actively to this and push Beam as close as
currently possible towards Java11 both in terms of running and compiling
the project with it.
I needed a bigger picture so I created a spreadsheet to have a clear
roadmap for the whole process. It starts with testing existing java 8
artifacts (part of this is already done) and continues with providing
compile support and later JPMS support for the project. I figured that
before I storm JIRA with some new subtasks of BEAM-2530 it's good to have
something like this thought through. I hope this is also helpful for others
if they want to help to migrate the project to Java 11. Here's the
spreadsheet:
https://s.apache.org/java11-support-roadmap
Any comments highly appreciated. :)
FWIW, grpc devs ’’will be looking into options" for resolving the
above-mentioned gprc issue "this quarter":
https://github.com/grpc/grpc-java/issues/3522
Thanks!
Łukasz
śr., 21 sie 2019 o 20:46 Kenneth Knowles <ke...@apache.org> napisał(a):
>
>
> On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <el...@ibiblio.org>
> wrote:
>
>>
>>
>> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> a per case approach (the exception could be portable runners not based
>>> on Java).
>>>
>>> Of course other definitions of being Java 11 compatible are interesting
>>> but probably not part of our current scope. Actions like change the
>>> codebase to use Java 11 specific APIs / idioms, publish Java 11 specific
>>> artifacts or use Java Platform Modules (JPM). All of these may be nice to
>>> have but are probably less important for end users who may just want to be
>>> able to use Beam in its current form in Java 11 VMs.
>>>
>>> What do others think? Is this enough to announce Java 11 compatibility
>>> and add the documentation to the webpage?
>>>
>>
>> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam really
>> does need to be compatible with JPMS-using apps. The bare minimum here is
>> avoiding split packages, and that needs to include all transitive
>> dependencies, not just Beam itself. I don't think we meet that bar now.
>>
>
> We definitely don't meet the basic bar ourselves, unless someone has done
> a lot of clean up. We've had classes shuffled from jar to jar quite a lot
> without changing their namespace appropriately. It may be mostly limited to
> runner-facing pieces, but I expect for a number of runners (notably the
> Direct Runner) that is enough to bite users.
>
> Kenn
>
>
>>
>> --
>> Elliotte Rusty Harold
>> elharo@ibiblio.org
>>
>
Re: Java 11 compatibility question
Posted by Kenneth Knowles <ke...@apache.org>.
On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <el...@ibiblio.org>
wrote:
>
>
> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> a per case approach (the exception could be portable runners not based on
>> Java).
>>
>> Of course other definitions of being Java 11 compatible are interesting
>> but probably not part of our current scope. Actions like change the
>> codebase to use Java 11 specific APIs / idioms, publish Java 11 specific
>> artifacts or use Java Platform Modules (JPM). All of these may be nice to
>> have but are probably less important for end users who may just want to be
>> able to use Beam in its current form in Java 11 VMs.
>>
>> What do others think? Is this enough to announce Java 11 compatibility
>> and add the documentation to the webpage?
>>
>
> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam really
> does need to be compatible with JPMS-using apps. The bare minimum here is
> avoiding split packages, and that needs to include all transitive
> dependencies, not just Beam itself. I don't think we meet that bar now.
>
We definitely don't meet the basic bar ourselves, unless someone has done a
lot of clean up. We've had classes shuffled from jar to jar quite a lot
without changing their namespace appropriately. It may be mostly limited to
runner-facing pieces, but I expect for a number of runners (notably the
Direct Runner) that is enough to bite users.
Kenn
>
> --
> Elliotte Rusty Harold
> elharo@ibiblio.org
>
Re: Java 11 compatibility question
Posted by Łukasz Gajowy <lg...@apache.org>.
https://issues.apache.org/jira/browse/BEAM-8024 I created the other issue
regarding importing beam to a Java11 project that uses JPMS. I confirmed*
in a pet project that this is happening (linked in the issue).
*no shock here, I just wanted to play with it.
Łukasz
śr., 21 sie 2019 o 14:53 Łukasz Gajowy <lg...@apache.org> napisał(a):
> Thank you, Elliotte for showing us the issues with JPMS.
>
> So maybe we should just announce for end users that they can run Beam
> pipelines in Java 11 but that for the moment Beam modules cannot be
> used in Java 11 module style. I know that there is already a lot of
> fear around Java 8 not being maintained so this will probably be
> perceived well even if not perfect.
>
> +1 for sharing this information to the users. IMO this is really valuable
> knowledge.
>
> Łukasz
>
>
>
>
> śr., 21 sie 2019 o 12:22 Ismaël Mejía <ie...@gmail.com> napisał(a):
>
>> Thanks again Elliotte for the clear information and references. It
>> seems that being compatible with Java 11 modules will be more elusive
>> than expected considering the transitive dependencies. Do you (or
>> someone else) knows if there there is a plugin or easy way to discover
>> this?
>>
>> I think that solving this for transitive dependencies will be elusive
>> for a LONG LONG time (I had not even thought about IO modules that
>> have dependencies and commonly live in ‘older’ stable versions). So
>> maybe we should just announce for end users that they can run Beam
>> pipelines in Java 11 but that for the moment Beam modules cannot be
>> used in Java 11 module style. I know that there is already a lot of
>> fear around Java 8 not being maintained so this will probably be
>> perceived well even if not perfect.
>>
>> I filled https://issues.apache.org/jira/browse/BEAM-8021 to set
>> explicitly the module names. We will probably need also to put some
>> validation in place that the jars always include the module name so
>> new modules don’t forget to do so.
>>
>>
>> On Wed, Aug 21, 2019 at 3:28 AM Elliotte Rusty Harold
>> <el...@ibiblio.org> wrote:
>> >
>> > If somebody is using JPMS and they attempt to import beam, they get a
>> > compile time error. Some other projects I work on have been getting
>> > user reports about this.
>> >
>> > See
>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-19.md
>> > for more details.
>> >
>> > On Tue, Aug 20, 2019 at 5:30 PM Ahmet Altay <al...@google.com> wrote:
>> > >
>> > >
>> > >
>> > > On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <
>> elharo@ibiblio.org> wrote:
>> > >>
>> > >>
>> > >>
>> > >> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com>
>> wrote:
>> > >>>
>> > >>> a per case approach (the exception could be portable runners not
>> based on Java).
>> > >>>
>> > >>> Of course other definitions of being Java 11 compatible are
>> interesting but probably not part of our current scope. Actions like change
>> the codebase to use Java 11 specific APIs / idioms, publish Java 11
>> specific artifacts or use Java Platform Modules (JPM). All of these may be
>> nice to have but are probably less important for end users who may just
>> want to be able to use Beam in its current form in Java 11 VMs.
>> > >>>
>> > >>> What do others think? Is this enough to announce Java 11
>> compatibility and add the documentation to the webpage?
>> > >>
>> > >>
>> > >> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam
>> really does need to be compatible with JPMS-using apps. The bare minimum
>> here is avoiding split packages, and that needs to include all transitive
>> dependencies, not just Beam itself. I don't think we meet that bar now.
>> > >
>> > >
>> > > For my understanding, what would be the limitations of Beam's
>> dependencies having split dependencies? Would it limit Beam users from
>> using 3rd party libraries that require JPMS supports? Would it be in scope
>> for Beam to get its dependencies to meet a certain bar?
>> > >
>> > > Ismaël's definition of being able to run Beam published dependencies
>> in Java 11 VM sounds enough to me "to announce Java 11 compatibility _for
>> Beam_".
>> > >
>> > >>
>> > >>
>> > >> --
>> > >> Elliotte Rusty Harold
>> > >> elharo@ibiblio.org
>> >
>> >
>> >
>> > --
>> > Elliotte Rusty Harold
>> > elharo@ibiblio.org
>>
>
Re: Java 11 compatibility question
Posted by Łukasz Gajowy <lg...@apache.org>.
Thank you, Elliotte for showing us the issues with JPMS.
So maybe we should just announce for end users that they can run Beam
pipelines in Java 11 but that for the moment Beam modules cannot be
used in Java 11 module style. I know that there is already a lot of
fear around Java 8 not being maintained so this will probably be
perceived well even if not perfect.
+1 for sharing this information to the users. IMO this is really valuable
knowledge.
Łukasz
śr., 21 sie 2019 o 12:22 Ismaël Mejía <ie...@gmail.com> napisał(a):
> Thanks again Elliotte for the clear information and references. It
> seems that being compatible with Java 11 modules will be more elusive
> than expected considering the transitive dependencies. Do you (or
> someone else) knows if there there is a plugin or easy way to discover
> this?
>
> I think that solving this for transitive dependencies will be elusive
> for a LONG LONG time (I had not even thought about IO modules that
> have dependencies and commonly live in ‘older’ stable versions). So
> maybe we should just announce for end users that they can run Beam
> pipelines in Java 11 but that for the moment Beam modules cannot be
> used in Java 11 module style. I know that there is already a lot of
> fear around Java 8 not being maintained so this will probably be
> perceived well even if not perfect.
>
> I filled https://issues.apache.org/jira/browse/BEAM-8021 to set
> explicitly the module names. We will probably need also to put some
> validation in place that the jars always include the module name so
> new modules don’t forget to do so.
>
>
> On Wed, Aug 21, 2019 at 3:28 AM Elliotte Rusty Harold
> <el...@ibiblio.org> wrote:
> >
> > If somebody is using JPMS and they attempt to import beam, they get a
> > compile time error. Some other projects I work on have been getting
> > user reports about this.
> >
> > See
> https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-19.md
> > for more details.
> >
> > On Tue, Aug 20, 2019 at 5:30 PM Ahmet Altay <al...@google.com> wrote:
> > >
> > >
> > >
> > > On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <
> elharo@ibiblio.org> wrote:
> > >>
> > >>
> > >>
> > >> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com>
> wrote:
> > >>>
> > >>> a per case approach (the exception could be portable runners not
> based on Java).
> > >>>
> > >>> Of course other definitions of being Java 11 compatible are
> interesting but probably not part of our current scope. Actions like change
> the codebase to use Java 11 specific APIs / idioms, publish Java 11
> specific artifacts or use Java Platform Modules (JPM). All of these may be
> nice to have but are probably less important for end users who may just
> want to be able to use Beam in its current form in Java 11 VMs.
> > >>>
> > >>> What do others think? Is this enough to announce Java 11
> compatibility and add the documentation to the webpage?
> > >>
> > >>
> > >> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam
> really does need to be compatible with JPMS-using apps. The bare minimum
> here is avoiding split packages, and that needs to include all transitive
> dependencies, not just Beam itself. I don't think we meet that bar now.
> > >
> > >
> > > For my understanding, what would be the limitations of Beam's
> dependencies having split dependencies? Would it limit Beam users from
> using 3rd party libraries that require JPMS supports? Would it be in scope
> for Beam to get its dependencies to meet a certain bar?
> > >
> > > Ismaël's definition of being able to run Beam published dependencies
> in Java 11 VM sounds enough to me "to announce Java 11 compatibility _for
> Beam_".
> > >
> > >>
> > >>
> > >> --
> > >> Elliotte Rusty Harold
> > >> elharo@ibiblio.org
> >
> >
> >
> > --
> > Elliotte Rusty Harold
> > elharo@ibiblio.org
>
Re: Java 11 compatibility question
Posted by Ismaël Mejía <ie...@gmail.com>.
Thanks again Elliotte for the clear information and references. It
seems that being compatible with Java 11 modules will be more elusive
than expected considering the transitive dependencies. Do you (or
someone else) knows if there there is a plugin or easy way to discover
this?
I think that solving this for transitive dependencies will be elusive
for a LONG LONG time (I had not even thought about IO modules that
have dependencies and commonly live in ‘older’ stable versions). So
maybe we should just announce for end users that they can run Beam
pipelines in Java 11 but that for the moment Beam modules cannot be
used in Java 11 module style. I know that there is already a lot of
fear around Java 8 not being maintained so this will probably be
perceived well even if not perfect.
I filled https://issues.apache.org/jira/browse/BEAM-8021 to set
explicitly the module names. We will probably need also to put some
validation in place that the jars always include the module name so
new modules don’t forget to do so.
On Wed, Aug 21, 2019 at 3:28 AM Elliotte Rusty Harold
<el...@ibiblio.org> wrote:
>
> If somebody is using JPMS and they attempt to import beam, they get a
> compile time error. Some other projects I work on have been getting
> user reports about this.
>
> See https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-19.md
> for more details.
>
> On Tue, Aug 20, 2019 at 5:30 PM Ahmet Altay <al...@google.com> wrote:
> >
> >
> >
> > On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <el...@ibiblio.org> wrote:
> >>
> >>
> >>
> >> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com> wrote:
> >>>
> >>> a per case approach (the exception could be portable runners not based on Java).
> >>>
> >>> Of course other definitions of being Java 11 compatible are interesting but probably not part of our current scope. Actions like change the codebase to use Java 11 specific APIs / idioms, publish Java 11 specific artifacts or use Java Platform Modules (JPM). All of these may be nice to have but are probably less important for end users who may just want to be able to use Beam in its current form in Java 11 VMs.
> >>>
> >>> What do others think? Is this enough to announce Java 11 compatibility and add the documentation to the webpage?
> >>
> >>
> >> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam really does need to be compatible with JPMS-using apps. The bare minimum here is avoiding split packages, and that needs to include all transitive dependencies, not just Beam itself. I don't think we meet that bar now.
> >
> >
> > For my understanding, what would be the limitations of Beam's dependencies having split dependencies? Would it limit Beam users from using 3rd party libraries that require JPMS supports? Would it be in scope for Beam to get its dependencies to meet a certain bar?
> >
> > Ismaël's definition of being able to run Beam published dependencies in Java 11 VM sounds enough to me "to announce Java 11 compatibility _for Beam_".
> >
> >>
> >>
> >> --
> >> Elliotte Rusty Harold
> >> elharo@ibiblio.org
>
>
>
> --
> Elliotte Rusty Harold
> elharo@ibiblio.org
Re: Java 11 compatibility question
Posted by Elliotte Rusty Harold <el...@ibiblio.org>.
If somebody is using JPMS and they attempt to import beam, they get a
compile time error. Some other projects I work on have been getting
user reports about this.
See https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-19.md
for more details.
On Tue, Aug 20, 2019 at 5:30 PM Ahmet Altay <al...@google.com> wrote:
>
>
>
> On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <el...@ibiblio.org> wrote:
>>
>>
>>
>> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>> a per case approach (the exception could be portable runners not based on Java).
>>>
>>> Of course other definitions of being Java 11 compatible are interesting but probably not part of our current scope. Actions like change the codebase to use Java 11 specific APIs / idioms, publish Java 11 specific artifacts or use Java Platform Modules (JPM). All of these may be nice to have but are probably less important for end users who may just want to be able to use Beam in its current form in Java 11 VMs.
>>>
>>> What do others think? Is this enough to announce Java 11 compatibility and add the documentation to the webpage?
>>
>>
>> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam really does need to be compatible with JPMS-using apps. The bare minimum here is avoiding split packages, and that needs to include all transitive dependencies, not just Beam itself. I don't think we meet that bar now.
>
>
> For my understanding, what would be the limitations of Beam's dependencies having split dependencies? Would it limit Beam users from using 3rd party libraries that require JPMS supports? Would it be in scope for Beam to get its dependencies to meet a certain bar?
>
> Ismaël's definition of being able to run Beam published dependencies in Java 11 VM sounds enough to me "to announce Java 11 compatibility _for Beam_".
>
>>
>>
>> --
>> Elliotte Rusty Harold
>> elharo@ibiblio.org
--
Elliotte Rusty Harold
elharo@ibiblio.org
Re: Java 11 compatibility question
Posted by Ahmet Altay <al...@google.com>.
On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold <el...@ibiblio.org>
wrote:
>
>
> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> a per case approach (the exception could be portable runners not based on
>> Java).
>>
>> Of course other definitions of being Java 11 compatible are interesting
>> but probably not part of our current scope. Actions like change the
>> codebase to use Java 11 specific APIs / idioms, publish Java 11 specific
>> artifacts or use Java Platform Modules (JPM). All of these may be nice to
>> have but are probably less important for end users who may just want to be
>> able to use Beam in its current form in Java 11 VMs.
>>
>> What do others think? Is this enough to announce Java 11 compatibility
>> and add the documentation to the webpage?
>>
>
> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam really
> does need to be compatible with JPMS-using apps. The bare minimum here is
> avoiding split packages, and that needs to include all transitive
> dependencies, not just Beam itself. I don't think we meet that bar now.
>
For my understanding, what would be the limitations of Beam's dependencies
having split dependencies? Would it limit Beam users from using 3rd party
libraries that require JPMS supports? Would it be in scope for Beam to get
its dependencies to meet a certain bar?
Ismaël's definition of being able to run Beam published dependencies in
Java 11 VM sounds enough to me "to announce Java 11 compatibility _for
Beam_".
>
> --
> Elliotte Rusty Harold
> elharo@ibiblio.org
>
Re: Java 11 compatibility question
Posted by Elliotte Rusty Harold <el...@ibiblio.org>.
On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía <ie...@gmail.com> wrote:
> a per case approach (the exception could be portable runners not based on
> Java).
>
> Of course other definitions of being Java 11 compatible are interesting
> but probably not part of our current scope. Actions like change the
> codebase to use Java 11 specific APIs / idioms, publish Java 11 specific
> artifacts or use Java Platform Modules (JPM). All of these may be nice to
> have but are probably less important for end users who may just want to be
> able to use Beam in its current form in Java 11 VMs.
>
> What do others think? Is this enough to announce Java 11 compatibility and
> add the documentation to the webpage?
>
No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam really
does need to be compatible with JPMS-using apps. The bare minimum here is
avoiding split packages, and that needs to include all transitive
dependencies, not just Beam itself. I don't think we meet that bar now.
--
Elliotte Rusty Harold
elharo@ibiblio.org
Re: Java 11 compatibility question
Posted by Ismaël Mejía <ie...@gmail.com>.
Many different people understand different things for Java 11 compatibility
and probably the easiest path for us is to define exactly what we (Beam)
meant with being Java 11 compatible.
The definition that Michał gave seems aligned with the current scope. Beam
published artifacts compiled with Java 8 work correctly in machines that
run a Java 11 VM (and we validate this with the ValidatesRunner tests +
examples).
By this definition, we may consider that both Direct runner and Dataflow
are Java 11 compatible, but we cannot argue this for other runners like
Spark or Flink because neither of those natively support Java 11 yet [1]
[2]. Probably we should specify for which runners we do validate Java 11
compatibility in the CI since this will evolve once Flink 1.10 and Spark
3.0 are released.
We could argue that with portability supporting Java 11 and newer versions
will be easier, but the runner part is still tight with the execution
system so for the moment we will have a per case approach (the exception
could be portable runners not based on Java).
Of course other definitions of being Java 11 compatible are interesting but
probably not part of our current scope. Actions like change the codebase to
use Java 11 specific APIs / idioms, publish Java 11 specific artifacts or
use Java Platform Modules (JPM). All of these may be nice to have but are
probably less important for end users who may just want to be able to use
Beam in its current form in Java 11 VMs.
What do others think? Is this enough to announce Java 11 compatibility and
add the documentation to the webpage?
Ismaël
[1] https://issues.apache.org/jira/browse/FLINK-10725
[2] https://issues.apache.org/jira/browse/SPARK-24417
ps. Thanks Elliotte for the pointer to the best practices repo, lots of
interesting info there.
On Fri, Aug 9, 2019 at 1:02 PM Elliotte Rusty Harold <el...@ibiblio.org>
wrote:
> Another useful thing to improve Java 11 support is to add
> Automatic-Module-Name headers to the various JARs Beam publishes:
>
>
> https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-20.md
>
> If a JAR doesn't include this, Java synthezizes one from the name of the
> jar file, and things get wonky fast. This is a low risk change that has no
> effect on non-modular and pre-Java-9 apps.
>
>
> On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia <mi...@polidea.com>
> wrote:
>
>> Hi everyone,
>>
>> I want to gather the collective knowledge here about Java 11
>> compatibility and ask about the tests needed to deem Beam compatible with
>> JDK 11.
>>
>> Right now concerning testing JDK 11 compatibility I implemented:
>>
>> - Jenkins jobs for running ValidatesRunner test sets in both Direct
>> and Dataflow runners, [1], [2]
>> - ValidatesRunner portability API tests for Dataflow [3],
>> - examples in normal and portable mode for the Dataflow runner [4],
>> [5].
>>
>>
>> Are these tests sufficient to say that we’re java 11 compatible? What
>> other aspects do we need to test to be able to say that?
>>
>>
>> Regards,
>>
>>
>> Michał
>>
>> [1]
>> https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Direct/
>> [2]
>> https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/
>> [3]
>> https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow/
>> [4]
>> https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow/
>> [5]
>> https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow_Portability/
>>
>> --
>>
>> Michał Walenia
>> Polidea <https://www.polidea.com/> | Software Engineer
>>
>> M: +48 791 432 002 <+48791432002>
>> E: michal.walenia@polidea.com
>>
>> Unique Tech
>> Check out our projects! <https://www.polidea.com/our-work>
>>
>
>
> --
> Elliotte Rusty Harold
> elharo@ibiblio.org
>
Re: Java 11 compatibility question
Posted by Elliotte Rusty Harold <el...@ibiblio.org>.
Another useful thing to improve Java 11 support is to add
Automatic-Module-Name headers to the various JARs Beam publishes:
https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-20.md
If a JAR doesn't include this, Java synthezizes one from the name of the
jar file, and things get wonky fast. This is a low risk change that has no
effect on non-modular and pre-Java-9 apps.
On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia <mi...@polidea.com>
wrote:
> Hi everyone,
>
> I want to gather the collective knowledge here about Java 11 compatibility
> and ask about the tests needed to deem Beam compatible with JDK 11.
>
> Right now concerning testing JDK 11 compatibility I implemented:
>
> - Jenkins jobs for running ValidatesRunner test sets in both Direct
> and Dataflow runners, [1], [2]
> - ValidatesRunner portability API tests for Dataflow [3],
> - examples in normal and portable mode for the Dataflow runner [4],
> [5].
>
>
> Are these tests sufficient to say that we’re java 11 compatible? What
> other aspects do we need to test to be able to say that?
>
>
> Regards,
>
>
> Michał
>
> [1]
> https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Direct/
> [2]
> https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/
> [3]
> https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow/
> [4]
> https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow/
> [5]
> https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow_Portability/
>
> --
>
> Michał Walenia
> Polidea <https://www.polidea.com/> | Software Engineer
>
> M: +48 791 432 002 <+48791432002>
> E: michal.walenia@polidea.com
>
> Unique Tech
> Check out our projects! <https://www.polidea.com/our-work>
>
--
Elliotte Rusty Harold
elharo@ibiblio.org
Re: Java 11 compatibility question
Posted by Robert Bradshaw <ro...@google.com>.
On Fri, Aug 9, 2019 at 12:48 PM Michał Walenia <mi...@polidea.com>
wrote:
> From what I understand, the Java 8 -> 11 testing isn't in essence similar
> to py2 -> py3 checks.
>
True. Python 3 is in many ways a new language, and much less (and more
subtly) backwards compatible. You also can't "link" Python 3 code against
Python 2 code the way you can use old Java classes in new JVMs.
> In the case of Java, all we want to do is check if Beam downloaded by
> users from Maven (and compiled with JDK8) won't act up if used from a
> JDK/JRE 11 environment. We don't want to migrate the tool itself to a newer
> language version. As I mentioned in my previous email, there already are
> test suites checking compatibility - ValidatesRunner on Direct and Dataflow
> runners running in normal and portable mode.
> Those tests keep passing, so I believe we're mostly fine regarding
> compatibility.
> All I want to know is - is this enough?
> How else can we test Beam to be sure it works in JRE 11? After several
> accidental launches of build tasks in JDK 11, I am sure that it's not
> buildable with it, but this is not the compatibility type we want to check.
>
Well, we will want this eventually. Before that, we'll want to be sure
users can build their Java 11 code against our artifacts.
>
> Thank you for your replies,
> Michal
>
>
> On Thu, Aug 8, 2019 at 10:25 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> From Python 3 migration standpoint, some high level pillars that increase
>> our confidence are:
>> - Test coverage: (PreCommit, PostCommit), creating a system to make it
>> easy for add test coverage in new language for new functionality.
>> - Support of new language version by core runners + ValidatesRunner test
>> coverage.
>> - Test of time: offer new functionality in a few releases, monitor &
>> address user feedback.
>>
>> Dependency audit and critical feature support in new language, as
>> mentioned by others, are important points. If you are curious about
>> detailed AIs that went into Python 3 support, feel free to look into
>> BEAM-1251 or Py3 Kanban Board (
>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail
>> ).
>>
>> Thanks,
>> Valentyn
>>
>>
>> On Thu, Aug 8, 2019 at 7:24 PM Mark Liu <ma...@google.com> wrote:
>>
>>> Some actions we did for py2 to py3 works:
>>> - Check and resolve incompatible dependencies.
>>> - Enable py3 lint.
>>> - Fill feature gaps between py2 and py3 (e.g. new py3 container, new
>>> solution for type hint)
>>> - Add unit tests, integration tests and other tests on py3 for coverage.
>>> - Release (p3) and deprecation (p2) plan.
>>>
>>> Hope this helps on Java upgrade.
>>>
>>> Mark
>>>
>>> On Wed, Aug 7, 2019 at 3:19 PM Ahmet Altay <al...@google.com> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Aug 7, 2019 at 12:21 PM Elliotte Rusty Harold <
>>>> elharo@ibiblio.org> wrote:
>>>>
>>>>> gRPC bug here: https://github.com/grpc/grpc-java/issues/3522
>>>>>
>>>>> google-cloud-java bug:
>>>>> https://github.com/googleapis/google-cloud-java/issues/5760
>>>>>
>>>>> Neither has a cheap or easy fix, I'm afraid. Commenting on these
>>>>> issues might help us prove that there's a demand to priorotize these
>>>>> compared to other work. If anyone has a support contract and could
>>>>> file a ticket asking for a fix, that would help even more.
>>>>>
>>>>> Those are the two I know about. There might be others elsewhere in the
>>>>> dependency tree.
>>>>>
>>>>>
>>>>> On Wed, Aug 7, 2019 at 2:25 PM Lukasz Cwik <lc...@google.com> wrote:
>>>>> >
>>>>> > Since java8 -> java11 is similar to python2 -> python3 migration,
>>>>> what was the acceptance criteria there?
>>>>>
>>>>
>>>> I do not remember formally discussing this. The bar used was, all
>>>> existing tests will pass for python2 and python3. New tests will be added
>>>> for python3 specific features. (To avoid any confusion this bar has not
>>>> been cleared yet.)
>>>>
>>>> cc: +Valentyn Tymofieiev <va...@google.com> could add more details.
>>>>
>>>>
>>>>> >
>>>>> > On Wed, Aug 7, 2019 at 1:54 PM Elliotte Rusty Harold <
>>>>> elharo@ibiblio.org> wrote:
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia <
>>>>> michal.walenia@polidea.com> wrote:
>>>>> >>>
>>>>> >>>
>>>>> >>> Are these tests sufficient to say that we’re java 11 compatible?
>>>>> What other aspects do we need to test to be able to say that?
>>>>> >>>
>>>>> >>>
>>>>> >>
>>>>> >> Are any packages split across multiple jar files, including
>>>>> packages beam dependns on? That's the one that's bitten some other
>>>>> projects, including google-cloud-java and gRPC. If so, beam is not going to
>>>>> work with the module system.
>>>>> >>
>>>>> >> Work is ongoing to fix splitn packages in both gRPC and
>>>>> google-cloud-java, but we're not very far down that path and I think it's
>>>>> going to be an API breaking change.
>>>>> >>
>>>>> > Romain pointed this out earlier and I fixed the last case of
>>>>> packages being split across multiple jars within Apache Beam but as you
>>>>> point out our transitive dependencies are not ready.
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Elliotte Rusty Harold
>>>>> >> elharo@ibiblio.org
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Elliotte Rusty Harold
>>>>> elharo@ibiblio.org
>>>>>
>>>>
>
> --
>
> Michał Walenia
> Polidea <https://www.polidea.com/> | Software Engineer
>
> M: +48 791 432 002 <+48791432002>
> E: michal.walenia@polidea.com
>
> Unique Tech
> Check out our projects! <https://www.polidea.com/our-work>
>
Re: Java 11 compatibility question
Posted by Michał Walenia <mi...@polidea.com>.
From what I understand, the Java 8 -> 11 testing isn't in essence similar
to py2 -> py3 checks.
In the case of Java, all we want to do is check if Beam downloaded by users
from Maven (and compiled with JDK8) won't act up if used from a JDK/JRE 11
environment. We don't want to migrate the tool itself to a newer language
version. As I mentioned in my previous email, there already are test suites
checking compatibility - ValidatesRunner on Direct and Dataflow runners
running in normal and portable mode.
Those tests keep passing, so I believe we're mostly fine regarding
compatibility.
All I want to know is - is this enough?
How else can we test Beam to be sure it works in JRE 11? After several
accidental launches of build tasks in JDK 11, I am sure that it's not
buildable with it, but this is not the compatibility type we want to check.
Thank you for your replies,
Michal
On Thu, Aug 8, 2019 at 10:25 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> From Python 3 migration standpoint, some high level pillars that increase
> our confidence are:
> - Test coverage: (PreCommit, PostCommit), creating a system to make it
> easy for add test coverage in new language for new functionality.
> - Support of new language version by core runners + ValidatesRunner test
> coverage.
> - Test of time: offer new functionality in a few releases, monitor &
> address user feedback.
>
> Dependency audit and critical feature support in new language, as
> mentioned by others, are important points. If you are curious about
> detailed AIs that went into Python 3 support, feel free to look into
> BEAM-1251 or Py3 Kanban Board (
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail
> ).
>
> Thanks,
> Valentyn
>
>
> On Thu, Aug 8, 2019 at 7:24 PM Mark Liu <ma...@google.com> wrote:
>
>> Some actions we did for py2 to py3 works:
>> - Check and resolve incompatible dependencies.
>> - Enable py3 lint.
>> - Fill feature gaps between py2 and py3 (e.g. new py3 container, new
>> solution for type hint)
>> - Add unit tests, integration tests and other tests on py3 for coverage.
>> - Release (p3) and deprecation (p2) plan.
>>
>> Hope this helps on Java upgrade.
>>
>> Mark
>>
>> On Wed, Aug 7, 2019 at 3:19 PM Ahmet Altay <al...@google.com> wrote:
>>
>>>
>>>
>>> On Wed, Aug 7, 2019 at 12:21 PM Elliotte Rusty Harold <
>>> elharo@ibiblio.org> wrote:
>>>
>>>> gRPC bug here: https://github.com/grpc/grpc-java/issues/3522
>>>>
>>>> google-cloud-java bug:
>>>> https://github.com/googleapis/google-cloud-java/issues/5760
>>>>
>>>> Neither has a cheap or easy fix, I'm afraid. Commenting on these
>>>> issues might help us prove that there's a demand to priorotize these
>>>> compared to other work. If anyone has a support contract and could
>>>> file a ticket asking for a fix, that would help even more.
>>>>
>>>> Those are the two I know about. There might be others elsewhere in the
>>>> dependency tree.
>>>>
>>>>
>>>> On Wed, Aug 7, 2019 at 2:25 PM Lukasz Cwik <lc...@google.com> wrote:
>>>> >
>>>> > Since java8 -> java11 is similar to python2 -> python3 migration,
>>>> what was the acceptance criteria there?
>>>>
>>>
>>> I do not remember formally discussing this. The bar used was, all
>>> existing tests will pass for python2 and python3. New tests will be added
>>> for python3 specific features. (To avoid any confusion this bar has not
>>> been cleared yet.)
>>>
>>> cc: +Valentyn Tymofieiev <va...@google.com> could add more details.
>>>
>>>
>>>> >
>>>> > On Wed, Aug 7, 2019 at 1:54 PM Elliotte Rusty Harold <
>>>> elharo@ibiblio.org> wrote:
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia <
>>>> michal.walenia@polidea.com> wrote:
>>>> >>>
>>>> >>>
>>>> >>> Are these tests sufficient to say that we’re java 11 compatible?
>>>> What other aspects do we need to test to be able to say that?
>>>> >>>
>>>> >>>
>>>> >>
>>>> >> Are any packages split across multiple jar files, including packages
>>>> beam dependns on? That's the one that's bitten some other projects,
>>>> including google-cloud-java and gRPC. If so, beam is not going to work with
>>>> the module system.
>>>> >>
>>>> >> Work is ongoing to fix splitn packages in both gRPC and
>>>> google-cloud-java, but we're not very far down that path and I think it's
>>>> going to be an API breaking change.
>>>> >>
>>>> > Romain pointed this out earlier and I fixed the last case of packages
>>>> being split across multiple jars within Apache Beam but as you point out
>>>> our transitive dependencies are not ready.
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Elliotte Rusty Harold
>>>> >> elharo@ibiblio.org
>>>>
>>>>
>>>>
>>>> --
>>>> Elliotte Rusty Harold
>>>> elharo@ibiblio.org
>>>>
>>>
--
Michał Walenia
Polidea <https://www.polidea.com/> | Software Engineer
M: +48 791 432 002 <+48791432002>
E: michal.walenia@polidea.com
Unique Tech
Check out our projects! <https://www.polidea.com/our-work>
Re: Java 11 compatibility question
Posted by Valentyn Tymofieiev <va...@google.com>.
From Python 3 migration standpoint, some high level pillars that increase
our confidence are:
- Test coverage: (PreCommit, PostCommit), creating a system to make it easy
for add test coverage in new language for new functionality.
- Support of new language version by core runners + ValidatesRunner test
coverage.
- Test of time: offer new functionality in a few releases, monitor &
address user feedback.
Dependency audit and critical feature support in new language, as mentioned
by others, are important points. If you are curious about detailed AIs
that went into Python 3 support, feel free to look into BEAM-1251 or Py3
Kanban Board (
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail
).
Thanks,
Valentyn
On Thu, Aug 8, 2019 at 7:24 PM Mark Liu <ma...@google.com> wrote:
> Some actions we did for py2 to py3 works:
> - Check and resolve incompatible dependencies.
> - Enable py3 lint.
> - Fill feature gaps between py2 and py3 (e.g. new py3 container, new
> solution for type hint)
> - Add unit tests, integration tests and other tests on py3 for coverage.
> - Release (p3) and deprecation (p2) plan.
>
> Hope this helps on Java upgrade.
>
> Mark
>
> On Wed, Aug 7, 2019 at 3:19 PM Ahmet Altay <al...@google.com> wrote:
>
>>
>>
>> On Wed, Aug 7, 2019 at 12:21 PM Elliotte Rusty Harold <el...@ibiblio.org>
>> wrote:
>>
>>> gRPC bug here: https://github.com/grpc/grpc-java/issues/3522
>>>
>>> google-cloud-java bug:
>>> https://github.com/googleapis/google-cloud-java/issues/5760
>>>
>>> Neither has a cheap or easy fix, I'm afraid. Commenting on these
>>> issues might help us prove that there's a demand to priorotize these
>>> compared to other work. If anyone has a support contract and could
>>> file a ticket asking for a fix, that would help even more.
>>>
>>> Those are the two I know about. There might be others elsewhere in the
>>> dependency tree.
>>>
>>>
>>> On Wed, Aug 7, 2019 at 2:25 PM Lukasz Cwik <lc...@google.com> wrote:
>>> >
>>> > Since java8 -> java11 is similar to python2 -> python3 migration, what
>>> was the acceptance criteria there?
>>>
>>
>> I do not remember formally discussing this. The bar used was, all
>> existing tests will pass for python2 and python3. New tests will be added
>> for python3 specific features. (To avoid any confusion this bar has not
>> been cleared yet.)
>>
>> cc: +Valentyn Tymofieiev <va...@google.com> could add more details.
>>
>>
>>> >
>>> > On Wed, Aug 7, 2019 at 1:54 PM Elliotte Rusty Harold <
>>> elharo@ibiblio.org> wrote:
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia <
>>> michal.walenia@polidea.com> wrote:
>>> >>>
>>> >>>
>>> >>> Are these tests sufficient to say that we’re java 11 compatible?
>>> What other aspects do we need to test to be able to say that?
>>> >>>
>>> >>>
>>> >>
>>> >> Are any packages split across multiple jar files, including packages
>>> beam dependns on? That's the one that's bitten some other projects,
>>> including google-cloud-java and gRPC. If so, beam is not going to work with
>>> the module system.
>>> >>
>>> >> Work is ongoing to fix splitn packages in both gRPC and
>>> google-cloud-java, but we're not very far down that path and I think it's
>>> going to be an API breaking change.
>>> >>
>>> > Romain pointed this out earlier and I fixed the last case of packages
>>> being split across multiple jars within Apache Beam but as you point out
>>> our transitive dependencies are not ready.
>>> >>
>>> >>
>>> >> --
>>> >> Elliotte Rusty Harold
>>> >> elharo@ibiblio.org
>>>
>>>
>>>
>>> --
>>> Elliotte Rusty Harold
>>> elharo@ibiblio.org
>>>
>>
Re: Java 11 compatibility question
Posted by Mark Liu <ma...@google.com>.
Some actions we did for py2 to py3 works:
- Check and resolve incompatible dependencies.
- Enable py3 lint.
- Fill feature gaps between py2 and py3 (e.g. new py3 container, new
solution for type hint)
- Add unit tests, integration tests and other tests on py3 for coverage.
- Release (p3) and deprecation (p2) plan.
Hope this helps on Java upgrade.
Mark
On Wed, Aug 7, 2019 at 3:19 PM Ahmet Altay <al...@google.com> wrote:
>
>
> On Wed, Aug 7, 2019 at 12:21 PM Elliotte Rusty Harold <el...@ibiblio.org>
> wrote:
>
>> gRPC bug here: https://github.com/grpc/grpc-java/issues/3522
>>
>> google-cloud-java bug:
>> https://github.com/googleapis/google-cloud-java/issues/5760
>>
>> Neither has a cheap or easy fix, I'm afraid. Commenting on these
>> issues might help us prove that there's a demand to priorotize these
>> compared to other work. If anyone has a support contract and could
>> file a ticket asking for a fix, that would help even more.
>>
>> Those are the two I know about. There might be others elsewhere in the
>> dependency tree.
>>
>>
>> On Wed, Aug 7, 2019 at 2:25 PM Lukasz Cwik <lc...@google.com> wrote:
>> >
>> > Since java8 -> java11 is similar to python2 -> python3 migration, what
>> was the acceptance criteria there?
>>
>
> I do not remember formally discussing this. The bar used was, all existing
> tests will pass for python2 and python3. New tests will be added for
> python3 specific features. (To avoid any confusion this bar has not been
> cleared yet.)
>
> cc: +Valentyn Tymofieiev <va...@google.com> could add more details.
>
>
>> >
>> > On Wed, Aug 7, 2019 at 1:54 PM Elliotte Rusty Harold <
>> elharo@ibiblio.org> wrote:
>> >>
>> >>
>> >>
>> >> On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia <
>> michal.walenia@polidea.com> wrote:
>> >>>
>> >>>
>> >>> Are these tests sufficient to say that we’re java 11 compatible? What
>> other aspects do we need to test to be able to say that?
>> >>>
>> >>>
>> >>
>> >> Are any packages split across multiple jar files, including packages
>> beam dependns on? That's the one that's bitten some other projects,
>> including google-cloud-java and gRPC. If so, beam is not going to work with
>> the module system.
>> >>
>> >> Work is ongoing to fix splitn packages in both gRPC and
>> google-cloud-java, but we're not very far down that path and I think it's
>> going to be an API breaking change.
>> >>
>> > Romain pointed this out earlier and I fixed the last case of packages
>> being split across multiple jars within Apache Beam but as you point out
>> our transitive dependencies are not ready.
>> >>
>> >>
>> >> --
>> >> Elliotte Rusty Harold
>> >> elharo@ibiblio.org
>>
>>
>>
>> --
>> Elliotte Rusty Harold
>> elharo@ibiblio.org
>>
>
Re: Java 11 compatibility question
Posted by Ahmet Altay <al...@google.com>.
On Wed, Aug 7, 2019 at 12:21 PM Elliotte Rusty Harold <el...@ibiblio.org>
wrote:
> gRPC bug here: https://github.com/grpc/grpc-java/issues/3522
>
> google-cloud-java bug:
> https://github.com/googleapis/google-cloud-java/issues/5760
>
> Neither has a cheap or easy fix, I'm afraid. Commenting on these
> issues might help us prove that there's a demand to priorotize these
> compared to other work. If anyone has a support contract and could
> file a ticket asking for a fix, that would help even more.
>
> Those are the two I know about. There might be others elsewhere in the
> dependency tree.
>
>
> On Wed, Aug 7, 2019 at 2:25 PM Lukasz Cwik <lc...@google.com> wrote:
> >
> > Since java8 -> java11 is similar to python2 -> python3 migration, what
> was the acceptance criteria there?
>
I do not remember formally discussing this. The bar used was, all existing
tests will pass for python2 and python3. New tests will be added for
python3 specific features. (To avoid any confusion this bar has not been
cleared yet.)
cc: +Valentyn Tymofieiev <va...@google.com> could add more details.
> >
> > On Wed, Aug 7, 2019 at 1:54 PM Elliotte Rusty Harold <el...@ibiblio.org>
> wrote:
> >>
> >>
> >>
> >> On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia <
> michal.walenia@polidea.com> wrote:
> >>>
> >>>
> >>> Are these tests sufficient to say that we’re java 11 compatible? What
> other aspects do we need to test to be able to say that?
> >>>
> >>>
> >>
> >> Are any packages split across multiple jar files, including packages
> beam dependns on? That's the one that's bitten some other projects,
> including google-cloud-java and gRPC. If so, beam is not going to work with
> the module system.
> >>
> >> Work is ongoing to fix splitn packages in both gRPC and
> google-cloud-java, but we're not very far down that path and I think it's
> going to be an API breaking change.
> >>
> > Romain pointed this out earlier and I fixed the last case of packages
> being split across multiple jars within Apache Beam but as you point out
> our transitive dependencies are not ready.
> >>
> >>
> >> --
> >> Elliotte Rusty Harold
> >> elharo@ibiblio.org
>
>
>
> --
> Elliotte Rusty Harold
> elharo@ibiblio.org
>
Re: Java 11 compatibility question
Posted by Elliotte Rusty Harold <el...@ibiblio.org>.
gRPC bug here: https://github.com/grpc/grpc-java/issues/3522
google-cloud-java bug:
https://github.com/googleapis/google-cloud-java/issues/5760
Neither has a cheap or easy fix, I'm afraid. Commenting on these
issues might help us prove that there's a demand to priorotize these
compared to other work. If anyone has a support contract and could
file a ticket asking for a fix, that would help even more.
Those are the two I know about. There might be others elsewhere in the
dependency tree.
On Wed, Aug 7, 2019 at 2:25 PM Lukasz Cwik <lc...@google.com> wrote:
>
> Since java8 -> java11 is similar to python2 -> python3 migration, what was the acceptance criteria there?
>
> On Wed, Aug 7, 2019 at 1:54 PM Elliotte Rusty Harold <el...@ibiblio.org> wrote:
>>
>>
>>
>> On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia <mi...@polidea.com> wrote:
>>>
>>>
>>> Are these tests sufficient to say that we’re java 11 compatible? What other aspects do we need to test to be able to say that?
>>>
>>>
>>
>> Are any packages split across multiple jar files, including packages beam dependns on? That's the one that's bitten some other projects, including google-cloud-java and gRPC. If so, beam is not going to work with the module system.
>>
>> Work is ongoing to fix splitn packages in both gRPC and google-cloud-java, but we're not very far down that path and I think it's going to be an API breaking change.
>>
> Romain pointed this out earlier and I fixed the last case of packages being split across multiple jars within Apache Beam but as you point out our transitive dependencies are not ready.
>>
>>
>> --
>> Elliotte Rusty Harold
>> elharo@ibiblio.org
--
Elliotte Rusty Harold
elharo@ibiblio.org
Re: Java 11 compatibility question
Posted by Lukasz Cwik <lc...@google.com>.
Since java8 -> java11 is similar to python2 -> python3 migration, what was
the acceptance criteria there?
On Wed, Aug 7, 2019 at 1:54 PM Elliotte Rusty Harold <el...@ibiblio.org>
wrote:
>
>
> On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia <mi...@polidea.com>
> wrote:
>
>>
>> Are these tests sufficient to say that we’re java 11 compatible? What
>> other aspects do we need to test to be able to say that?
>>
>>
> Are any packages split across multiple jar files, including packages beam
> dependns on? That's the one that's bitten some other projects, including
> google-cloud-java and gRPC. If so, beam is not going to work with the
> module system.
>
> Work is ongoing to fix splitn packages in both gRPC and google-cloud-java,
> but we're not very far down that path and I think it's going to be an API
> breaking change.
>
> Romain pointed this out earlier and I fixed the last case of packages
being split across multiple jars within Apache Beam but as you point out
our transitive dependencies are not ready.
>
> --
> Elliotte Rusty Harold
> elharo@ibiblio.org
>
Re: Java 11 compatibility question
Posted by Elliotte Rusty Harold <el...@ibiblio.org>.
On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia <mi...@polidea.com>
wrote:
>
> Are these tests sufficient to say that we’re java 11 compatible? What
> other aspects do we need to test to be able to say that?
>
>
Are any packages split across multiple jar files, including packages beam
dependns on? That's the one that's bitten some other projects, including
google-cloud-java and gRPC. If so, beam is not going to work with the
module system.
Work is ongoing to fix splitn packages in both gRPC and google-cloud-java,
but we're not very far down that path and I think it's going to be an API
breaking change.
--
Elliotte Rusty Harold
elharo@ibiblio.org