You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by James DeMichele <ja...@redfin.com.INVALID> on 2022/09/02 15:55:39 UTC
Java 11 Checkin again
Hey y'all. I just am not sure how to get some traction on these Java 11
PRs.
https://github.com/apache/samza/pull/1628
https://github.com/apache/samza-hello-samza/pull/87
Would someone that is a maintainer for Samza just let us know that y'all
are looking at them? I can stop pestering you :)
I ran all tests in both PRs, all pass. I also confirmed that using my Samza
PR in the Hello World app all works with Java 11.
Thanks!
-Jamie
Re: Java 11 Checkin again
Posted by Yi Pan <ni...@gmail.com>.
Hi, James,
Sorry that I was busy during the day and couldn't check your email. I have
joined the slack channel you created. Let's discuss there.
Best,
-Yi
On Mon, Sep 19, 2022 at 9:23 AM James DeMichele
<ja...@redfin.com.invalid> wrote:
> Hey, Yi! Thanks.
>
> I started a slack channel that maybe would make it easier to communicate if
> I have questions. I do have one issue that I am hitting between the 2
> different Yarn versions I think and that I am not entirely sure what to do
> about. I made a change to a Test class that was needed for a
> compilation fix:
>
> https://github.com/apache/samza/pull/1628/files#diff-34db8b18730bda1058014e87ec3ad88dfc03f79854b00a339407761d224f66e9
> and the class is TestSamzaYarnAppMasterLifecycle.scala.
>
> I'm not sure how to go about toggling this class between a compatible one
> for yarn 2.10.1 and 3.3.4.
>
> Thanks!
>
> -Jamie
>
> On Mon, Sep 19, 2022 at 11:00 AM Yi Pan <ni...@gmail.com> wrote:
>
> > Hi, James,
> >
> > Thanks a lot for reporting this. I will take a look this week.
> >
> > Best!
> >
> > -Yi
> >
> > On Mon, Sep 19, 2022 at 8:20 AM James DeMichele
> > <ja...@redfin.com.invalid> wrote:
> >
> > > Hey Yi. I can take a look at this. I do want to point out that your
> > current
> > > "master" branch is actually broken for using Scala 2.11.
> > >
> > > You can repro by just going into the master branch using Java 8 and
> > > compiling like this
> > >
> > > $ java -version
> > > openjdk version "1.8.0_332"
> > > OpenJDK Runtime Environment (Temurin)(build 1.8.0_332-b09)
> > > OpenJDK 64-Bit Server VM (Temurin)(build 25.332-b09, mixed mode)
> > >
> > > ./gradlew build -PscalaSuffix=2.11
> > >
> > > The build fails with this command using that version of Java 8 ^.
> > >
> > > Anyway, just wanted to point that out since I hit this in my branch
> > > trying to utilize the "bin/check-all.sh" script. That doesn't block
> > > me/us, but just wanted to call it out.
> > >
> > > -Jamie
> > >
> > >
> > >
> > > On Wed, Sep 14, 2022 at 5:52 PM Yi Pan <ni...@gmail.com> wrote:
> > >
> > > > Hey, James,
> > > >
> > > > In order to merge your PR without breaking the jdk8 older modules, we
> > > will
> > > > need the changes proposed here. Can you try to add those build script
> > > > changes in the same PR? We will definitely help review and merge it.
> > > >
> > > > Best!
> > > >
> > > > -Yi
> > > >
> > > > On Wed, Sep 14, 2022 at 8:00 AM James DeMichele
> > > > <ja...@redfin.com.invalid> wrote:
> > > >
> > > > > Also, do you have a timeline for when this could be completed?
> > Thanks.
> > > > >
> > > > > On Wed, Sep 14, 2022 at 7:16 AM James DeMichele <
> > > > > james.demichele@redfin.com>
> > > > > wrote:
> > > > >
> > > > > > That sounds like a great solution to me if that works for y'all!
> > > > > >
> > > > > > Note too, the Java 11 and yarn 3 module need to only use the
> Scala
> > > 2.12
> > > > > > version of the build.
> > > > > >
> > > > > > Jamie
> > > > > >
> > > > > >
> > > > > > On Wed, Sep 14, 2022, 2:38 AM Yi Pan <ni...@gmail.com>
> wrote:
> > > > > >
> > > > > >> Hi, James,
> > > > > >>
> > > > > >> Sorry to reply late. I just came back from a trip and had a
> > > discussion
> > > > > >> with
> > > > > >> our internal team as well. So, there is one proposal other than
> > > > > creating a
> > > > > >> branch. Let me elaborate it below:
> > > > > >> a) creating a new module samza-yarn3 that depends on YARN 3.3.0
> > and
> > > be
> > > > > the
> > > > > >> hosting module for most of the jdk11 related changes.
> > > > > >> b) modify the build script s.t. samza-yarn3 will only compile
> and
> > > > build
> > > > > >> with jdk11 and samza-yarn only compile and build with jdk8.
> > > > > >> Thus, we can have two builds: jdk8 build that builds with
> > samza-yarn
> > > > w/
> > > > > >> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/
> YARN
> > > > 3.3.0.
> > > > > >> We
> > > > > >> can manage to publish both jdk8 and jdk11 artifacts if needed.
> > > > > >> The benefit of this approach is that we can still maintain the
> > trunk
> > > > > >> release while opening up the jdk11 support.
> > > > > >>
> > > > > >> Let me know if that works for you and we can work together to
> get
> > > the
> > > > > code
> > > > > >> in.
> > > > > >>
> > > > > >> Best!
> > > > > >>
> > > > > >> -Yi
> > > > > >>
> > > > > >> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele
> > > > > >> <ja...@redfin.com.invalid> wrote:
> > > > > >>
> > > > > >> > Hey Yi, I wanted to follow up here and figure what a path
> > forward
> > > is
> > > > > >> here.
> > > > > >> > We need to move to Java 11, and Samza currently is our only
> > > blocking
> > > > > >> issue.
> > > > > >> > In order to move to Java 11, the Yarn Cluster would need to
> run
> > on
> > > > > Java
> > > > > >> 11
> > > > > >> > correct? If that's the case, then it would need to be 3.3+. I
> > > don't
> > > > > know
> > > > > >> > what it entails on your end to have a new Major version, but
> > that
> > > > > seems
> > > > > >> > like a good option here right? Version 2 could be where we can
> > > move
> > > > > this
> > > > > >> > project forward to Java 11, while Version 1 can still remain,
> > and
> > > > > would
> > > > > >> not
> > > > > >> > break people that can't/won't upgrade to Java 11.
> > > > > >> >
> > > > > >> > -Jamie
> > > > > >> >
> > > > > >> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <
> > > > > >> james.demichele@redfin.com
> > > > > >> > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Yeah I mean if Samza works fine with the hadoop-yarn library
> > > > running
> > > > > >> > > against a 3.3.x YARN cluster, then I don't mind keeping that
> > > > library
> > > > > >> of
> > > > > >> > > 2.10.x in Samza's code. But it is still a moot point in
> terms
> > of
> > > > > >> > upgrading
> > > > > >> > > your YARN cluster, since it must be upgraded to 3.3.x+ in
> > order
> > > to
> > > > > be
> > > > > >> > able
> > > > > >> > > to run the Cluster with Java 11.
> > > > > >> > >
> > > > > >> > > @Yi, I think that moving to a new major version might be the
> > > > > solution
> > > > > >> > > here. That way Linkedin can still have a pathway of
> upgrading
> > > code
> > > > > for
> > > > > >> > the
> > > > > >> > > old legacy 1.x version of Samza. While a new major version
> of
> > > 2.x
> > > > of
> > > > > >> > Samza
> > > > > >> > > could then make it a requirement that it runs with a YARN
> > > cluster
> > > > of
> > > > > >> > 3.3.x
> > > > > >> > > if you want to use Java 11.
> > > > > >> > >
> > > > > >> > > The only issue there is that you'll probably need to
> backport
> > > > > changes
> > > > > >> > > between the 2 versions. But in all honestly, this project
> does
> > > not
> > > > > >> look
> > > > > >> > > extremely active with commits so it might not be that big
> of a
> > > > > >> problem.
> > > > > >> > >
> > > > > >> > > -Jamie
> > > > > >> > >
> > > > > >> > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <
> > > > > >> mmcfarland@cavulus.com
> > > > > >> > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > >> Hi all,
> > > > > >> > >>
> > > > > >> > >> I've been doing a little bit of testing with Samza and
> Hadoop
> > > > > 3.3.4;
> > > > > >> > >> afaict, in light testing, Samza seems to work fine using
> the
> > > > 2.10.x
> > > > > >> > >> hadoop-yarn library against a YARN cluster running 3.3.x.
> As
> > > > Jamie
> > > > > >> > pointed
> > > > > >> > >> out, YARN didn't incorporate Java 11 compatibility until
> > > v3.3.0 (
> > > > > >> > >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are
> there
> > > any
> > > > > >> unit
> > > > > >> > >> tests
> > > > > >> > >> in Samza that verify compatibility against a YARN cluster?
> If
> > > so,
> > > > > >> that
> > > > > >> > >> could be a place to validate YARN v2.10/v3.3
> > > cross-compatibility.
> > > > > >> > >>
> > > > > >> > >> Just throwing my 2 cents out there,
> > > > > >> > >> Malcolm McFarland
> > > > > >> > >> Cavulus
> > > > > >> > >>
> > > > > >> > >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
> > > > > >> > >> <ja...@redfin.com.invalid> wrote:
> > > > > >> > >>
> > > > > >> > >> > Hey Yi,
> > > > > >> > >> >
> > > > > >> > >> > Thanks for getting back to me. I have not tried the older
> > > yarn
> > > > > >> cluster
> > > > > >> > >> > version yet in the Samza app running against 3.3.4 but I
> am
> > > > wary
> > > > > it
> > > > > >> > >> would
> > > > > >> > >> > work. Yarn itself is not compatible at 2.10.1 with Java
> 11
> > so
> > > > you
> > > > > >> > would
> > > > > >> > >> > have to update yarn even if the Java library here wasn't
> > > > updated.
> > > > > >> > >> >
> > > > > >> > >> > Could we move this version I'm proposing to a 2.x version
> > of
> > > > > >> Samza? So
> > > > > >> > >> > people that wanted to move forward with yarn upgrade and
> > > Samza
> > > > > and
> > > > > >> > Java
> > > > > >> > >> 11
> > > > > >> > >> > (like us) could do so? Then 1.x could only be java 8
> > > compatible
> > > > > and
> > > > > >> > 2.x
> > > > > >> > >> > could be java 11.
> > > > > >> > >> >
> > > > > >> > >> > Jamie
> > > > > >> > >> >
> > > > > >> > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <nickpan47@gmail.com
> >
> > > > wrote:
> > > > > >> > >> >
> > > > > >> > >> > > Hey, James,
> > > > > >> > >> > >
> > > > > >> > >> > > Thanks for the ping. @prateek, can we have someone to
> > > review
> > > > > this
> > > > > >> > >> change?
> > > > > >> > >> > >
> > > > > >> > >> > > One question: have you tested the change w/ the older
> > YARN
> > > > > >> cluster
> > > > > >> > >> > version
> > > > > >> > >> > > (running 2.10.1)? If this change requires YARN cluster
> > > > upgrade
> > > > > to
> > > > > >> > >> 3.3.4
> > > > > >> > >> > as
> > > > > >> > >> > > well, that may be a breaking change to existing Samza
> > users
> > > > > (i.e.
> > > > > >> > >> > LinkedIn
> > > > > >> > >> > > is still running a YARN cluster with version 2.10.1).
> > > > > >> > >> > >
> > > > > >> > >> > > Best and apologize for the delay.
> > > > > >> > >> > >
> > > > > >> > >> > > -Yi
> > > > > >> > >> > >
> > > > > >> > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
> > > > > >> > >> > > <ja...@redfin.com.invalid> wrote:
> > > > > >> > >> > >
> > > > > >> > >> > > > Hey y'all. I just am not sure how to get some
> traction
> > on
> > > > > these
> > > > > >> > >> Java 11
> > > > > >> > >> > > > PRs.
> > > > > >> > >> > > >
> > > > > >> > >> > > > https://github.com/apache/samza/pull/1628
> > > > > >> > >> > > > https://github.com/apache/samza-hello-samza/pull/87
> > > > > >> > >> > > >
> > > > > >> > >> > > > Would someone that is a maintainer for Samza just let
> > us
> > > > know
> > > > > >> that
> > > > > >> > >> > y'all
> > > > > >> > >> > > > are looking at them? I can stop pestering you :)
> > > > > >> > >> > > >
> > > > > >> > >> > > > I ran all tests in both PRs, all pass. I also
> confirmed
> > > > that
> > > > > >> using
> > > > > >> > >> my
> > > > > >> > >> > > Samza
> > > > > >> > >> > > > PR in the Hello World app all works with Java 11.
> > > > > >> > >> > > >
> > > > > >> > >> > > > Thanks!
> > > > > >> > >> > > >
> > > > > >> > >> > > > -Jamie
> > > > > >> > >> > > >
> > > > > >> > >> > >
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Java 11 Checkin again
Posted by James DeMichele <ja...@redfin.com.INVALID>.
Hey, Yi! Thanks.
I started a slack channel that maybe would make it easier to communicate if
I have questions. I do have one issue that I am hitting between the 2
different Yarn versions I think and that I am not entirely sure what to do
about. I made a change to a Test class that was needed for a
compilation fix:
https://github.com/apache/samza/pull/1628/files#diff-34db8b18730bda1058014e87ec3ad88dfc03f79854b00a339407761d224f66e9
and the class is TestSamzaYarnAppMasterLifecycle.scala.
I'm not sure how to go about toggling this class between a compatible one
for yarn 2.10.1 and 3.3.4.
Thanks!
-Jamie
On Mon, Sep 19, 2022 at 11:00 AM Yi Pan <ni...@gmail.com> wrote:
> Hi, James,
>
> Thanks a lot for reporting this. I will take a look this week.
>
> Best!
>
> -Yi
>
> On Mon, Sep 19, 2022 at 8:20 AM James DeMichele
> <ja...@redfin.com.invalid> wrote:
>
> > Hey Yi. I can take a look at this. I do want to point out that your
> current
> > "master" branch is actually broken for using Scala 2.11.
> >
> > You can repro by just going into the master branch using Java 8 and
> > compiling like this
> >
> > $ java -version
> > openjdk version "1.8.0_332"
> > OpenJDK Runtime Environment (Temurin)(build 1.8.0_332-b09)
> > OpenJDK 64-Bit Server VM (Temurin)(build 25.332-b09, mixed mode)
> >
> > ./gradlew build -PscalaSuffix=2.11
> >
> > The build fails with this command using that version of Java 8 ^.
> >
> > Anyway, just wanted to point that out since I hit this in my branch
> > trying to utilize the "bin/check-all.sh" script. That doesn't block
> > me/us, but just wanted to call it out.
> >
> > -Jamie
> >
> >
> >
> > On Wed, Sep 14, 2022 at 5:52 PM Yi Pan <ni...@gmail.com> wrote:
> >
> > > Hey, James,
> > >
> > > In order to merge your PR without breaking the jdk8 older modules, we
> > will
> > > need the changes proposed here. Can you try to add those build script
> > > changes in the same PR? We will definitely help review and merge it.
> > >
> > > Best!
> > >
> > > -Yi
> > >
> > > On Wed, Sep 14, 2022 at 8:00 AM James DeMichele
> > > <ja...@redfin.com.invalid> wrote:
> > >
> > > > Also, do you have a timeline for when this could be completed?
> Thanks.
> > > >
> > > > On Wed, Sep 14, 2022 at 7:16 AM James DeMichele <
> > > > james.demichele@redfin.com>
> > > > wrote:
> > > >
> > > > > That sounds like a great solution to me if that works for y'all!
> > > > >
> > > > > Note too, the Java 11 and yarn 3 module need to only use the Scala
> > 2.12
> > > > > version of the build.
> > > > >
> > > > > Jamie
> > > > >
> > > > >
> > > > > On Wed, Sep 14, 2022, 2:38 AM Yi Pan <ni...@gmail.com> wrote:
> > > > >
> > > > >> Hi, James,
> > > > >>
> > > > >> Sorry to reply late. I just came back from a trip and had a
> > discussion
> > > > >> with
> > > > >> our internal team as well. So, there is one proposal other than
> > > > creating a
> > > > >> branch. Let me elaborate it below:
> > > > >> a) creating a new module samza-yarn3 that depends on YARN 3.3.0
> and
> > be
> > > > the
> > > > >> hosting module for most of the jdk11 related changes.
> > > > >> b) modify the build script s.t. samza-yarn3 will only compile and
> > > build
> > > > >> with jdk11 and samza-yarn only compile and build with jdk8.
> > > > >> Thus, we can have two builds: jdk8 build that builds with
> samza-yarn
> > > w/
> > > > >> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN
> > > 3.3.0.
> > > > >> We
> > > > >> can manage to publish both jdk8 and jdk11 artifacts if needed.
> > > > >> The benefit of this approach is that we can still maintain the
> trunk
> > > > >> release while opening up the jdk11 support.
> > > > >>
> > > > >> Let me know if that works for you and we can work together to get
> > the
> > > > code
> > > > >> in.
> > > > >>
> > > > >> Best!
> > > > >>
> > > > >> -Yi
> > > > >>
> > > > >> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele
> > > > >> <ja...@redfin.com.invalid> wrote:
> > > > >>
> > > > >> > Hey Yi, I wanted to follow up here and figure what a path
> forward
> > is
> > > > >> here.
> > > > >> > We need to move to Java 11, and Samza currently is our only
> > blocking
> > > > >> issue.
> > > > >> > In order to move to Java 11, the Yarn Cluster would need to run
> on
> > > > Java
> > > > >> 11
> > > > >> > correct? If that's the case, then it would need to be 3.3+. I
> > don't
> > > > know
> > > > >> > what it entails on your end to have a new Major version, but
> that
> > > > seems
> > > > >> > like a good option here right? Version 2 could be where we can
> > move
> > > > this
> > > > >> > project forward to Java 11, while Version 1 can still remain,
> and
> > > > would
> > > > >> not
> > > > >> > break people that can't/won't upgrade to Java 11.
> > > > >> >
> > > > >> > -Jamie
> > > > >> >
> > > > >> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <
> > > > >> james.demichele@redfin.com
> > > > >> > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Yeah I mean if Samza works fine with the hadoop-yarn library
> > > running
> > > > >> > > against a 3.3.x YARN cluster, then I don't mind keeping that
> > > library
> > > > >> of
> > > > >> > > 2.10.x in Samza's code. But it is still a moot point in terms
> of
> > > > >> > upgrading
> > > > >> > > your YARN cluster, since it must be upgraded to 3.3.x+ in
> order
> > to
> > > > be
> > > > >> > able
> > > > >> > > to run the Cluster with Java 11.
> > > > >> > >
> > > > >> > > @Yi, I think that moving to a new major version might be the
> > > > solution
> > > > >> > > here. That way Linkedin can still have a pathway of upgrading
> > code
> > > > for
> > > > >> > the
> > > > >> > > old legacy 1.x version of Samza. While a new major version of
> > 2.x
> > > of
> > > > >> > Samza
> > > > >> > > could then make it a requirement that it runs with a YARN
> > cluster
> > > of
> > > > >> > 3.3.x
> > > > >> > > if you want to use Java 11.
> > > > >> > >
> > > > >> > > The only issue there is that you'll probably need to backport
> > > > changes
> > > > >> > > between the 2 versions. But in all honestly, this project does
> > not
> > > > >> look
> > > > >> > > extremely active with commits so it might not be that big of a
> > > > >> problem.
> > > > >> > >
> > > > >> > > -Jamie
> > > > >> > >
> > > > >> > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <
> > > > >> mmcfarland@cavulus.com
> > > > >> > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > >> Hi all,
> > > > >> > >>
> > > > >> > >> I've been doing a little bit of testing with Samza and Hadoop
> > > > 3.3.4;
> > > > >> > >> afaict, in light testing, Samza seems to work fine using the
> > > 2.10.x
> > > > >> > >> hadoop-yarn library against a YARN cluster running 3.3.x. As
> > > Jamie
> > > > >> > pointed
> > > > >> > >> out, YARN didn't incorporate Java 11 compatibility until
> > v3.3.0 (
> > > > >> > >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there
> > any
> > > > >> unit
> > > > >> > >> tests
> > > > >> > >> in Samza that verify compatibility against a YARN cluster? If
> > so,
> > > > >> that
> > > > >> > >> could be a place to validate YARN v2.10/v3.3
> > cross-compatibility.
> > > > >> > >>
> > > > >> > >> Just throwing my 2 cents out there,
> > > > >> > >> Malcolm McFarland
> > > > >> > >> Cavulus
> > > > >> > >>
> > > > >> > >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
> > > > >> > >> <ja...@redfin.com.invalid> wrote:
> > > > >> > >>
> > > > >> > >> > Hey Yi,
> > > > >> > >> >
> > > > >> > >> > Thanks for getting back to me. I have not tried the older
> > yarn
> > > > >> cluster
> > > > >> > >> > version yet in the Samza app running against 3.3.4 but I am
> > > wary
> > > > it
> > > > >> > >> would
> > > > >> > >> > work. Yarn itself is not compatible at 2.10.1 with Java 11
> so
> > > you
> > > > >> > would
> > > > >> > >> > have to update yarn even if the Java library here wasn't
> > > updated.
> > > > >> > >> >
> > > > >> > >> > Could we move this version I'm proposing to a 2.x version
> of
> > > > >> Samza? So
> > > > >> > >> > people that wanted to move forward with yarn upgrade and
> > Samza
> > > > and
> > > > >> > Java
> > > > >> > >> 11
> > > > >> > >> > (like us) could do so? Then 1.x could only be java 8
> > compatible
> > > > and
> > > > >> > 2.x
> > > > >> > >> > could be java 11.
> > > > >> > >> >
> > > > >> > >> > Jamie
> > > > >> > >> >
> > > > >> > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <ni...@gmail.com>
> > > wrote:
> > > > >> > >> >
> > > > >> > >> > > Hey, James,
> > > > >> > >> > >
> > > > >> > >> > > Thanks for the ping. @prateek, can we have someone to
> > review
> > > > this
> > > > >> > >> change?
> > > > >> > >> > >
> > > > >> > >> > > One question: have you tested the change w/ the older
> YARN
> > > > >> cluster
> > > > >> > >> > version
> > > > >> > >> > > (running 2.10.1)? If this change requires YARN cluster
> > > upgrade
> > > > to
> > > > >> > >> 3.3.4
> > > > >> > >> > as
> > > > >> > >> > > well, that may be a breaking change to existing Samza
> users
> > > > (i.e.
> > > > >> > >> > LinkedIn
> > > > >> > >> > > is still running a YARN cluster with version 2.10.1).
> > > > >> > >> > >
> > > > >> > >> > > Best and apologize for the delay.
> > > > >> > >> > >
> > > > >> > >> > > -Yi
> > > > >> > >> > >
> > > > >> > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
> > > > >> > >> > > <ja...@redfin.com.invalid> wrote:
> > > > >> > >> > >
> > > > >> > >> > > > Hey y'all. I just am not sure how to get some traction
> on
> > > > these
> > > > >> > >> Java 11
> > > > >> > >> > > > PRs.
> > > > >> > >> > > >
> > > > >> > >> > > > https://github.com/apache/samza/pull/1628
> > > > >> > >> > > > https://github.com/apache/samza-hello-samza/pull/87
> > > > >> > >> > > >
> > > > >> > >> > > > Would someone that is a maintainer for Samza just let
> us
> > > know
> > > > >> that
> > > > >> > >> > y'all
> > > > >> > >> > > > are looking at them? I can stop pestering you :)
> > > > >> > >> > > >
> > > > >> > >> > > > I ran all tests in both PRs, all pass. I also confirmed
> > > that
> > > > >> using
> > > > >> > >> my
> > > > >> > >> > > Samza
> > > > >> > >> > > > PR in the Hello World app all works with Java 11.
> > > > >> > >> > > >
> > > > >> > >> > > > Thanks!
> > > > >> > >> > > >
> > > > >> > >> > > > -Jamie
> > > > >> > >> > > >
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Java 11 Checkin again
Posted by Yi Pan <ni...@gmail.com>.
Hi, James,
Thanks a lot for reporting this. I will take a look this week.
Best!
-Yi
On Mon, Sep 19, 2022 at 8:20 AM James DeMichele
<ja...@redfin.com.invalid> wrote:
> Hey Yi. I can take a look at this. I do want to point out that your current
> "master" branch is actually broken for using Scala 2.11.
>
> You can repro by just going into the master branch using Java 8 and
> compiling like this
>
> $ java -version
> openjdk version "1.8.0_332"
> OpenJDK Runtime Environment (Temurin)(build 1.8.0_332-b09)
> OpenJDK 64-Bit Server VM (Temurin)(build 25.332-b09, mixed mode)
>
> ./gradlew build -PscalaSuffix=2.11
>
> The build fails with this command using that version of Java 8 ^.
>
> Anyway, just wanted to point that out since I hit this in my branch
> trying to utilize the "bin/check-all.sh" script. That doesn't block
> me/us, but just wanted to call it out.
>
> -Jamie
>
>
>
> On Wed, Sep 14, 2022 at 5:52 PM Yi Pan <ni...@gmail.com> wrote:
>
> > Hey, James,
> >
> > In order to merge your PR without breaking the jdk8 older modules, we
> will
> > need the changes proposed here. Can you try to add those build script
> > changes in the same PR? We will definitely help review and merge it.
> >
> > Best!
> >
> > -Yi
> >
> > On Wed, Sep 14, 2022 at 8:00 AM James DeMichele
> > <ja...@redfin.com.invalid> wrote:
> >
> > > Also, do you have a timeline for when this could be completed? Thanks.
> > >
> > > On Wed, Sep 14, 2022 at 7:16 AM James DeMichele <
> > > james.demichele@redfin.com>
> > > wrote:
> > >
> > > > That sounds like a great solution to me if that works for y'all!
> > > >
> > > > Note too, the Java 11 and yarn 3 module need to only use the Scala
> 2.12
> > > > version of the build.
> > > >
> > > > Jamie
> > > >
> > > >
> > > > On Wed, Sep 14, 2022, 2:38 AM Yi Pan <ni...@gmail.com> wrote:
> > > >
> > > >> Hi, James,
> > > >>
> > > >> Sorry to reply late. I just came back from a trip and had a
> discussion
> > > >> with
> > > >> our internal team as well. So, there is one proposal other than
> > > creating a
> > > >> branch. Let me elaborate it below:
> > > >> a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and
> be
> > > the
> > > >> hosting module for most of the jdk11 related changes.
> > > >> b) modify the build script s.t. samza-yarn3 will only compile and
> > build
> > > >> with jdk11 and samza-yarn only compile and build with jdk8.
> > > >> Thus, we can have two builds: jdk8 build that builds with samza-yarn
> > w/
> > > >> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN
> > 3.3.0.
> > > >> We
> > > >> can manage to publish both jdk8 and jdk11 artifacts if needed.
> > > >> The benefit of this approach is that we can still maintain the trunk
> > > >> release while opening up the jdk11 support.
> > > >>
> > > >> Let me know if that works for you and we can work together to get
> the
> > > code
> > > >> in.
> > > >>
> > > >> Best!
> > > >>
> > > >> -Yi
> > > >>
> > > >> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele
> > > >> <ja...@redfin.com.invalid> wrote:
> > > >>
> > > >> > Hey Yi, I wanted to follow up here and figure what a path forward
> is
> > > >> here.
> > > >> > We need to move to Java 11, and Samza currently is our only
> blocking
> > > >> issue.
> > > >> > In order to move to Java 11, the Yarn Cluster would need to run on
> > > Java
> > > >> 11
> > > >> > correct? If that's the case, then it would need to be 3.3+. I
> don't
> > > know
> > > >> > what it entails on your end to have a new Major version, but that
> > > seems
> > > >> > like a good option here right? Version 2 could be where we can
> move
> > > this
> > > >> > project forward to Java 11, while Version 1 can still remain, and
> > > would
> > > >> not
> > > >> > break people that can't/won't upgrade to Java 11.
> > > >> >
> > > >> > -Jamie
> > > >> >
> > > >> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <
> > > >> james.demichele@redfin.com
> > > >> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Yeah I mean if Samza works fine with the hadoop-yarn library
> > running
> > > >> > > against a 3.3.x YARN cluster, then I don't mind keeping that
> > library
> > > >> of
> > > >> > > 2.10.x in Samza's code. But it is still a moot point in terms of
> > > >> > upgrading
> > > >> > > your YARN cluster, since it must be upgraded to 3.3.x+ in order
> to
> > > be
> > > >> > able
> > > >> > > to run the Cluster with Java 11.
> > > >> > >
> > > >> > > @Yi, I think that moving to a new major version might be the
> > > solution
> > > >> > > here. That way Linkedin can still have a pathway of upgrading
> code
> > > for
> > > >> > the
> > > >> > > old legacy 1.x version of Samza. While a new major version of
> 2.x
> > of
> > > >> > Samza
> > > >> > > could then make it a requirement that it runs with a YARN
> cluster
> > of
> > > >> > 3.3.x
> > > >> > > if you want to use Java 11.
> > > >> > >
> > > >> > > The only issue there is that you'll probably need to backport
> > > changes
> > > >> > > between the 2 versions. But in all honestly, this project does
> not
> > > >> look
> > > >> > > extremely active with commits so it might not be that big of a
> > > >> problem.
> > > >> > >
> > > >> > > -Jamie
> > > >> > >
> > > >> > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <
> > > >> mmcfarland@cavulus.com
> > > >> > >
> > > >> > > wrote:
> > > >> > >
> > > >> > >> Hi all,
> > > >> > >>
> > > >> > >> I've been doing a little bit of testing with Samza and Hadoop
> > > 3.3.4;
> > > >> > >> afaict, in light testing, Samza seems to work fine using the
> > 2.10.x
> > > >> > >> hadoop-yarn library against a YARN cluster running 3.3.x. As
> > Jamie
> > > >> > pointed
> > > >> > >> out, YARN didn't incorporate Java 11 compatibility until
> v3.3.0 (
> > > >> > >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there
> any
> > > >> unit
> > > >> > >> tests
> > > >> > >> in Samza that verify compatibility against a YARN cluster? If
> so,
> > > >> that
> > > >> > >> could be a place to validate YARN v2.10/v3.3
> cross-compatibility.
> > > >> > >>
> > > >> > >> Just throwing my 2 cents out there,
> > > >> > >> Malcolm McFarland
> > > >> > >> Cavulus
> > > >> > >>
> > > >> > >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
> > > >> > >> <ja...@redfin.com.invalid> wrote:
> > > >> > >>
> > > >> > >> > Hey Yi,
> > > >> > >> >
> > > >> > >> > Thanks for getting back to me. I have not tried the older
> yarn
> > > >> cluster
> > > >> > >> > version yet in the Samza app running against 3.3.4 but I am
> > wary
> > > it
> > > >> > >> would
> > > >> > >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so
> > you
> > > >> > would
> > > >> > >> > have to update yarn even if the Java library here wasn't
> > updated.
> > > >> > >> >
> > > >> > >> > Could we move this version I'm proposing to a 2.x version of
> > > >> Samza? So
> > > >> > >> > people that wanted to move forward with yarn upgrade and
> Samza
> > > and
> > > >> > Java
> > > >> > >> 11
> > > >> > >> > (like us) could do so? Then 1.x could only be java 8
> compatible
> > > and
> > > >> > 2.x
> > > >> > >> > could be java 11.
> > > >> > >> >
> > > >> > >> > Jamie
> > > >> > >> >
> > > >> > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <ni...@gmail.com>
> > wrote:
> > > >> > >> >
> > > >> > >> > > Hey, James,
> > > >> > >> > >
> > > >> > >> > > Thanks for the ping. @prateek, can we have someone to
> review
> > > this
> > > >> > >> change?
> > > >> > >> > >
> > > >> > >> > > One question: have you tested the change w/ the older YARN
> > > >> cluster
> > > >> > >> > version
> > > >> > >> > > (running 2.10.1)? If this change requires YARN cluster
> > upgrade
> > > to
> > > >> > >> 3.3.4
> > > >> > >> > as
> > > >> > >> > > well, that may be a breaking change to existing Samza users
> > > (i.e.
> > > >> > >> > LinkedIn
> > > >> > >> > > is still running a YARN cluster with version 2.10.1).
> > > >> > >> > >
> > > >> > >> > > Best and apologize for the delay.
> > > >> > >> > >
> > > >> > >> > > -Yi
> > > >> > >> > >
> > > >> > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
> > > >> > >> > > <ja...@redfin.com.invalid> wrote:
> > > >> > >> > >
> > > >> > >> > > > Hey y'all. I just am not sure how to get some traction on
> > > these
> > > >> > >> Java 11
> > > >> > >> > > > PRs.
> > > >> > >> > > >
> > > >> > >> > > > https://github.com/apache/samza/pull/1628
> > > >> > >> > > > https://github.com/apache/samza-hello-samza/pull/87
> > > >> > >> > > >
> > > >> > >> > > > Would someone that is a maintainer for Samza just let us
> > know
> > > >> that
> > > >> > >> > y'all
> > > >> > >> > > > are looking at them? I can stop pestering you :)
> > > >> > >> > > >
> > > >> > >> > > > I ran all tests in both PRs, all pass. I also confirmed
> > that
> > > >> using
> > > >> > >> my
> > > >> > >> > > Samza
> > > >> > >> > > > PR in the Hello World app all works with Java 11.
> > > >> > >> > > >
> > > >> > >> > > > Thanks!
> > > >> > >> > > >
> > > >> > >> > > > -Jamie
> > > >> > >> > > >
> > > >> > >> > >
> > > >> > >> >
> > > >> > >>
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: Java 11 Checkin again
Posted by James DeMichele <ja...@redfin.com.INVALID>.
Hey Yi. I can take a look at this. I do want to point out that your current
"master" branch is actually broken for using Scala 2.11.
You can repro by just going into the master branch using Java 8 and
compiling like this
$ java -version
openjdk version "1.8.0_332"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (Temurin)(build 25.332-b09, mixed mode)
./gradlew build -PscalaSuffix=2.11
The build fails with this command using that version of Java 8 ^.
Anyway, just wanted to point that out since I hit this in my branch
trying to utilize the "bin/check-all.sh" script. That doesn't block
me/us, but just wanted to call it out.
-Jamie
On Wed, Sep 14, 2022 at 5:52 PM Yi Pan <ni...@gmail.com> wrote:
> Hey, James,
>
> In order to merge your PR without breaking the jdk8 older modules, we will
> need the changes proposed here. Can you try to add those build script
> changes in the same PR? We will definitely help review and merge it.
>
> Best!
>
> -Yi
>
> On Wed, Sep 14, 2022 at 8:00 AM James DeMichele
> <ja...@redfin.com.invalid> wrote:
>
> > Also, do you have a timeline for when this could be completed? Thanks.
> >
> > On Wed, Sep 14, 2022 at 7:16 AM James DeMichele <
> > james.demichele@redfin.com>
> > wrote:
> >
> > > That sounds like a great solution to me if that works for y'all!
> > >
> > > Note too, the Java 11 and yarn 3 module need to only use the Scala 2.12
> > > version of the build.
> > >
> > > Jamie
> > >
> > >
> > > On Wed, Sep 14, 2022, 2:38 AM Yi Pan <ni...@gmail.com> wrote:
> > >
> > >> Hi, James,
> > >>
> > >> Sorry to reply late. I just came back from a trip and had a discussion
> > >> with
> > >> our internal team as well. So, there is one proposal other than
> > creating a
> > >> branch. Let me elaborate it below:
> > >> a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and be
> > the
> > >> hosting module for most of the jdk11 related changes.
> > >> b) modify the build script s.t. samza-yarn3 will only compile and
> build
> > >> with jdk11 and samza-yarn only compile and build with jdk8.
> > >> Thus, we can have two builds: jdk8 build that builds with samza-yarn
> w/
> > >> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN
> 3.3.0.
> > >> We
> > >> can manage to publish both jdk8 and jdk11 artifacts if needed.
> > >> The benefit of this approach is that we can still maintain the trunk
> > >> release while opening up the jdk11 support.
> > >>
> > >> Let me know if that works for you and we can work together to get the
> > code
> > >> in.
> > >>
> > >> Best!
> > >>
> > >> -Yi
> > >>
> > >> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele
> > >> <ja...@redfin.com.invalid> wrote:
> > >>
> > >> > Hey Yi, I wanted to follow up here and figure what a path forward is
> > >> here.
> > >> > We need to move to Java 11, and Samza currently is our only blocking
> > >> issue.
> > >> > In order to move to Java 11, the Yarn Cluster would need to run on
> > Java
> > >> 11
> > >> > correct? If that's the case, then it would need to be 3.3+. I don't
> > know
> > >> > what it entails on your end to have a new Major version, but that
> > seems
> > >> > like a good option here right? Version 2 could be where we can move
> > this
> > >> > project forward to Java 11, while Version 1 can still remain, and
> > would
> > >> not
> > >> > break people that can't/won't upgrade to Java 11.
> > >> >
> > >> > -Jamie
> > >> >
> > >> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <
> > >> james.demichele@redfin.com
> > >> > >
> > >> > wrote:
> > >> >
> > >> > > Yeah I mean if Samza works fine with the hadoop-yarn library
> running
> > >> > > against a 3.3.x YARN cluster, then I don't mind keeping that
> library
> > >> of
> > >> > > 2.10.x in Samza's code. But it is still a moot point in terms of
> > >> > upgrading
> > >> > > your YARN cluster, since it must be upgraded to 3.3.x+ in order to
> > be
> > >> > able
> > >> > > to run the Cluster with Java 11.
> > >> > >
> > >> > > @Yi, I think that moving to a new major version might be the
> > solution
> > >> > > here. That way Linkedin can still have a pathway of upgrading code
> > for
> > >> > the
> > >> > > old legacy 1.x version of Samza. While a new major version of 2.x
> of
> > >> > Samza
> > >> > > could then make it a requirement that it runs with a YARN cluster
> of
> > >> > 3.3.x
> > >> > > if you want to use Java 11.
> > >> > >
> > >> > > The only issue there is that you'll probably need to backport
> > changes
> > >> > > between the 2 versions. But in all honestly, this project does not
> > >> look
> > >> > > extremely active with commits so it might not be that big of a
> > >> problem.
> > >> > >
> > >> > > -Jamie
> > >> > >
> > >> > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <
> > >> mmcfarland@cavulus.com
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > >> Hi all,
> > >> > >>
> > >> > >> I've been doing a little bit of testing with Samza and Hadoop
> > 3.3.4;
> > >> > >> afaict, in light testing, Samza seems to work fine using the
> 2.10.x
> > >> > >> hadoop-yarn library against a YARN cluster running 3.3.x. As
> Jamie
> > >> > pointed
> > >> > >> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 (
> > >> > >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any
> > >> unit
> > >> > >> tests
> > >> > >> in Samza that verify compatibility against a YARN cluster? If so,
> > >> that
> > >> > >> could be a place to validate YARN v2.10/v3.3 cross-compatibility.
> > >> > >>
> > >> > >> Just throwing my 2 cents out there,
> > >> > >> Malcolm McFarland
> > >> > >> Cavulus
> > >> > >>
> > >> > >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
> > >> > >> <ja...@redfin.com.invalid> wrote:
> > >> > >>
> > >> > >> > Hey Yi,
> > >> > >> >
> > >> > >> > Thanks for getting back to me. I have not tried the older yarn
> > >> cluster
> > >> > >> > version yet in the Samza app running against 3.3.4 but I am
> wary
> > it
> > >> > >> would
> > >> > >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so
> you
> > >> > would
> > >> > >> > have to update yarn even if the Java library here wasn't
> updated.
> > >> > >> >
> > >> > >> > Could we move this version I'm proposing to a 2.x version of
> > >> Samza? So
> > >> > >> > people that wanted to move forward with yarn upgrade and Samza
> > and
> > >> > Java
> > >> > >> 11
> > >> > >> > (like us) could do so? Then 1.x could only be java 8 compatible
> > and
> > >> > 2.x
> > >> > >> > could be java 11.
> > >> > >> >
> > >> > >> > Jamie
> > >> > >> >
> > >> > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <ni...@gmail.com>
> wrote:
> > >> > >> >
> > >> > >> > > Hey, James,
> > >> > >> > >
> > >> > >> > > Thanks for the ping. @prateek, can we have someone to review
> > this
> > >> > >> change?
> > >> > >> > >
> > >> > >> > > One question: have you tested the change w/ the older YARN
> > >> cluster
> > >> > >> > version
> > >> > >> > > (running 2.10.1)? If this change requires YARN cluster
> upgrade
> > to
> > >> > >> 3.3.4
> > >> > >> > as
> > >> > >> > > well, that may be a breaking change to existing Samza users
> > (i.e.
> > >> > >> > LinkedIn
> > >> > >> > > is still running a YARN cluster with version 2.10.1).
> > >> > >> > >
> > >> > >> > > Best and apologize for the delay.
> > >> > >> > >
> > >> > >> > > -Yi
> > >> > >> > >
> > >> > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
> > >> > >> > > <ja...@redfin.com.invalid> wrote:
> > >> > >> > >
> > >> > >> > > > Hey y'all. I just am not sure how to get some traction on
> > these
> > >> > >> Java 11
> > >> > >> > > > PRs.
> > >> > >> > > >
> > >> > >> > > > https://github.com/apache/samza/pull/1628
> > >> > >> > > > https://github.com/apache/samza-hello-samza/pull/87
> > >> > >> > > >
> > >> > >> > > > Would someone that is a maintainer for Samza just let us
> know
> > >> that
> > >> > >> > y'all
> > >> > >> > > > are looking at them? I can stop pestering you :)
> > >> > >> > > >
> > >> > >> > > > I ran all tests in both PRs, all pass. I also confirmed
> that
> > >> using
> > >> > >> my
> > >> > >> > > Samza
> > >> > >> > > > PR in the Hello World app all works with Java 11.
> > >> > >> > > >
> > >> > >> > > > Thanks!
> > >> > >> > > >
> > >> > >> > > > -Jamie
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: Java 11 Checkin again
Posted by Yi Pan <ni...@gmail.com>.
Hey, James,
In order to merge your PR without breaking the jdk8 older modules, we will
need the changes proposed here. Can you try to add those build script
changes in the same PR? We will definitely help review and merge it.
Best!
-Yi
On Wed, Sep 14, 2022 at 8:00 AM James DeMichele
<ja...@redfin.com.invalid> wrote:
> Also, do you have a timeline for when this could be completed? Thanks.
>
> On Wed, Sep 14, 2022 at 7:16 AM James DeMichele <
> james.demichele@redfin.com>
> wrote:
>
> > That sounds like a great solution to me if that works for y'all!
> >
> > Note too, the Java 11 and yarn 3 module need to only use the Scala 2.12
> > version of the build.
> >
> > Jamie
> >
> >
> > On Wed, Sep 14, 2022, 2:38 AM Yi Pan <ni...@gmail.com> wrote:
> >
> >> Hi, James,
> >>
> >> Sorry to reply late. I just came back from a trip and had a discussion
> >> with
> >> our internal team as well. So, there is one proposal other than
> creating a
> >> branch. Let me elaborate it below:
> >> a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and be
> the
> >> hosting module for most of the jdk11 related changes.
> >> b) modify the build script s.t. samza-yarn3 will only compile and build
> >> with jdk11 and samza-yarn only compile and build with jdk8.
> >> Thus, we can have two builds: jdk8 build that builds with samza-yarn w/
> >> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN 3.3.0.
> >> We
> >> can manage to publish both jdk8 and jdk11 artifacts if needed.
> >> The benefit of this approach is that we can still maintain the trunk
> >> release while opening up the jdk11 support.
> >>
> >> Let me know if that works for you and we can work together to get the
> code
> >> in.
> >>
> >> Best!
> >>
> >> -Yi
> >>
> >> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele
> >> <ja...@redfin.com.invalid> wrote:
> >>
> >> > Hey Yi, I wanted to follow up here and figure what a path forward is
> >> here.
> >> > We need to move to Java 11, and Samza currently is our only blocking
> >> issue.
> >> > In order to move to Java 11, the Yarn Cluster would need to run on
> Java
> >> 11
> >> > correct? If that's the case, then it would need to be 3.3+. I don't
> know
> >> > what it entails on your end to have a new Major version, but that
> seems
> >> > like a good option here right? Version 2 could be where we can move
> this
> >> > project forward to Java 11, while Version 1 can still remain, and
> would
> >> not
> >> > break people that can't/won't upgrade to Java 11.
> >> >
> >> > -Jamie
> >> >
> >> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <
> >> james.demichele@redfin.com
> >> > >
> >> > wrote:
> >> >
> >> > > Yeah I mean if Samza works fine with the hadoop-yarn library running
> >> > > against a 3.3.x YARN cluster, then I don't mind keeping that library
> >> of
> >> > > 2.10.x in Samza's code. But it is still a moot point in terms of
> >> > upgrading
> >> > > your YARN cluster, since it must be upgraded to 3.3.x+ in order to
> be
> >> > able
> >> > > to run the Cluster with Java 11.
> >> > >
> >> > > @Yi, I think that moving to a new major version might be the
> solution
> >> > > here. That way Linkedin can still have a pathway of upgrading code
> for
> >> > the
> >> > > old legacy 1.x version of Samza. While a new major version of 2.x of
> >> > Samza
> >> > > could then make it a requirement that it runs with a YARN cluster of
> >> > 3.3.x
> >> > > if you want to use Java 11.
> >> > >
> >> > > The only issue there is that you'll probably need to backport
> changes
> >> > > between the 2 versions. But in all honestly, this project does not
> >> look
> >> > > extremely active with commits so it might not be that big of a
> >> problem.
> >> > >
> >> > > -Jamie
> >> > >
> >> > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <
> >> mmcfarland@cavulus.com
> >> > >
> >> > > wrote:
> >> > >
> >> > >> Hi all,
> >> > >>
> >> > >> I've been doing a little bit of testing with Samza and Hadoop
> 3.3.4;
> >> > >> afaict, in light testing, Samza seems to work fine using the 2.10.x
> >> > >> hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie
> >> > pointed
> >> > >> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 (
> >> > >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any
> >> unit
> >> > >> tests
> >> > >> in Samza that verify compatibility against a YARN cluster? If so,
> >> that
> >> > >> could be a place to validate YARN v2.10/v3.3 cross-compatibility.
> >> > >>
> >> > >> Just throwing my 2 cents out there,
> >> > >> Malcolm McFarland
> >> > >> Cavulus
> >> > >>
> >> > >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
> >> > >> <ja...@redfin.com.invalid> wrote:
> >> > >>
> >> > >> > Hey Yi,
> >> > >> >
> >> > >> > Thanks for getting back to me. I have not tried the older yarn
> >> cluster
> >> > >> > version yet in the Samza app running against 3.3.4 but I am wary
> it
> >> > >> would
> >> > >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you
> >> > would
> >> > >> > have to update yarn even if the Java library here wasn't updated.
> >> > >> >
> >> > >> > Could we move this version I'm proposing to a 2.x version of
> >> Samza? So
> >> > >> > people that wanted to move forward with yarn upgrade and Samza
> and
> >> > Java
> >> > >> 11
> >> > >> > (like us) could do so? Then 1.x could only be java 8 compatible
> and
> >> > 2.x
> >> > >> > could be java 11.
> >> > >> >
> >> > >> > Jamie
> >> > >> >
> >> > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <ni...@gmail.com> wrote:
> >> > >> >
> >> > >> > > Hey, James,
> >> > >> > >
> >> > >> > > Thanks for the ping. @prateek, can we have someone to review
> this
> >> > >> change?
> >> > >> > >
> >> > >> > > One question: have you tested the change w/ the older YARN
> >> cluster
> >> > >> > version
> >> > >> > > (running 2.10.1)? If this change requires YARN cluster upgrade
> to
> >> > >> 3.3.4
> >> > >> > as
> >> > >> > > well, that may be a breaking change to existing Samza users
> (i.e.
> >> > >> > LinkedIn
> >> > >> > > is still running a YARN cluster with version 2.10.1).
> >> > >> > >
> >> > >> > > Best and apologize for the delay.
> >> > >> > >
> >> > >> > > -Yi
> >> > >> > >
> >> > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
> >> > >> > > <ja...@redfin.com.invalid> wrote:
> >> > >> > >
> >> > >> > > > Hey y'all. I just am not sure how to get some traction on
> these
> >> > >> Java 11
> >> > >> > > > PRs.
> >> > >> > > >
> >> > >> > > > https://github.com/apache/samza/pull/1628
> >> > >> > > > https://github.com/apache/samza-hello-samza/pull/87
> >> > >> > > >
> >> > >> > > > Would someone that is a maintainer for Samza just let us know
> >> that
> >> > >> > y'all
> >> > >> > > > are looking at them? I can stop pestering you :)
> >> > >> > > >
> >> > >> > > > I ran all tests in both PRs, all pass. I also confirmed that
> >> using
> >> > >> my
> >> > >> > > Samza
> >> > >> > > > PR in the Hello World app all works with Java 11.
> >> > >> > > >
> >> > >> > > > Thanks!
> >> > >> > > >
> >> > >> > > > -Jamie
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >
> >> >
> >>
> >
>
Re: Java 11 Checkin again
Posted by James DeMichele <ja...@redfin.com.INVALID>.
Also, do you have a timeline for when this could be completed? Thanks.
On Wed, Sep 14, 2022 at 7:16 AM James DeMichele <ja...@redfin.com>
wrote:
> That sounds like a great solution to me if that works for y'all!
>
> Note too, the Java 11 and yarn 3 module need to only use the Scala 2.12
> version of the build.
>
> Jamie
>
>
> On Wed, Sep 14, 2022, 2:38 AM Yi Pan <ni...@gmail.com> wrote:
>
>> Hi, James,
>>
>> Sorry to reply late. I just came back from a trip and had a discussion
>> with
>> our internal team as well. So, there is one proposal other than creating a
>> branch. Let me elaborate it below:
>> a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and be the
>> hosting module for most of the jdk11 related changes.
>> b) modify the build script s.t. samza-yarn3 will only compile and build
>> with jdk11 and samza-yarn only compile and build with jdk8.
>> Thus, we can have two builds: jdk8 build that builds with samza-yarn w/
>> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN 3.3.0.
>> We
>> can manage to publish both jdk8 and jdk11 artifacts if needed.
>> The benefit of this approach is that we can still maintain the trunk
>> release while opening up the jdk11 support.
>>
>> Let me know if that works for you and we can work together to get the code
>> in.
>>
>> Best!
>>
>> -Yi
>>
>> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele
>> <ja...@redfin.com.invalid> wrote:
>>
>> > Hey Yi, I wanted to follow up here and figure what a path forward is
>> here.
>> > We need to move to Java 11, and Samza currently is our only blocking
>> issue.
>> > In order to move to Java 11, the Yarn Cluster would need to run on Java
>> 11
>> > correct? If that's the case, then it would need to be 3.3+. I don't know
>> > what it entails on your end to have a new Major version, but that seems
>> > like a good option here right? Version 2 could be where we can move this
>> > project forward to Java 11, while Version 1 can still remain, and would
>> not
>> > break people that can't/won't upgrade to Java 11.
>> >
>> > -Jamie
>> >
>> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <
>> james.demichele@redfin.com
>> > >
>> > wrote:
>> >
>> > > Yeah I mean if Samza works fine with the hadoop-yarn library running
>> > > against a 3.3.x YARN cluster, then I don't mind keeping that library
>> of
>> > > 2.10.x in Samza's code. But it is still a moot point in terms of
>> > upgrading
>> > > your YARN cluster, since it must be upgraded to 3.3.x+ in order to be
>> > able
>> > > to run the Cluster with Java 11.
>> > >
>> > > @Yi, I think that moving to a new major version might be the solution
>> > > here. That way Linkedin can still have a pathway of upgrading code for
>> > the
>> > > old legacy 1.x version of Samza. While a new major version of 2.x of
>> > Samza
>> > > could then make it a requirement that it runs with a YARN cluster of
>> > 3.3.x
>> > > if you want to use Java 11.
>> > >
>> > > The only issue there is that you'll probably need to backport changes
>> > > between the 2 versions. But in all honestly, this project does not
>> look
>> > > extremely active with commits so it might not be that big of a
>> problem.
>> > >
>> > > -Jamie
>> > >
>> > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <
>> mmcfarland@cavulus.com
>> > >
>> > > wrote:
>> > >
>> > >> Hi all,
>> > >>
>> > >> I've been doing a little bit of testing with Samza and Hadoop 3.3.4;
>> > >> afaict, in light testing, Samza seems to work fine using the 2.10.x
>> > >> hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie
>> > pointed
>> > >> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 (
>> > >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any
>> unit
>> > >> tests
>> > >> in Samza that verify compatibility against a YARN cluster? If so,
>> that
>> > >> could be a place to validate YARN v2.10/v3.3 cross-compatibility.
>> > >>
>> > >> Just throwing my 2 cents out there,
>> > >> Malcolm McFarland
>> > >> Cavulus
>> > >>
>> > >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
>> > >> <ja...@redfin.com.invalid> wrote:
>> > >>
>> > >> > Hey Yi,
>> > >> >
>> > >> > Thanks for getting back to me. I have not tried the older yarn
>> cluster
>> > >> > version yet in the Samza app running against 3.3.4 but I am wary it
>> > >> would
>> > >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you
>> > would
>> > >> > have to update yarn even if the Java library here wasn't updated.
>> > >> >
>> > >> > Could we move this version I'm proposing to a 2.x version of
>> Samza? So
>> > >> > people that wanted to move forward with yarn upgrade and Samza and
>> > Java
>> > >> 11
>> > >> > (like us) could do so? Then 1.x could only be java 8 compatible and
>> > 2.x
>> > >> > could be java 11.
>> > >> >
>> > >> > Jamie
>> > >> >
>> > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <ni...@gmail.com> wrote:
>> > >> >
>> > >> > > Hey, James,
>> > >> > >
>> > >> > > Thanks for the ping. @prateek, can we have someone to review this
>> > >> change?
>> > >> > >
>> > >> > > One question: have you tested the change w/ the older YARN
>> cluster
>> > >> > version
>> > >> > > (running 2.10.1)? If this change requires YARN cluster upgrade to
>> > >> 3.3.4
>> > >> > as
>> > >> > > well, that may be a breaking change to existing Samza users (i.e.
>> > >> > LinkedIn
>> > >> > > is still running a YARN cluster with version 2.10.1).
>> > >> > >
>> > >> > > Best and apologize for the delay.
>> > >> > >
>> > >> > > -Yi
>> > >> > >
>> > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
>> > >> > > <ja...@redfin.com.invalid> wrote:
>> > >> > >
>> > >> > > > Hey y'all. I just am not sure how to get some traction on these
>> > >> Java 11
>> > >> > > > PRs.
>> > >> > > >
>> > >> > > > https://github.com/apache/samza/pull/1628
>> > >> > > > https://github.com/apache/samza-hello-samza/pull/87
>> > >> > > >
>> > >> > > > Would someone that is a maintainer for Samza just let us know
>> that
>> > >> > y'all
>> > >> > > > are looking at them? I can stop pestering you :)
>> > >> > > >
>> > >> > > > I ran all tests in both PRs, all pass. I also confirmed that
>> using
>> > >> my
>> > >> > > Samza
>> > >> > > > PR in the Hello World app all works with Java 11.
>> > >> > > >
>> > >> > > > Thanks!
>> > >> > > >
>> > >> > > > -Jamie
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: Java 11 Checkin again
Posted by James DeMichele <ja...@redfin.com.INVALID>.
That sounds like a great solution to me if that works for y'all!
Note too, the Java 11 and yarn 3 module need to only use the Scala 2.12
version of the build.
Jamie
On Wed, Sep 14, 2022, 2:38 AM Yi Pan <ni...@gmail.com> wrote:
> Hi, James,
>
> Sorry to reply late. I just came back from a trip and had a discussion with
> our internal team as well. So, there is one proposal other than creating a
> branch. Let me elaborate it below:
> a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and be the
> hosting module for most of the jdk11 related changes.
> b) modify the build script s.t. samza-yarn3 will only compile and build
> with jdk11 and samza-yarn only compile and build with jdk8.
> Thus, we can have two builds: jdk8 build that builds with samza-yarn w/
> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN 3.3.0. We
> can manage to publish both jdk8 and jdk11 artifacts if needed.
> The benefit of this approach is that we can still maintain the trunk
> release while opening up the jdk11 support.
>
> Let me know if that works for you and we can work together to get the code
> in.
>
> Best!
>
> -Yi
>
> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele
> <ja...@redfin.com.invalid> wrote:
>
> > Hey Yi, I wanted to follow up here and figure what a path forward is
> here.
> > We need to move to Java 11, and Samza currently is our only blocking
> issue.
> > In order to move to Java 11, the Yarn Cluster would need to run on Java
> 11
> > correct? If that's the case, then it would need to be 3.3+. I don't know
> > what it entails on your end to have a new Major version, but that seems
> > like a good option here right? Version 2 could be where we can move this
> > project forward to Java 11, while Version 1 can still remain, and would
> not
> > break people that can't/won't upgrade to Java 11.
> >
> > -Jamie
> >
> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <
> james.demichele@redfin.com
> > >
> > wrote:
> >
> > > Yeah I mean if Samza works fine with the hadoop-yarn library running
> > > against a 3.3.x YARN cluster, then I don't mind keeping that library of
> > > 2.10.x in Samza's code. But it is still a moot point in terms of
> > upgrading
> > > your YARN cluster, since it must be upgraded to 3.3.x+ in order to be
> > able
> > > to run the Cluster with Java 11.
> > >
> > > @Yi, I think that moving to a new major version might be the solution
> > > here. That way Linkedin can still have a pathway of upgrading code for
> > the
> > > old legacy 1.x version of Samza. While a new major version of 2.x of
> > Samza
> > > could then make it a requirement that it runs with a YARN cluster of
> > 3.3.x
> > > if you want to use Java 11.
> > >
> > > The only issue there is that you'll probably need to backport changes
> > > between the 2 versions. But in all honestly, this project does not look
> > > extremely active with commits so it might not be that big of a problem.
> > >
> > > -Jamie
> > >
> > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <
> mmcfarland@cavulus.com
> > >
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> I've been doing a little bit of testing with Samza and Hadoop 3.3.4;
> > >> afaict, in light testing, Samza seems to work fine using the 2.10.x
> > >> hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie
> > pointed
> > >> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 (
> > >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any unit
> > >> tests
> > >> in Samza that verify compatibility against a YARN cluster? If so, that
> > >> could be a place to validate YARN v2.10/v3.3 cross-compatibility.
> > >>
> > >> Just throwing my 2 cents out there,
> > >> Malcolm McFarland
> > >> Cavulus
> > >>
> > >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
> > >> <ja...@redfin.com.invalid> wrote:
> > >>
> > >> > Hey Yi,
> > >> >
> > >> > Thanks for getting back to me. I have not tried the older yarn
> cluster
> > >> > version yet in the Samza app running against 3.3.4 but I am wary it
> > >> would
> > >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you
> > would
> > >> > have to update yarn even if the Java library here wasn't updated.
> > >> >
> > >> > Could we move this version I'm proposing to a 2.x version of Samza?
> So
> > >> > people that wanted to move forward with yarn upgrade and Samza and
> > Java
> > >> 11
> > >> > (like us) could do so? Then 1.x could only be java 8 compatible and
> > 2.x
> > >> > could be java 11.
> > >> >
> > >> > Jamie
> > >> >
> > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <ni...@gmail.com> wrote:
> > >> >
> > >> > > Hey, James,
> > >> > >
> > >> > > Thanks for the ping. @prateek, can we have someone to review this
> > >> change?
> > >> > >
> > >> > > One question: have you tested the change w/ the older YARN cluster
> > >> > version
> > >> > > (running 2.10.1)? If this change requires YARN cluster upgrade to
> > >> 3.3.4
> > >> > as
> > >> > > well, that may be a breaking change to existing Samza users (i.e.
> > >> > LinkedIn
> > >> > > is still running a YARN cluster with version 2.10.1).
> > >> > >
> > >> > > Best and apologize for the delay.
> > >> > >
> > >> > > -Yi
> > >> > >
> > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
> > >> > > <ja...@redfin.com.invalid> wrote:
> > >> > >
> > >> > > > Hey y'all. I just am not sure how to get some traction on these
> > >> Java 11
> > >> > > > PRs.
> > >> > > >
> > >> > > > https://github.com/apache/samza/pull/1628
> > >> > > > https://github.com/apache/samza-hello-samza/pull/87
> > >> > > >
> > >> > > > Would someone that is a maintainer for Samza just let us know
> that
> > >> > y'all
> > >> > > > are looking at them? I can stop pestering you :)
> > >> > > >
> > >> > > > I ran all tests in both PRs, all pass. I also confirmed that
> using
> > >> my
> > >> > > Samza
> > >> > > > PR in the Hello World app all works with Java 11.
> > >> > > >
> > >> > > > Thanks!
> > >> > > >
> > >> > > > -Jamie
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: Java 11 Checkin again
Posted by Yi Pan <ni...@gmail.com>.
Hi, James,
Sorry to reply late. I just came back from a trip and had a discussion with
our internal team as well. So, there is one proposal other than creating a
branch. Let me elaborate it below:
a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and be the
hosting module for most of the jdk11 related changes.
b) modify the build script s.t. samza-yarn3 will only compile and build
with jdk11 and samza-yarn only compile and build with jdk8.
Thus, we can have two builds: jdk8 build that builds with samza-yarn w/
YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN 3.3.0. We
can manage to publish both jdk8 and jdk11 artifacts if needed.
The benefit of this approach is that we can still maintain the trunk
release while opening up the jdk11 support.
Let me know if that works for you and we can work together to get the code
in.
Best!
-Yi
On Tue, Sep 13, 2022 at 7:53 AM James DeMichele
<ja...@redfin.com.invalid> wrote:
> Hey Yi, I wanted to follow up here and figure what a path forward is here.
> We need to move to Java 11, and Samza currently is our only blocking issue.
> In order to move to Java 11, the Yarn Cluster would need to run on Java 11
> correct? If that's the case, then it would need to be 3.3+. I don't know
> what it entails on your end to have a new Major version, but that seems
> like a good option here right? Version 2 could be where we can move this
> project forward to Java 11, while Version 1 can still remain, and would not
> break people that can't/won't upgrade to Java 11.
>
> -Jamie
>
> On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <james.demichele@redfin.com
> >
> wrote:
>
> > Yeah I mean if Samza works fine with the hadoop-yarn library running
> > against a 3.3.x YARN cluster, then I don't mind keeping that library of
> > 2.10.x in Samza's code. But it is still a moot point in terms of
> upgrading
> > your YARN cluster, since it must be upgraded to 3.3.x+ in order to be
> able
> > to run the Cluster with Java 11.
> >
> > @Yi, I think that moving to a new major version might be the solution
> > here. That way Linkedin can still have a pathway of upgrading code for
> the
> > old legacy 1.x version of Samza. While a new major version of 2.x of
> Samza
> > could then make it a requirement that it runs with a YARN cluster of
> 3.3.x
> > if you want to use Java 11.
> >
> > The only issue there is that you'll probably need to backport changes
> > between the 2 versions. But in all honestly, this project does not look
> > extremely active with commits so it might not be that big of a problem.
> >
> > -Jamie
> >
> > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <mmcfarland@cavulus.com
> >
> > wrote:
> >
> >> Hi all,
> >>
> >> I've been doing a little bit of testing with Samza and Hadoop 3.3.4;
> >> afaict, in light testing, Samza seems to work fine using the 2.10.x
> >> hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie
> pointed
> >> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 (
> >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any unit
> >> tests
> >> in Samza that verify compatibility against a YARN cluster? If so, that
> >> could be a place to validate YARN v2.10/v3.3 cross-compatibility.
> >>
> >> Just throwing my 2 cents out there,
> >> Malcolm McFarland
> >> Cavulus
> >>
> >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
> >> <ja...@redfin.com.invalid> wrote:
> >>
> >> > Hey Yi,
> >> >
> >> > Thanks for getting back to me. I have not tried the older yarn cluster
> >> > version yet in the Samza app running against 3.3.4 but I am wary it
> >> would
> >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you
> would
> >> > have to update yarn even if the Java library here wasn't updated.
> >> >
> >> > Could we move this version I'm proposing to a 2.x version of Samza? So
> >> > people that wanted to move forward with yarn upgrade and Samza and
> Java
> >> 11
> >> > (like us) could do so? Then 1.x could only be java 8 compatible and
> 2.x
> >> > could be java 11.
> >> >
> >> > Jamie
> >> >
> >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <ni...@gmail.com> wrote:
> >> >
> >> > > Hey, James,
> >> > >
> >> > > Thanks for the ping. @prateek, can we have someone to review this
> >> change?
> >> > >
> >> > > One question: have you tested the change w/ the older YARN cluster
> >> > version
> >> > > (running 2.10.1)? If this change requires YARN cluster upgrade to
> >> 3.3.4
> >> > as
> >> > > well, that may be a breaking change to existing Samza users (i.e.
> >> > LinkedIn
> >> > > is still running a YARN cluster with version 2.10.1).
> >> > >
> >> > > Best and apologize for the delay.
> >> > >
> >> > > -Yi
> >> > >
> >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
> >> > > <ja...@redfin.com.invalid> wrote:
> >> > >
> >> > > > Hey y'all. I just am not sure how to get some traction on these
> >> Java 11
> >> > > > PRs.
> >> > > >
> >> > > > https://github.com/apache/samza/pull/1628
> >> > > > https://github.com/apache/samza-hello-samza/pull/87
> >> > > >
> >> > > > Would someone that is a maintainer for Samza just let us know that
> >> > y'all
> >> > > > are looking at them? I can stop pestering you :)
> >> > > >
> >> > > > I ran all tests in both PRs, all pass. I also confirmed that using
> >> my
> >> > > Samza
> >> > > > PR in the Hello World app all works with Java 11.
> >> > > >
> >> > > > Thanks!
> >> > > >
> >> > > > -Jamie
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: Java 11 Checkin again
Posted by James DeMichele <ja...@redfin.com.INVALID>.
Hey Yi, I wanted to follow up here and figure what a path forward is here.
We need to move to Java 11, and Samza currently is our only blocking issue.
In order to move to Java 11, the Yarn Cluster would need to run on Java 11
correct? If that's the case, then it would need to be 3.3+. I don't know
what it entails on your end to have a new Major version, but that seems
like a good option here right? Version 2 could be where we can move this
project forward to Java 11, while Version 1 can still remain, and would not
break people that can't/won't upgrade to Java 11.
-Jamie
On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <ja...@redfin.com>
wrote:
> Yeah I mean if Samza works fine with the hadoop-yarn library running
> against a 3.3.x YARN cluster, then I don't mind keeping that library of
> 2.10.x in Samza's code. But it is still a moot point in terms of upgrading
> your YARN cluster, since it must be upgraded to 3.3.x+ in order to be able
> to run the Cluster with Java 11.
>
> @Yi, I think that moving to a new major version might be the solution
> here. That way Linkedin can still have a pathway of upgrading code for the
> old legacy 1.x version of Samza. While a new major version of 2.x of Samza
> could then make it a requirement that it runs with a YARN cluster of 3.3.x
> if you want to use Java 11.
>
> The only issue there is that you'll probably need to backport changes
> between the 2 versions. But in all honestly, this project does not look
> extremely active with commits so it might not be that big of a problem.
>
> -Jamie
>
> On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <mm...@cavulus.com>
> wrote:
>
>> Hi all,
>>
>> I've been doing a little bit of testing with Samza and Hadoop 3.3.4;
>> afaict, in light testing, Samza seems to work fine using the 2.10.x
>> hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie pointed
>> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 (
>> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any unit
>> tests
>> in Samza that verify compatibility against a YARN cluster? If so, that
>> could be a place to validate YARN v2.10/v3.3 cross-compatibility.
>>
>> Just throwing my 2 cents out there,
>> Malcolm McFarland
>> Cavulus
>>
>> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
>> <ja...@redfin.com.invalid> wrote:
>>
>> > Hey Yi,
>> >
>> > Thanks for getting back to me. I have not tried the older yarn cluster
>> > version yet in the Samza app running against 3.3.4 but I am wary it
>> would
>> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you would
>> > have to update yarn even if the Java library here wasn't updated.
>> >
>> > Could we move this version I'm proposing to a 2.x version of Samza? So
>> > people that wanted to move forward with yarn upgrade and Samza and Java
>> 11
>> > (like us) could do so? Then 1.x could only be java 8 compatible and 2.x
>> > could be java 11.
>> >
>> > Jamie
>> >
>> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <ni...@gmail.com> wrote:
>> >
>> > > Hey, James,
>> > >
>> > > Thanks for the ping. @prateek, can we have someone to review this
>> change?
>> > >
>> > > One question: have you tested the change w/ the older YARN cluster
>> > version
>> > > (running 2.10.1)? If this change requires YARN cluster upgrade to
>> 3.3.4
>> > as
>> > > well, that may be a breaking change to existing Samza users (i.e.
>> > LinkedIn
>> > > is still running a YARN cluster with version 2.10.1).
>> > >
>> > > Best and apologize for the delay.
>> > >
>> > > -Yi
>> > >
>> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
>> > > <ja...@redfin.com.invalid> wrote:
>> > >
>> > > > Hey y'all. I just am not sure how to get some traction on these
>> Java 11
>> > > > PRs.
>> > > >
>> > > > https://github.com/apache/samza/pull/1628
>> > > > https://github.com/apache/samza-hello-samza/pull/87
>> > > >
>> > > > Would someone that is a maintainer for Samza just let us know that
>> > y'all
>> > > > are looking at them? I can stop pestering you :)
>> > > >
>> > > > I ran all tests in both PRs, all pass. I also confirmed that using
>> my
>> > > Samza
>> > > > PR in the Hello World app all works with Java 11.
>> > > >
>> > > > Thanks!
>> > > >
>> > > > -Jamie
>> > > >
>> > >
>> >
>>
>
Re: Java 11 Checkin again
Posted by James DeMichele <ja...@redfin.com.INVALID>.
Yeah I mean if Samza works fine with the hadoop-yarn library running
against a 3.3.x YARN cluster, then I don't mind keeping that library of
2.10.x in Samza's code. But it is still a moot point in terms of upgrading
your YARN cluster, since it must be upgraded to 3.3.x+ in order to be able
to run the Cluster with Java 11.
@Yi, I think that moving to a new major version might be the solution here.
That way Linkedin can still have a pathway of upgrading code for the old
legacy 1.x version of Samza. While a new major version of 2.x of Samza
could then make it a requirement that it runs with a YARN cluster of 3.3.x
if you want to use Java 11.
The only issue there is that you'll probably need to backport changes
between the 2 versions. But in all honestly, this project does not look
extremely active with commits so it might not be that big of a problem.
-Jamie
On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <mm...@cavulus.com>
wrote:
> Hi all,
>
> I've been doing a little bit of testing with Samza and Hadoop 3.3.4;
> afaict, in light testing, Samza seems to work fine using the 2.10.x
> hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie pointed
> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 (
> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any unit
> tests
> in Samza that verify compatibility against a YARN cluster? If so, that
> could be a place to validate YARN v2.10/v3.3 cross-compatibility.
>
> Just throwing my 2 cents out there,
> Malcolm McFarland
> Cavulus
>
> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
> <ja...@redfin.com.invalid> wrote:
>
> > Hey Yi,
> >
> > Thanks for getting back to me. I have not tried the older yarn cluster
> > version yet in the Samza app running against 3.3.4 but I am wary it would
> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you would
> > have to update yarn even if the Java library here wasn't updated.
> >
> > Could we move this version I'm proposing to a 2.x version of Samza? So
> > people that wanted to move forward with yarn upgrade and Samza and Java
> 11
> > (like us) could do so? Then 1.x could only be java 8 compatible and 2.x
> > could be java 11.
> >
> > Jamie
> >
> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <ni...@gmail.com> wrote:
> >
> > > Hey, James,
> > >
> > > Thanks for the ping. @prateek, can we have someone to review this
> change?
> > >
> > > One question: have you tested the change w/ the older YARN cluster
> > version
> > > (running 2.10.1)? If this change requires YARN cluster upgrade to 3.3.4
> > as
> > > well, that may be a breaking change to existing Samza users (i.e.
> > LinkedIn
> > > is still running a YARN cluster with version 2.10.1).
> > >
> > > Best and apologize for the delay.
> > >
> > > -Yi
> > >
> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
> > > <ja...@redfin.com.invalid> wrote:
> > >
> > > > Hey y'all. I just am not sure how to get some traction on these Java
> 11
> > > > PRs.
> > > >
> > > > https://github.com/apache/samza/pull/1628
> > > > https://github.com/apache/samza-hello-samza/pull/87
> > > >
> > > > Would someone that is a maintainer for Samza just let us know that
> > y'all
> > > > are looking at them? I can stop pestering you :)
> > > >
> > > > I ran all tests in both PRs, all pass. I also confirmed that using my
> > > Samza
> > > > PR in the Hello World app all works with Java 11.
> > > >
> > > > Thanks!
> > > >
> > > > -Jamie
> > > >
> > >
> >
>
Re: Java 11 Checkin again
Posted by Malcolm McFarland <mm...@cavulus.com>.
Hi all,
I've been doing a little bit of testing with Samza and Hadoop 3.3.4;
afaict, in light testing, Samza seems to work fine using the 2.10.x
hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie pointed
out, YARN didn't incorporate Java 11 compatibility until v3.3.0 (
https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any unit tests
in Samza that verify compatibility against a YARN cluster? If so, that
could be a place to validate YARN v2.10/v3.3 cross-compatibility.
Just throwing my 2 cents out there,
Malcolm McFarland
Cavulus
On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
<ja...@redfin.com.invalid> wrote:
> Hey Yi,
>
> Thanks for getting back to me. I have not tried the older yarn cluster
> version yet in the Samza app running against 3.3.4 but I am wary it would
> work. Yarn itself is not compatible at 2.10.1 with Java 11 so you would
> have to update yarn even if the Java library here wasn't updated.
>
> Could we move this version I'm proposing to a 2.x version of Samza? So
> people that wanted to move forward with yarn upgrade and Samza and Java 11
> (like us) could do so? Then 1.x could only be java 8 compatible and 2.x
> could be java 11.
>
> Jamie
>
> On Fri, Sep 2, 2022, 6:44 PM Yi Pan <ni...@gmail.com> wrote:
>
> > Hey, James,
> >
> > Thanks for the ping. @prateek, can we have someone to review this change?
> >
> > One question: have you tested the change w/ the older YARN cluster
> version
> > (running 2.10.1)? If this change requires YARN cluster upgrade to 3.3.4
> as
> > well, that may be a breaking change to existing Samza users (i.e.
> LinkedIn
> > is still running a YARN cluster with version 2.10.1).
> >
> > Best and apologize for the delay.
> >
> > -Yi
> >
> > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
> > <ja...@redfin.com.invalid> wrote:
> >
> > > Hey y'all. I just am not sure how to get some traction on these Java 11
> > > PRs.
> > >
> > > https://github.com/apache/samza/pull/1628
> > > https://github.com/apache/samza-hello-samza/pull/87
> > >
> > > Would someone that is a maintainer for Samza just let us know that
> y'all
> > > are looking at them? I can stop pestering you :)
> > >
> > > I ran all tests in both PRs, all pass. I also confirmed that using my
> > Samza
> > > PR in the Hello World app all works with Java 11.
> > >
> > > Thanks!
> > >
> > > -Jamie
> > >
> >
>
Re: Java 11 Checkin again
Posted by James DeMichele <ja...@redfin.com.INVALID>.
Hey Yi,
Thanks for getting back to me. I have not tried the older yarn cluster
version yet in the Samza app running against 3.3.4 but I am wary it would
work. Yarn itself is not compatible at 2.10.1 with Java 11 so you would
have to update yarn even if the Java library here wasn't updated.
Could we move this version I'm proposing to a 2.x version of Samza? So
people that wanted to move forward with yarn upgrade and Samza and Java 11
(like us) could do so? Then 1.x could only be java 8 compatible and 2.x
could be java 11.
Jamie
On Fri, Sep 2, 2022, 6:44 PM Yi Pan <ni...@gmail.com> wrote:
> Hey, James,
>
> Thanks for the ping. @prateek, can we have someone to review this change?
>
> One question: have you tested the change w/ the older YARN cluster version
> (running 2.10.1)? If this change requires YARN cluster upgrade to 3.3.4 as
> well, that may be a breaking change to existing Samza users (i.e. LinkedIn
> is still running a YARN cluster with version 2.10.1).
>
> Best and apologize for the delay.
>
> -Yi
>
> On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
> <ja...@redfin.com.invalid> wrote:
>
> > Hey y'all. I just am not sure how to get some traction on these Java 11
> > PRs.
> >
> > https://github.com/apache/samza/pull/1628
> > https://github.com/apache/samza-hello-samza/pull/87
> >
> > Would someone that is a maintainer for Samza just let us know that y'all
> > are looking at them? I can stop pestering you :)
> >
> > I ran all tests in both PRs, all pass. I also confirmed that using my
> Samza
> > PR in the Hello World app all works with Java 11.
> >
> > Thanks!
> >
> > -Jamie
> >
>
Re: Java 11 Checkin again
Posted by Yi Pan <ni...@gmail.com>.
Hey, James,
Thanks for the ping. @prateek, can we have someone to review this change?
One question: have you tested the change w/ the older YARN cluster version
(running 2.10.1)? If this change requires YARN cluster upgrade to 3.3.4 as
well, that may be a breaking change to existing Samza users (i.e. LinkedIn
is still running a YARN cluster with version 2.10.1).
Best and apologize for the delay.
-Yi
On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
<ja...@redfin.com.invalid> wrote:
> Hey y'all. I just am not sure how to get some traction on these Java 11
> PRs.
>
> https://github.com/apache/samza/pull/1628
> https://github.com/apache/samza-hello-samza/pull/87
>
> Would someone that is a maintainer for Samza just let us know that y'all
> are looking at them? I can stop pestering you :)
>
> I ran all tests in both PRs, all pass. I also confirmed that using my Samza
> PR in the Hello World app all works with Java 11.
>
> Thanks!
>
> -Jamie
>