You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Julien Faye <ju...@gmail.com> on 2021/12/06 10:06:15 UTC
RE: RE: Testing Oozie on Linux ARM64
Hi,
Not sure whether my vote matters but I'd also like to support this proposal!
Lately we use more and more ARM64 servers and I believe this is the future!
Julien
On 2021/11/30 09:28:27 Mark Jens wrote:
> Hi Martin,
>
> Since recently at my daily job we evaluate using Linux ARM64 servers for
> our needs.
> Based on my humble experience with Linux ARM64 I agree that even JVM based
> software should be tested on non-x86_64 platforms.
> I agree that most of the time JVM software just works but in few weeks
> since I experiment with ARM64 I've found issues in Apache Phoenix, Apache
> HBase (2.x) and Apache Accumulo .
>
> Thank you for your work to improve Oozie to work on ARM64!
>
> Kind regards,
> Mark
>
> On 2021/10/11 07:27:50 Martin Grigorov wrote:
> > Dear Apache Oozie devs,
> >
> > Recently I've tried to build Oozie on Linux Ubuntu 20.04 ARM64 [1].
> > There were two small issues but after some debugging I was able to fix
> them
> > and build it successfully!
> >
> > As a Java application Oozie should run on any of the supported platforms
> by
> > JDK/JRE!
> > This is true until the application uses native libraries. In Oozie's
case
> > the problem was an old version of snappy-java library coming as a
> > transitive dependency of Apache Spark.
> > The second issue was due to too new version of Python (3.9.5). It seems
> > PySpark does not support it yet.
> >
> > Since Linux ARM64 is more and more being used on the server/cloud I'd
like
> > to ask Oozie devs whether you are interested adding Jenkins agent that
> will
> > test the commits/patches or even run periodically (e.g. nightly/weekly)
to
> > prevent regressions when updating dependencies and to give some
confidence
> > to Oozie users that they could deploy it on such platform.
> >
> > Disclaimer: I work for OpenLab Testing (https://docs.openlabtesting.org/
)
> > and we could donate a Linux ARM64 VM that could be used as Jenkins agent
> > for long term usage! Several other Apache projects already use such VMs,
> > e.g. Hadoop, Cassandra, Kafka, Mesos, ...
> > Other projects use TravisCI for testing on arm64/ppc64/s390x. In case
you
> > prefer TravisCI then I could help with a patch adding .travis.yml to
> Oozie!
> >
> > 1. https://issues.apache.org/jira/browse/OOZIE-3639
> >
> > Regards,
> > Martin
> >
>
Re: RE: Testing Oozie on Linux ARM64
Posted by Martin Grigorov <mg...@apache.org>.
Hi Dénes,
On Tue, Dec 7, 2021 at 2:38 PM Dénes Bodó <di...@apache.org> wrote:
> Hey Martin,
>
> Thank you for your offer and also for your fix in Spark action.
>
> Testing on ARM sounds good and currently this is not happening with Oozie.
> I'd happily accept your offer. However, it depends on the requirements.
>
I'm glad to hear that!
> Can you please share what are financial, legal and technical preconditions
> to meet to test Oozie on ARM?
>
There are no financial and legal requirements/preconditions!
The VM will be donated by OpenLab Testing to ASF for long term usage for
free.
The technical ones are:
1) please tell me:
- what Linux distro and version do you prefer? CentOS, Ubuntu, ...
- preferred hardware spec (CPU, RAM, disk space, anything else that is
important for the project). Those parameters could be modified later too if
needed!
2) I will provide SSH details of the Linux ARM64 VM to the Oozie and Infra
teams.
Infra team will set it up as an Jenkins agent and configure it to be used
by Oozie project
Oozie team can use SSH to connect to the VM to debug build/test issues when
needed.
That's it!
Regards,
Martin
> Thanks,
> Denes
>
> Julien Faye <ju...@gmail.com> ezt írta (időpont: 2021. dec. 6., H,
> 11:06):
>
> > Hi,
> >
> > Not sure whether my vote matters but I'd also like to support this
> > proposal!
> > Lately we use more and more ARM64 servers and I believe this is the
> future!
> >
> > Julien
> >
> > On 2021/11/30 09:28:27 Mark Jens wrote:
> > > Hi Martin,
> > >
> > > Since recently at my daily job we evaluate using Linux ARM64 servers
> for
> > > our needs.
> > > Based on my humble experience with Linux ARM64 I agree that even JVM
> > based
> > > software should be tested on non-x86_64 platforms.
> > > I agree that most of the time JVM software just works but in few weeks
> > > since I experiment with ARM64 I've found issues in Apache Phoenix,
> Apache
> > > HBase (2.x) and Apache Accumulo .
> > >
> > > Thank you for your work to improve Oozie to work on ARM64!
> > >
> > > Kind regards,
> > > Mark
> > >
> > > On 2021/10/11 07:27:50 Martin Grigorov wrote:
> > > > Dear Apache Oozie devs,
> > > >
> > > > Recently I've tried to build Oozie on Linux Ubuntu 20.04 ARM64 [1].
> > > > There were two small issues but after some debugging I was able to
> fix
> > > them
> > > > and build it successfully!
> > > >
> > > > As a Java application Oozie should run on any of the supported
> > platforms
> > > by
> > > > JDK/JRE!
> > > > This is true until the application uses native libraries. In Oozie's
> > case
> > > > the problem was an old version of snappy-java library coming as a
> > > > transitive dependency of Apache Spark.
> > > > The second issue was due to too new version of Python (3.9.5). It
> seems
> > > > PySpark does not support it yet.
> > > >
> > > > Since Linux ARM64 is more and more being used on the server/cloud I'd
> > like
> > > > to ask Oozie devs whether you are interested adding Jenkins agent
> that
> > > will
> > > > test the commits/patches or even run periodically (e.g.
> nightly/weekly)
> > to
> > > > prevent regressions when updating dependencies and to give some
> > confidence
> > > > to Oozie users that they could deploy it on such platform.
> > > >
> > > > Disclaimer: I work for OpenLab Testing (
> > https://docs.openlabtesting.org/
> > )
> > > > and we could donate a Linux ARM64 VM that could be used as Jenkins
> > agent
> > > > for long term usage! Several other Apache projects already use such
> > VMs,
> > > > e.g. Hadoop, Cassandra, Kafka, Mesos, ...
> > > > Other projects use TravisCI for testing on arm64/ppc64/s390x. In case
> > you
> > > > prefer TravisCI then I could help with a patch adding .travis.yml to
> > > Oozie!
> > > >
> > > > 1. https://issues.apache.org/jira/browse/OOZIE-3639
> > > >
> > > > Regards,
> > > > Martin
> > > >
> > >
> >
>
Re: RE: Testing Oozie on Linux ARM64
Posted by Dénes Bodó <di...@apache.org>.
Hey Martin,
Thank you for your offer and also for your fix in Spark action.
Testing on ARM sounds good and currently this is not happening with Oozie.
I'd happily accept your offer. However, it depends on the requirements.
Can you please share what are financial, legal and technical preconditions
to meet to test Oozie on ARM?
Thanks,
Denes
Julien Faye <ju...@gmail.com> ezt írta (időpont: 2021. dec. 6., H,
11:06):
> Hi,
>
> Not sure whether my vote matters but I'd also like to support this
> proposal!
> Lately we use more and more ARM64 servers and I believe this is the future!
>
> Julien
>
> On 2021/11/30 09:28:27 Mark Jens wrote:
> > Hi Martin,
> >
> > Since recently at my daily job we evaluate using Linux ARM64 servers for
> > our needs.
> > Based on my humble experience with Linux ARM64 I agree that even JVM
> based
> > software should be tested on non-x86_64 platforms.
> > I agree that most of the time JVM software just works but in few weeks
> > since I experiment with ARM64 I've found issues in Apache Phoenix, Apache
> > HBase (2.x) and Apache Accumulo .
> >
> > Thank you for your work to improve Oozie to work on ARM64!
> >
> > Kind regards,
> > Mark
> >
> > On 2021/10/11 07:27:50 Martin Grigorov wrote:
> > > Dear Apache Oozie devs,
> > >
> > > Recently I've tried to build Oozie on Linux Ubuntu 20.04 ARM64 [1].
> > > There were two small issues but after some debugging I was able to fix
> > them
> > > and build it successfully!
> > >
> > > As a Java application Oozie should run on any of the supported
> platforms
> > by
> > > JDK/JRE!
> > > This is true until the application uses native libraries. In Oozie's
> case
> > > the problem was an old version of snappy-java library coming as a
> > > transitive dependency of Apache Spark.
> > > The second issue was due to too new version of Python (3.9.5). It seems
> > > PySpark does not support it yet.
> > >
> > > Since Linux ARM64 is more and more being used on the server/cloud I'd
> like
> > > to ask Oozie devs whether you are interested adding Jenkins agent that
> > will
> > > test the commits/patches or even run periodically (e.g. nightly/weekly)
> to
> > > prevent regressions when updating dependencies and to give some
> confidence
> > > to Oozie users that they could deploy it on such platform.
> > >
> > > Disclaimer: I work for OpenLab Testing (
> https://docs.openlabtesting.org/
> )
> > > and we could donate a Linux ARM64 VM that could be used as Jenkins
> agent
> > > for long term usage! Several other Apache projects already use such
> VMs,
> > > e.g. Hadoop, Cassandra, Kafka, Mesos, ...
> > > Other projects use TravisCI for testing on arm64/ppc64/s390x. In case
> you
> > > prefer TravisCI then I could help with a patch adding .travis.yml to
> > Oozie!
> > >
> > > 1. https://issues.apache.org/jira/browse/OOZIE-3639
> > >
> > > Regards,
> > > Martin
> > >
> >
>