Add an ARM CI for Hadoop

[Zhenyu Zheng <zh...@gmail.com>]

Hi Yetus,

Currently we are working on making Big Data software more capable on ARM
plaforms, and one of the method is to add ARM CI to these projects and try
to keep the CI green. So the first step is to add the CI, I've reported an
issue: https://issues.apache.org/jira/browse/HADOOP-16358 and yetus is
where I suggested to ask for help since it is what has been used for Hadoop
CI. So here I am.

So, is here the correct place to ask for help and propose to add this CI?
We are willing to provide resources for the CI and man power to setup the
CI and keep it green in the future. We have already found some issues and
started to working on fixing them in the community, such as
https://issues.apache.org/jira/browse/HADOOP-13363 , but it will be better
to show the process of what we have fixed and overall view of what is still
wrong if we can have an ARM CI in upstream. Here is an example for what we
have recently done for Apache Flink:
https://issues.apache.org/jira/browse/INFRA-18822

Thanks in advance,

Kevin Zheng

Re: Add an ARM CI for Hadoop

[Zhenyu Zheng <zh...@gmail.com>]

Thanks alot for the info and advise, it seems that option2 will be a better
choice since it will be something we can start to do right away while we
looking for a champion, and it might be easier for us to push forward after
we find
our champion if we already have something in yetus. I will read the links.

Thanks again

Kevin

On Wed, Aug 14, 2019 at 11:10 PM Allen Wittenauer
<aw...@effectivemachines.com.invalid> wrote:

>
>
> > On Aug 11, 2019, at 11:55 PM, Zhenyu Zheng <zh...@gmail.com>
> wrote:
> > So, is here the correct place to ask for help and propose to add this CI?
>
>         This is the place to come ask questions related to Apache Yetus,
> not necessarily any particular project’s usage of it.
>
> > We are willing to provide resources for the CI and man power to setup the
> > CI and keep it green in the future. We have already found some issues and
> > started to working on fixing them in the community, such as
> > https://issues.apache.org/jira/browse/HADOOP-13363 , but it will be
> better
> > to show the process of what we have fixed and overall view of what is
> still
> > wrong if we can have an ARM CI in upstream. Here is an example for what
> we
> > have recently done for Apache Flink:
> > https://issues.apache.org/jira/browse/INFRA-18822
>
> There are two routes to take:
>
>         #1: Getting an ARM machine hooked up to the ASF Jenkins system
>
>         INFRA is your best contact for doing that.  After that, you’ll
> need to work with the Hadoop community on getting jobs/processes in place
> to make ARM a first class citizen. (One of the big ones will be making an
> ARM-specific Dockerfile!)
>
>         Unfortunately, the folks in charge of Hadoop have a track record
> of not being particularly friendly to contributions outside of the clique
> or if it doesn’t further goals by their employers. The only way to really
> grab their attention is to start an email campaign on
> common-dev@hadoop.apache.org.  Just be aware that you’ll need to be
> extremely persistent to get anywhere.  It’s probably also worth mentioning
> that attempts to get other architectures/OSes (Windows, PowerPC, OS X, etc)
> as first class CI tests have failed multiple times because, again, the
> majority of the community’s employers don’t work with that architecture and
> do not have a vested interested in making it succeed.
>
>         #2: Going a completely different route with OpenLab CI
>
>         If you plan on following the same route as you did with Apache
> Flink (bypass Jenkins and do something completely custom), then you’ll
> likely be interested in integrating Apache Yetus with OpenLab CI. This is
> where we can help.  But to get started, you’ll want to create some robot
> support:
>
> Code:
>
>
> https://github.com/apache/yetus/tree/master/precommit/src/main/shell/robots.d
>
> Docs:
>
>
> http://yetus.apache.org/documentation/in-progress/precommit-robots/
>
>         I’d recommend starting with a source tree you know well or is
> relatively simple while building the support before tackling Hadoop
> straight on. Yetus tends to shed a lot of light on short-signed CI system
> design so there may be some workarounds required and/or new Yetus APIs
> we’ll need to add.  To make matters worse, Hadoop is notoriously broken in
> lots of ways, so has a tendency to light up quality test results that will
> make it seem like your robot code is broken.
>
>         After you get the robot support, you’re likely back to talking
> with the Hadoop community about how to integrate your CI system with their
> contribution procedures. So ultimately, you’ll be back to convincing them
> that this is something important to work on.  You really need to find a
> champion on the PMC to help make it happen.
>
>         Good luck!
>
>

Re: Add an ARM CI for Hadoop

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Aug 11, 2019, at 11:55 PM, Zhenyu Zheng <zh...@gmail.com> wrote:
> So, is here the correct place to ask for help and propose to add this CI?

	This is the place to come ask questions related to Apache Yetus, not necessarily any particular project’s usage of it.

> We are willing to provide resources for the CI and man power to setup the
> CI and keep it green in the future. We have already found some issues and
> started to working on fixing them in the community, such as
> https://issues.apache.org/jira/browse/HADOOP-13363 , but it will be better
> to show the process of what we have fixed and overall view of what is still
> wrong if we can have an ARM CI in upstream. Here is an example for what we
> have recently done for Apache Flink:
> https://issues.apache.org/jira/browse/INFRA-18822

There are two routes to take:

	#1: Getting an ARM machine hooked up to the ASF Jenkins system

	INFRA is your best contact for doing that.  After that, you’ll need to work with the Hadoop community on getting jobs/processes in place to make ARM a first class citizen. (One of the big ones will be making an ARM-specific Dockerfile!)

	Unfortunately, the folks in charge of Hadoop have a track record of not being particularly friendly to contributions outside of the clique or if it doesn’t further goals by their employers. The only way to really grab their attention is to start an email campaign on common-dev@hadoop.apache.org.  Just be aware that you’ll need to be extremely persistent to get anywhere.  It’s probably also worth mentioning that attempts to get other architectures/OSes (Windows, PowerPC, OS X, etc) as first class CI tests have failed multiple times because, again, the majority of the community’s employers don’t work with that architecture and do not have a vested interested in making it succeed.

	#2: Going a completely different route with OpenLab CI

	If you plan on following the same route as you did with Apache Flink (bypass Jenkins and do something completely custom), then you’ll likely be interested in integrating Apache Yetus with OpenLab CI. This is where we can help.  But to get started, you’ll want to create some robot support: 

Code:

	https://github.com/apache/yetus/tree/master/precommit/src/main/shell/robots.d

Docs:

	http://yetus.apache.org/documentation/in-progress/precommit-robots/

	I’d recommend starting with a source tree you know well or is relatively simple while building the support before tackling Hadoop straight on. Yetus tends to shed a lot of light on short-signed CI system design so there may be some workarounds required and/or new Yetus APIs we’ll need to add.  To make matters worse, Hadoop is notoriously broken in lots of ways, so has a tendency to light up quality test results that will make it seem like your robot code is broken. 

	After you get the robot support, you’re likely back to talking with the Hadoop community about how to integrate your CI system with their contribution procedures. So ultimately, you’ll be back to convincing them that this is something important to work on.  You really need to find a champion on the PMC to help make it happen.

	Good luck!