Building 1.2.1 on aarch64 platform

[Marcin Juszkiewicz <ma...@linaro.org>]

Hi

I was supposed to work on getting two aarch64 VM systems operational for
bigtop CI tasks. But how to get such stuff working without finding out
how it works first? And does it work?

So I did builds of 1.2.1 and master branches on x86-64 under Debian 9
'stretch' and reported bunch of issues.

Today I did a build of 1.2.1 on fresh Debian 8 'jessie' on aarch64
system as this distribution was supported both by Linaro and by Bigtop
1.2.1 release.

Packages which built fine:

- alluxio
- apex
- bigtop-groovy
- bigtop-utils
- crunch
- flink
- flume
- giraph
- hadoop
- hama
- hive
- hue
- ignite-hadoop
- mahout
- oozie
- phoenix
- pig
- qfs
- solr
- spark1
- sqoop
- sqoop2
- tajo
- ycsb
- zeppelin
- zookeeper

Failures:

- ambari (uses x86-64 nodejs binaries)
- bigtop-jsvc ("Java Home not defined")
- bigtop-tomcat (have to check)
- datafu (no gradle binary)
- hbase (something about org.asciidoctor)
- kafka (no gradle binary)
- spark (no JAVA_HOME)
- tez (x86-64 nodejs binaries)

I followed "how to build bigtop trunk" document (using 'release-1.2.1'
git tag as base) on Debian 8 'jessie' system with 'jessie-backports'
repository enabled.

Will take a look at failures. Also plan to do Bigtop 'master' build on
fresh Debian 9 'stretch' aarch64 system to catch some bugs and to check
existing fixes.

Sorry for short mail without many details but it is 22:00 here and I do
that only because Olaf is online and updates Jira so my MiBand 2 thing
interrupted watching some TV series ;D

Ask, comment - will reply with details and provide logs.

Re: Building 1.2.1 on aarch64 platform

[Marcin Juszkiewicz <ma...@linaro.org>]

W dniu 04.12.2017 o 22:14, Olaf Flebbe pisze:

> I like your approach first to reproduce. But I do not think it does
> make sense to reproduce 1.2.1, lets advance on 1.3

I was asked for 1.2.1 too.

> First things first. Do you know:
> 
> https://ci.bigtop.apache.org/job/Bigtop-trunk-packages/ ?
> 
> Please create an account at the ci and I will look into it giving you
> more rights that you can see the magic behind it. Its poorly
> documented.

Created "hrw" account there.

> IMHO the hue patch will turn the hue red line to blue.

Cool.

> I will commit your oozie patch. I think it will not help and let
> oozie fail differently. Hope you are able to reproduce that as well.

oozie is blue again.

>> Failures:
>> 
>> - ambari (uses x86-64 nodejs binaries)
> 
> noteable it is phantomjs. You may use the workaround from tez in
> 1.2.1 (which I removed for 1.3) to solve ths issue.

>> - bigtop-jsvc ("Java Home not defined")
> I will propose to remove that because it is supplied by every linux
> distro. Was there because of centos-6

built

>> - bigtop-tomcat (have to check)
> ok? Will propose to update to 6.0.52

built

>> - datafu (no gradle binary)
> ???

 built

>> - hbase (something about org.asciidoctor)
> It is fixed with 1.3. We had old versions in nexus.
> 
>> - kafka (no gradle binary)
> ??

 built

>> - spark (no JAVA_HOME)
> ??

 built

>> - tez (x86-64 nodejs binaries)
> Solved in 1.3

>> Will take a look at failures. Also plan to do Bigtop 'master' build
>> on fresh Debian 9 'stretch' aarch64 system to catch some bugs and
>> to check existing fixes.

> Do you have a stable docker for aarch64 ? This is a prerequiste for
> advancing with bigtop.

On Debian 'stretch' we have docker-ce 17.06 available. And 1.12.x for
CentOS 7/Debian 8.

I already used Docker during tests on x86-64. Plan to use dockerized way
on aarch64 as well.

Re: Building 1.2.1 on aarch64 platform

[Marcin Juszkiewicz <ma...@linaro.org>]

W dniu 04.12.2017 o 22:14, Olaf Flebbe pisze:
> I like your approach first to reproduce. But I do not think it does
> make sense to reproduce 1.2.1, lets advance on 1.3

package name	|	Debian 9	|	Fedora 26
----------------+-----------------------+--------------------
alluxio		|	built		|	built
ambari		|	failed		|	failed
apex		|	built		|	built
bigtop-groovy	|	built		|	built
bigtop-jsvc	|	built		|	failed
bigtop-tomcat	|	built		|	built
bigtop-utils	|	built		|	built
crunch		|	built		|	built
datafu		|	built		|	built
flink		|	built		|	built
flume		|	built		|	built
giraph		|	built		|	built
gpdb		|	built		|	built
hadoop		|	built		|	failed
hama		|	built		|	failed
hbase		|	failed		|	failed
hive		|	built		|	built
hue		|	failed		|	failed
ignite-hadoop	|	built		|	built
kafka		|	built		|	built
mahout		|	built		|	built
oozie		|	built		|	built
phoenix		|	built		|	built
pig		|	built		|	built
qfs		|	failed		|	failed
solr		|	built		|	built
spark		|	built		|	built
spark1		|	built		|	built
sqoop		|	built		|	built
sqoop2		|	built		|	built
tajo		|	built		|	failed
tez		|	built		|	failed
ycsb		|	built		|	built
zeppelin	|	built		|	built
zookeeper	|	built		|	built

Have to reattempt to build Fedora packages one by one and gather logs.

Re: Building 1.2.1 on aarch64 platform

[Olaf Flebbe <of...@oflebbe.de>]

Hi Marcin,

we are in the same timezone :-) You may try to contact me on IRC. But not tonight any more, will leave shortly.

I like your approach first to reproduce. But I do not think it does make sense to reproduce 1.2.1, lets advance on 1.3

First things first. Do you know:

https://ci.bigtop.apache.org/job/Bigtop-trunk-packages/ ?

Please create an account at the ci and I will look into it giving you more rights that you can see the magic behind it. Its poorly documented.

IMHO the hue patch will turn the hue red line to blue.

I will commit your oozie patch. I think it will not help and let oozie fail differently. 
Hope you are able to reproduce that as well.


> 
> Failures:
> 
> - ambari (uses x86-64 nodejs binaries)

noteable it is phantomjs. You may use the workaround from tez in 1.2.1 (which I removed for 1.3) to solve ths issue.

> - bigtop-jsvc ("Java Home not defined")
I will propose to remove that because it is supplied by every linux distro. Was there because of centos-6


> - bigtop-tomcat (have to check)
ok? Will propose to update to 6.0.52

> - datafu (no gradle binary)
???

> - hbase (something about org.asciidoctor)
It is fixed with 1.3. We had old versions in nexus.

> - kafka (no gradle binary)
??
> - spark (no JAVA_HOME)
??
> - tez (x86-64 nodejs binaries)
Solved in 1.3


> Will take a look at failures. Also plan to do Bigtop 'master' build on
> fresh Debian 9 'stretch' aarch64 system to catch some bugs and to check
> existing fixes.
> 

Do you have a stable docker for aarch64 ? This is a prerequiste for advancing with bigtop.

> Sorry for short mail without many details but it is 22:00 here and I do
> that only because Olaf is online and updates Jira so my MiBand 2 thing
> interrupted watching some TV series ;D
> 

Olaf

Re: Building 1.2.1 on aarch64 platform

[Marcin Juszkiewicz <ma...@linaro.org>]

W dniu 04.12.2017 o 22:00, Marcin Juszkiewicz pisze:

Forgot ". /etc/profile" to get new PATH...

> Failures:
> 
> - ambari (uses x86-64 nodejs binaries)

> - bigtop-jsvc ("Java Home not defined")

built

> - bigtop-tomcat (have to check)

built

> - datafu (no gradle binary)

built