aarch64 binary convenience artifacts for Bigtop 1.2.0

[Roman Shaposhnik <ro...@shaposhnik.org>]

Hi!

as you saw I staged all the convenience artifacts for
the release except for the aarch64. That is because
the build slave is highly unstable at this point and I can't
get a build at all. The symptoms range from docker daemon
hanging all the time to the writes failing.

I'm starting to suspect that there's something wonky going
on with the filesystem.

Now, my assumption is that this is a VM. As such a good
option could be simply to recreate it from scratch and
assign ample storage and CPU resources to it.

Ganesh, Jun, would something like this be possible in the
next few days?

Thanks,
Roman.

Re: aarch64 binary convenience artifacts for Bigtop 1.2.0

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Thu, Mar 30, 2017 at 8:35 AM,  <ga...@linaro.org> wrote:
> Roman,
>
>
>
> Absolutely. I can get that done asap.
>
> Also we would want to look at the cause itself, so we will be trying to do
> that as well.

Great. You can see one of the leading causes of instability right here:
    https://ci.bigtop.apache.org/job/Bigtop-trunk-packages-by-jenkins/COMPONENTS=bigtop-jsvc,OS=ubuntu-16.04-aarch64/27/console
As you can see the build is hanging on trying to either download or
write to disk. It has been hanging since yesterday.

Btw, the slave I'm talking about is: 64.28.99.98

Also, the Docker image is available on GitHub if you're curious
trunk-ubuntu-16.04-aarch64:
    https://hub.docker.com/r/bigtop/slaves/tags/

> And thanks for your efforts as well.

Thank you guys for all the help!

Roman.

RE: aarch64 binary convenience artifacts for Bigtop 1.2.0

[Eugene Xie <Eu...@arm.com>]

Hi, Roman,

Here comes some good news -- although a bit late. I think I've figured out why the docker daemon hang.

The reason is docker runs in 'bridge' networking mode by default which has some weird networking issue. The workaround is straight forward. I specified the docker container to run in 'host' working mode and the problem disappeared.  I managed to build 30+ components in Dev Cloud instance using the docker container.

This is how I ran the container:
docker run --net=host -v `pwd`:/ws bigtop/slaves:trunk-ubuntu-16.04-aarch64 bash -l -c 'cd /ws ; ./gradlew --continue apex-deb bigtop-groovy-deb bigtop-jsvc-deb bigtop-utils-deb crunch-deb datafu-deb flink-deb flume-deb zookeeper-deb'

Why the 'bridge' mode is failing remains unanswered/unresolved. Some of our engineer is working on that.

I suppose you can resume the work of setting up bigtop CI for AArch64 on Linaro Dev. Cloud now. Ganesh has created Dev. Cloud instances for you for that purpose. Please feel free to let us know if the workaround I provided is OK for you.

Please refer to the comments I made to Linaro bug system https://bugs.linaro.org/show_bug.cgi?id=2960 for more detailed info.


Best Regards,
Eugene

-----Original Message-----
From: ganesh.raju@linaro.org [mailto:ganesh.raju@linaro.org]
Sent: 2017年3月30日 23:35
To: Roman Shaposhnik <ro...@shaposhnik.org>; dev@bigtop.apache.org
Cc: Jun He <ju...@linaro.org>
Subject: RE: aarch64 binary convenience artifacts for Bigtop 1.2.0

Roman,

Absolutely. I can get that done asap.
Also we would want to look at the cause itself, so we will be trying to do that as well.
And thanks for your efforts as well.

Thanks,
Ganesh

Sent from Mail for Windows 10

From: Roman Shaposhnik
Sent: Thursday, March 30, 2017 10:23 AM
To: dev@bigtop.apache.org
Cc: Ganesh Raju; Jun He
Subject: aarch64 binary convenience artifacts for Bigtop 1.2.0

Hi!

as you saw I staged all the convenience artifacts for the release except for the aarch64. That is because the build slave is highly unstable at this point and I can't get a build at all. The symptoms range from docker daemon hanging all the time to the writes failing.

I'm starting to suspect that there's something wonky going on with the filesystem.

Now, my assumption is that this is a VM. As such a good option could be simply to recreate it from scratch and assign ample storage and CPU resources to it.

Ganesh, Jun, would something like this be possible in the next few days?

Thanks,
Roman.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

RE: aarch64 binary convenience artifacts for Bigtop 1.2.0

[ga...@linaro.org]

Roman,

Absolutely. I can get that done asap.
Also we would want to look at the cause itself, so we will be trying to do that as well.
And thanks for your efforts as well.

Thanks,
Ganesh

Sent from Mail for Windows 10

From: Roman Shaposhnik
Sent: Thursday, March 30, 2017 10:23 AM
To: dev@bigtop.apache.org
Cc: Ganesh Raju; Jun He
Subject: aarch64 binary convenience artifacts for Bigtop 1.2.0

Hi!

as you saw I staged all the convenience artifacts for
the release except for the aarch64. That is because
the build slave is highly unstable at this point and I can't
get a build at all. The symptoms range from docker daemon
hanging all the time to the writes failing.

I'm starting to suspect that there's something wonky going
on with the filesystem.

Now, my assumption is that this is a VM. As such a good
option could be simply to recreate it from scratch and
assign ample storage and CPU resources to it.

Ganesh, Jun, would something like this be possible in the
next few days?

Thanks,
Roman.