You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Roman Shaposhnik <ro...@shaposhnik.org> on 2016/04/13 23:39:21 UTC
ARM slave is now available
Hi!
as part of me working with Linaro on ODPi we now
have access to an ARM jenkins slave:
https://ci.bigtop.apache.org/computer/docker-slave-arm-1/
Let me know if you'd be interested in providing Bigtop
support for ARM. For now my plans are to hand build
a Docker build slave environment and just get the basic
build going as a POC.
Thanks,
Roman.
P.S. Now with ARM, PPC and x86 the only thing missing
is SPARC ;-)
Re: ARM slave is now available
Posted by "MrAsanjar ." <af...@gmail.com>.
No worries cos.
On Apr 13, 2016 10:34 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
> Sorry, I re-read it later and realized that my comment wasn't suitable. I
> am
> teared between six different tasks and my work-day is now at 13 hours mark
> ;(
>
> Cheers,
> Cos
>
> On Wed, Apr 13, 2016 at 10:03PM, MrAsanjar . wrote:
> > Cos, that meant as a joke. Beside SPARC, systemZ was the only platform
> > not meantioned by Roman.
> > On Apr 13, 2016 5:47 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
> >
> > > On Wed, Apr 13, 2016 at 04:47PM, MrAsanjar . wrote:
> > > > Good news, but don't forget SystemZ support :)
> > >
> > > Do we have anyone committing to support SystemZ? And computing
> resources to
> > > carry on the work? Unless it is on ci.bigtop.apache.org - it isn't
> > > officially
> > > supported ;)
> > >
> > > > btw, you will need to create docker puppet image before slave image.
> > > >
> > >
> https://github.com/apache/bigtop/blob/master/docker/bigtop-slaves/ubuntu-15.04-ppc64le/Dockerfile
> > > >
> > > > On Wed, Apr 13, 2016 at 4:39 PM, Roman Shaposhnik <
> roman@shaposhnik.org>
> > > > wrote:
> > > >
> > > > > Hi!
> > > > >
> > > > > as part of me working with Linaro on ODPi we now
> > > > > have access to an ARM jenkins slave:
> > > > > https://ci.bigtop.apache.org/computer/docker-slave-arm-1/
> > > > >
> > > > > Let me know if you'd be interested in providing Bigtop
> > > > > support for ARM. For now my plans are to hand build
> > > > > a Docker build slave environment and just get the basic
> > > > > build going as a POC.
> > > > >
> > > > > Thanks,
> > > > > Roman.
> > > > >
> > > > > P.S. Now with ARM, PPC and x86 the only thing missing
> > > > > is SPARC ;-)
> > > > >
> > >
>
Re: ARM slave is now available
Posted by Konstantin Boudnik <co...@apache.org>.
Sorry, I re-read it later and realized that my comment wasn't suitable. I am
teared between six different tasks and my work-day is now at 13 hours mark ;(
Cheers,
Cos
On Wed, Apr 13, 2016 at 10:03PM, MrAsanjar . wrote:
> Cos, that meant as a joke. Beside SPARC, systemZ was the only platform
> not meantioned by Roman.
> On Apr 13, 2016 5:47 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
>
> > On Wed, Apr 13, 2016 at 04:47PM, MrAsanjar . wrote:
> > > Good news, but don't forget SystemZ support :)
> >
> > Do we have anyone committing to support SystemZ? And computing resources to
> > carry on the work? Unless it is on ci.bigtop.apache.org - it isn't
> > officially
> > supported ;)
> >
> > > btw, you will need to create docker puppet image before slave image.
> > >
> > https://github.com/apache/bigtop/blob/master/docker/bigtop-slaves/ubuntu-15.04-ppc64le/Dockerfile
> > >
> > > On Wed, Apr 13, 2016 at 4:39 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> > > wrote:
> > >
> > > > Hi!
> > > >
> > > > as part of me working with Linaro on ODPi we now
> > > > have access to an ARM jenkins slave:
> > > > https://ci.bigtop.apache.org/computer/docker-slave-arm-1/
> > > >
> > > > Let me know if you'd be interested in providing Bigtop
> > > > support for ARM. For now my plans are to hand build
> > > > a Docker build slave environment and just get the basic
> > > > build going as a POC.
> > > >
> > > > Thanks,
> > > > Roman.
> > > >
> > > > P.S. Now with ARM, PPC and x86 the only thing missing
> > > > is SPARC ;-)
> > > >
> >
Re: ARM slave is now available
Posted by "MrAsanjar ." <af...@gmail.com>.
Cos, that meant as a joke. Beside SPARC, systemZ was the only platform
not meantioned by Roman.
On Apr 13, 2016 5:47 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
> On Wed, Apr 13, 2016 at 04:47PM, MrAsanjar . wrote:
> > Good news, but don't forget SystemZ support :)
>
> Do we have anyone committing to support SystemZ? And computing resources to
> carry on the work? Unless it is on ci.bigtop.apache.org - it isn't
> officially
> supported ;)
>
> > btw, you will need to create docker puppet image before slave image.
> >
> https://github.com/apache/bigtop/blob/master/docker/bigtop-slaves/ubuntu-15.04-ppc64le/Dockerfile
> >
> > On Wed, Apr 13, 2016 at 4:39 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> > wrote:
> >
> > > Hi!
> > >
> > > as part of me working with Linaro on ODPi we now
> > > have access to an ARM jenkins slave:
> > > https://ci.bigtop.apache.org/computer/docker-slave-arm-1/
> > >
> > > Let me know if you'd be interested in providing Bigtop
> > > support for ARM. For now my plans are to hand build
> > > a Docker build slave environment and just get the basic
> > > build going as a POC.
> > >
> > > Thanks,
> > > Roman.
> > >
> > > P.S. Now with ARM, PPC and x86 the only thing missing
> > > is SPARC ;-)
> > >
>
Re: ARM slave is now available
Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, Apr 13, 2016 at 04:47PM, MrAsanjar . wrote:
> Good news, but don't forget SystemZ support :)
Do we have anyone committing to support SystemZ? And computing resources to
carry on the work? Unless it is on ci.bigtop.apache.org - it isn't officially
supported ;)
> btw, you will need to create docker puppet image before slave image.
> https://github.com/apache/bigtop/blob/master/docker/bigtop-slaves/ubuntu-15.04-ppc64le/Dockerfile
>
> On Wed, Apr 13, 2016 at 4:39 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> wrote:
>
> > Hi!
> >
> > as part of me working with Linaro on ODPi we now
> > have access to an ARM jenkins slave:
> > https://ci.bigtop.apache.org/computer/docker-slave-arm-1/
> >
> > Let me know if you'd be interested in providing Bigtop
> > support for ARM. For now my plans are to hand build
> > a Docker build slave environment and just get the basic
> > build going as a POC.
> >
> > Thanks,
> > Roman.
> >
> > P.S. Now with ARM, PPC and x86 the only thing missing
> > is SPARC ;-)
> >
Re: ARM slave is now available
Posted by "MrAsanjar ." <af...@gmail.com>.
Good news, but don't forget SystemZ support :)
btw, you will need to create docker puppet image before slave image.
https://github.com/apache/bigtop/blob/master/docker/bigtop-slaves/ubuntu-15.04-ppc64le/Dockerfile
On Wed, Apr 13, 2016 at 4:39 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> Hi!
>
> as part of me working with Linaro on ODPi we now
> have access to an ARM jenkins slave:
> https://ci.bigtop.apache.org/computer/docker-slave-arm-1/
>
> Let me know if you'd be interested in providing Bigtop
> support for ARM. For now my plans are to hand build
> a Docker build slave environment and just get the basic
> build going as a POC.
>
> Thanks,
> Roman.
>
> P.S. Now with ARM, PPC and x86 the only thing missing
> is SPARC ;-)
>
Re: ARM slave is now available
Posted by Konstantin Boudnik <co...@apache.org>.
On Thu, Apr 14, 2016 at 07:00PM, Olaf Flebbe wrote:
> Hi,
>
> Let me welcome ARM aarch64 and many thanks to Linaro and Roman!
>
> My firm conviction is that compiling on multiple compilers/archs is very
> helpful to locate hidden bugs. That's the point why I pushed the POWER port.
>
> "Boldly linking where no man compiled before".
>
> And it showed up the unpleaseant side of hadoop: For instance the leveldb
> jni code hidden in Yarn. We have to fix it somehow ... Amir Sanjar (and me)
> is working on this, but there is only a very unclean solution right now.
>
> And let me add that IBM added two other POWER (ppc64le) machines to our
> pool. Two large ones, to replace the unstable rather small lease we had
> before. We have to thank Amir for providing us with it !
That's superbly great news! Looks like they aren't deployed with a slave,
right? Nonetheless, like was said the other day: Z and Spark are the only one
left now ;)
Cos
> > Am 13.04.2016 um 23:39 schrieb Roman Shaposhnik <ro...@shaposhnik.org>:
> >
> > Hi!
> >
> > as part of me working with Linaro on ODPi we now
> > have access to an ARM jenkins slave:
> > https://ci.bigtop.apache.org/computer/docker-slave-arm-1/
> >
> > Let me know if you'd be interested in providing Bigtop
> > support for ARM. For now my plans are to hand build
> > a Docker build slave environment and just get the basic
> > build going as a POC.
> >
> > Thanks,
> > Roman.
> >
> > P.S. Now with ARM, PPC and x86 the only thing missing
> > is SPARC ;-)
>
Re: ARM slave is now available
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Sun, Apr 24, 2016 at 12:34 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
>
>> Am 24.04.2016 um 08:07 schrieb Roman Shaposhnik <ro...@shaposhnik.org>:
>>
>> On Tue, Apr 19, 2016 at 2:40 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
>>> Hi,
>>>
>>> I changed the slave to only run bound jobs and changed binding from ppc64le to aarch64. Now it runs.
>>
>> Forgot to say: thank you for unlocking it! The job is green now. My only
>> question is this: for this type of one-off, how would you guys prefer
>> I handle pushing the image back to Docker hub? A special job?
>
> Hi Roman,
>
> That's what the *-push jobs are for .The missing bit is the dockerhub password.
> It's in the password file I sent you before. I can resend it to you, just in case you haven't it handy.
No, I got it. I've done login manually on the slave once:
$ [jenkins@odpi-ci ~]$ docker login
Username: bigtop
Password:
Email: private@bigtop.apache.org
WARNING: login credentials saved in /home/jenkins/.docker/config.json
Login Succeeded
But now I'm running into weird issues with the push:
https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-Trunk-push-aarch64/2/console
At first I thought it was broken credentials, but it appears this may
be some sort of a bug in Docker:
https://github.com/docker/distribution/issues/1165
Anybody seen anything like that?
> I do not like that we now have to triple all jobs platforms (amd64 / aarch64 / ppc64le ).
> We may restructure them to one job with an additional dimension "processor architecture"
> or "ABI" or "platform" .
Right. So what about making one matrix job for that? The matrix will be sparse
(since we don't currently suppose anything but Ubuntu 14.04 on ARM for example)
but perhaps easier to manage?
Thanks,
Roman.
Re: ARM slave is now available
Posted by Olaf Flebbe <of...@oflebbe.de>.
> Am 24.04.2016 um 08:07 schrieb Roman Shaposhnik <ro...@shaposhnik.org>:
>
> On Tue, Apr 19, 2016 at 2:40 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
>> Hi,
>>
>> I changed the slave to only run bound jobs and changed binding from ppc64le to aarch64. Now it runs.
>
> Forgot to say: thank you for unlocking it! The job is green now. My only
> question is this: for this type of one-off, how would you guys prefer
> I handle pushing the image back to Docker hub? A special job?
Hi Roman,
That's what the *-push jobs are for .The missing bit is the dockerhub password. It's in the password file I sent you before. I can resend it to you, just in case you haven't it handy.
I do not like that we now have to triple all jobs platforms (amd64 / aarch64 / ppc64le ). We may restructure them to one job with an additional dimension "processor architecture" or "ABI" or "platform" .
Olaf
>
> Thanks,
> Roman.
Re: ARM slave is now available
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Tue, Apr 19, 2016 at 2:40 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
> Hi,
>
> I changed the slave to only run bound jobs and changed binding from ppc64le to aarch64. Now it runs.
Forgot to say: thank you for unlocking it! The job is green now. My only
question is this: for this type of one-off, how would you guys prefer
I handle pushing the image back to Docker hub? A special job?
Thanks,
Roman.
Re: ARM slave is now available
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi,
I changed the slave to only run bound jobs and changed binding from ppc64le to aarch64. Now it runs.
Olaf
> Am 19.04.2016 um 04:42 schrieb Roman Shaposhnik <ro...@shaposhnik.org>:
>
> On Thu, Apr 14, 2016 at 10:00 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
>> Hi,
>>
>> Let me welcome ARM aarch64 and many thanks to Linaro and Roman!
>>
>> My firm conviction is that compiling on multiple compilers/archs is very helpful to locate hidden bugs. That's the point why I pushed the POWER port.
>>
>> "Boldly linking where no man compiled before".
>
> Indeed! Speaking of which, I've committed the ARM Docker slave update,
> but the job seems to be stuck:
> https://ci.bigtop.apache.org/job/Docker-Toolchain-Trunk-aarch64/
> Can somebody with more Jenkins foo than I have take a look and suggest
> what I may be doing wrong?
>
>> And it showed up the unpleaseant side of hadoop: For instance the leveldb jni code hidden in Yarn.
>> We have to fix it somehow ... Amir Sanjar (and me) is working on this, but there is only a very
>> unclean solution right now.
>
> ARM is already proving quite interesting. For example, jsvc build
> doesn't seem to recognize
> it as a valid target. And I'm sure I'm just scratching the surface there.
>
> Actually, speaking of jsvc -- modern HDFS doesn't really need it,
> right? Or am I thinking
> about Hadoop 3 vs Hadoop 2.x?
>
> There's still one last glitch left to resolve with the slaves, after
> which I'll add the
> usual bigtop keys for all of us to enjoy ;-)
>
>> And let me add that IBM added two other POWER (ppc64le) machines to our pool.
>> Two large ones, to replace the unstable rather small lease we had before. We have to thank
>> Amir for providing us with it !
>
> Great! Many thanks!
>
> Roman.
Re: ARM slave is now available
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Thu, Apr 14, 2016 at 10:00 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
> Hi,
>
> Let me welcome ARM aarch64 and many thanks to Linaro and Roman!
>
> My firm conviction is that compiling on multiple compilers/archs is very helpful to locate hidden bugs. That's the point why I pushed the POWER port.
>
> "Boldly linking where no man compiled before".
Indeed! Speaking of which, I've committed the ARM Docker slave update,
but the job seems to be stuck:
https://ci.bigtop.apache.org/job/Docker-Toolchain-Trunk-aarch64/
Can somebody with more Jenkins foo than I have take a look and suggest
what I may be doing wrong?
> And it showed up the unpleaseant side of hadoop: For instance the leveldb jni code hidden in Yarn.
> We have to fix it somehow ... Amir Sanjar (and me) is working on this, but there is only a very
> unclean solution right now.
ARM is already proving quite interesting. For example, jsvc build
doesn't seem to recognize
it as a valid target. And I'm sure I'm just scratching the surface there.
Actually, speaking of jsvc -- modern HDFS doesn't really need it,
right? Or am I thinking
about Hadoop 3 vs Hadoop 2.x?
There's still one last glitch left to resolve with the slaves, after
which I'll add the
usual bigtop keys for all of us to enjoy ;-)
> And let me add that IBM added two other POWER (ppc64le) machines to our pool.
> Two large ones, to replace the unstable rather small lease we had before. We have to thank
> Amir for providing us with it !
Great! Many thanks!
Roman.
Re: ARM slave is now available
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi,
Let me welcome ARM aarch64 and many thanks to Linaro and Roman!
My firm conviction is that compiling on multiple compilers/archs is very helpful to locate hidden bugs. That's the point why I pushed the POWER port.
"Boldly linking where no man compiled before".
And it showed up the unpleaseant side of hadoop: For instance the leveldb jni code hidden in Yarn. We have to fix it somehow ... Amir Sanjar (and me) is working on this, but there is only a very unclean solution right now.
And let me add that IBM added two other POWER (ppc64le) machines to our pool. Two large ones, to replace the unstable rather small lease we had before. We have to thank Amir for providing us with it !
Cheers,
Olaf
> Am 13.04.2016 um 23:39 schrieb Roman Shaposhnik <ro...@shaposhnik.org>:
>
> Hi!
>
> as part of me working with Linaro on ODPi we now
> have access to an ARM jenkins slave:
> https://ci.bigtop.apache.org/computer/docker-slave-arm-1/
>
> Let me know if you'd be interested in providing Bigtop
> support for ARM. For now my plans are to hand build
> a Docker build slave environment and just get the basic
> build going as a POC.
>
> Thanks,
> Roman.
>
> P.S. Now with ARM, PPC and x86 the only thing missing
> is SPARC ;-)