You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Evans Ye <ev...@apache.org> on 2015/04/27 20:25:42 UTC
Re: Bigtop CI [Re: release 1.0.0]
Hi folks,
Allow me to report some progress on our CI settings for 1.0 release.
(nate@reactor8.com asked me about the status and would like to help
but sorry I didn't respond you well back then)
For 1.0 to release, we need to build bigtop by those supported OS listed in
BIGTOP-1480.
And I plan to use use docker images to build 1.0 release directly so that
we don't need to setup jenkins slaves one-by-one. (the fastest way I can
think of for speeding up 1.0 release)
My plan is to prepare bigtop/puppet images by ourselves
<https://github.com/evans-ye/bigtop-dockerfiles>and apply bigtop_toolchain
by jenkins
<http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/>to get
bigtop/slaves images.
I've built and tested some of the OS in bigtop/puppet family (stored
as evansye/puppet
<https://registry.hub.docker.com/u/evansye/puppet/tags/manage/>temporary),
but some of the OS do not work properly since I'm lacking of knowledge on
them.
***worked***
centos-6
centos-7
ubuntu-14.04 (unchanged)
fedora-20 (unchanged)
***not working***
opensuse-13.2
---> I can't have "hostname" package installed, it seems that is needed by
puppet to determine what node to apply
debian 8
---> no wget and curl available
[root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c 'apt-get -y install
curl wget'
Reading package lists... Done
Building dependency tree... Done
E: Unable to locate package curl
E: Unable to locate package wget
Todo list:
1. It would be a great help if opensuse and debian expert can come to help
on those issues.
2. I'd like to have the credential for bigtop dockerhub so that I can put
new OS images on it.
3. Have all package in trunk built by new Docker slaves
Maybe Roman can help me out on *#2. :)*
You can enclose me the credential by my pgp:
curl http://people.apache.org/~evansye/evansye.pgp
That it. Suggestions are welcome!
Thanks,
Evans
2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> Currently all the slaves shut-down: looks like the instances have been
> turned
> off as I can not relaunch the slaves. Will try to restart them tonight
> once I
> am on machine with AWS keys.
>
> Cos
>
> On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> > Guys,
> >
> > I want to start a separate thread to track the CI preparations for the
> release
> > next month (fingers crossed). Clearly, we can make a release without CI,
> but
> > it'd way easier to test and create binary artifacts if we have a working
> > environment for official validation. Roman has done a lot in this
> direction
> > (many thanks!), but there are still a few rough edges, which might be
> easy to
> > finish of.
> >
> > I want to figure out a couple of things:
> > - what's the state of CI and how much still needs to be done (Rvs?
> Could you
> > share any first hand feedback?)
> > - who would be able to help with the CI completion? I can commit some
> of my
> > cycles, but it'd be great to have few more hands on that. Clearly,
> some
> > Jenkins-foo and prior CI skills won't hurt ;)
> >
> > Please chime in if you can help. Thanks a lot!
> > Cos
> >
>
Re: Bigtop CI [Re: release 1.0.0]
Posted by jay vyas <ja...@gmail.com>.
Whoa ! This is cool ! so much green !
On Mon, May 25, 2015 at 1:37 PM, Evans Ye <ev...@apache.org> wrote:
> Having jenkins master migrated to new EC2 instance would need to have a
> backup of config.xml and jobs configuration. I can help to setup a new
> jenkins but a backup configuration is needed. Unfortunately, I don't have
> credential to login the current master and fetch a backup, do you have any
> suggestion, cos?
>
> 2015-05-25 18:14 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>
> > Hi Evans.
> >
> > This is great! Thank you very much for putting this together in time for
> > 1.0
> > release! Ignite rc7 should get into the Incubator vote today, so in a
> > could of
> > days it will be ready. I will make a patch for the JIRA to fix it as
> well.
> >
> > I think the next step would be to move the master to an instance on new
> EC2
> > account, followed by a request to INFRA to setup new
> ci.bigtop.apache.org
> > CNAME to point to that instance.
> >
> > I guess during that transition we'd need to clean up the old builds. For
> > Debian and Suse: Olaf and Peter would be the experts in these. Guys,
> could
> > you
> > please chime in?
> >
> > I will try to have RC1 this week (pending the Ignite vote) before
> > switching off
> > for a couple weeks of vacation.
> >
> > Thanks again!
> > Cos
> > On Thu, May 21, 2015 at 12:22AM, Evans Ye wrote:
> > > Folks,
> > >
> > > News on our Docker based CI.
> > >
> > > We now have nightly build for all the bigtop components available at
> > > http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/
> > > And it's all green except the ignite-hadoop build.
> > >
> > > Those Docker prefixed jobs will run daily and fetch the newest code
> from
> > > master to build.
> > > This can give us a clear view to the status of Bigtop master code
> > > repository.
> > > For example, we can quickly know which commit breaks the build.
> > >
> > > What underlying supporting this feature are 4 EMR team donated EC2
> > > instances and my docker images
> > > <https://github.com/evans-ye/bigtop-dockerfiles> for each Bigtop 1.0
> > > targeted OS.
> > >
> > > To be notice that we only have centos 6, centos 7, ubuntu 14.04, and
> > fedora
> > > 20 builds available now, opensuse 13.2 and debian 8 are still not ready
> > > yet. As I mentioned in my previous mail, it would be great if someone
> > > familiar with the OS can help. :)
> > >
> > > Let's get back to the bigtop 1.0 release, with most of the CI
> > > infrastructure(slaves, images) are available and all the JIRAs that
> need
> > to
> > > be fixed in 1.0 are addressed, I believe we're close to the release.
> > > Now that the only blocker is ignite 1.1 (BIGTOP-1833
> > > <https://issues.apache.org/jira/browse/BIGTOP-1833>), and the opensuse
> > 13.2
> > > and debian 8 images.
> > >
> > > The next step of our CI improvement is to setup docker-provisioner and
> > runs
> > > smoke-tests to evaluate the system interoperability. I'll continue
> > working
> > > down this path.
> > >
> > > Anyone interesting in this or has any suggestion is welcomed as always.
> > >
> > >
> > > 2015-04-28 14:15 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >
> > > > ah, got it. Thanks for the explanations and all the work!
> > > >
> > > > On Tue, Apr 28, 2015 at 02:07PM, Evans Ye wrote:
> > > > > Hey Cos,
> > > > >
> > > > > I build new images only if needed.
> > > > >
> > > > > In 0.8, we have:
> > > > >
> > > > > centos-6.4
> > > > > fedora-20
> > > > > ubuntu-12.04
> > > > > ubuntu-14.04
> > > > > opensuse-12.3
> > > > >
> > > > > Compare with the 1.0 BOM:
> > > > >
> > > > > OS:
> > > > > CentOS6, CentOS7, Fedora 20
> > > > > SLES 11SP3, SLES12, OpenSUSE 13.2
> > > > > Ubuntu 14.04 LTS, Debian 8
> > > > >
> > > > >
> > > > > fedora-20 and ubuntu-14.04 are still roman's images in 1.0.
> > > > >
> > > > > But centos-6.4 is quite old, hence I build a new one based on
> > centos-6.6.
> > > > >
> > > > > centos-7 and debian are new comers in 1.0, so we have to create
> them
> > from
> > > > > scratch.
> > > > >
> > > > > opensuse has been upgraded from 12.3 to 13.2. We need a new image
> to
> > be
> > > > > built as well.
> > > > >
> > > > > Evans
> > > > >
> > > > > 2015-04-28 3:04 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > > > >
> > > > > > That's great! Thank you Evans!
> > > > > >
> > > > > > I am sure Peter Linnell should be able to give you a hand with
> > *Suse
> > > > > > platform.
> > > > > > Olaf, could you please help Evans with Debian?
> > > > > >
> > > > > > One question about dockers: any reason you've chosen not to use
> the
> > > > images
> > > > > > that Roman has put together initially? Not like I care one way or
> > > > another,
> > > > > > but
> > > > > > I am using those at the moment and want to make sure that we are
> > > > building
> > > > > > in
> > > > > > the same environment across the board.
> > > > > >
> > > > > > Also, per Nate's request I have added S3 policy to the bigtop-ci
> > group
> > > > on
> > > > > > AWS,
> > > > > > so you guys should be able to use S3 if needed.
> > > > > >
> > > > > > Thanks,
> > > > > > Cos
> > > > > >
> > > > > > On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote:
> > > > > > > Hi folks,
> > > > > > >
> > > > > > > Allow me to report some progress on our CI settings for 1.0
> > release.
> > > > > > > (nate@reactor8.com asked me about the status and would like to
> > help
> > > > > > > but sorry I didn't respond you well back then)
> > > > > > >
> > > > > > >
> > > > > > > For 1.0 to release, we need to build bigtop by those supported
> OS
> > > > listed
> > > > > > in
> > > > > > > BIGTOP-1480.
> > > > > > >
> > > > > > > And I plan to use use docker images to build 1.0 release
> > directly so
> > > > that
> > > > > > > we don't need to setup jenkins slaves one-by-one. (the fastest
> > way I
> > > > can
> > > > > > > think of for speeding up 1.0 release)
> > > > > > >
> > > > > > > My plan is to prepare bigtop/puppet images by ourselves
> > > > > > > <https://github.com/evans-ye/bigtop-dockerfiles>and apply
> > > > > > bigtop_toolchain
> > > > > > > by jenkins
> > > > > > > <
> > http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/
> > > > >to
> > > > > > get
> > > > > > > bigtop/slaves images.
> > > > > > >
> > > > > > > I've built and tested some of the OS in bigtop/puppet family
> > (stored
> > > > > > > as evansye/puppet
> > > > > > > <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/
> > > > > > >temporary),
> > > > > > > but some of the OS do not work properly since I'm lacking of
> > > > knowledge on
> > > > > > > them.
> > > > > > >
> > > > > > > ***worked***
> > > > > > >
> > > > > > > centos-6
> > > > > > > centos-7
> > > > > > > ubuntu-14.04 (unchanged)
> > > > > > > fedora-20 (unchanged)
> > > > > > >
> > > > > > > ***not working***
> > > > > > > opensuse-13.2
> > > > > > > ---> I can't have "hostname" package installed, it seems that
> is
> > > > needed
> > > > > > by
> > > > > > > puppet to determine what node to apply
> > > > > > > debian 8
> > > > > > > ---> no wget and curl available
> > > > > > > [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c
> > 'apt-get -y
> > > > > > install
> > > > > > > curl wget'
> > > > > > > Reading package lists... Done
> > > > > > > Building dependency tree... Done
> > > > > > > E: Unable to locate package curl
> > > > > > > E: Unable to locate package wget
> > > > > > >
> > > > > > >
> > > > > > > Todo list:
> > > > > > > 1. It would be a great help if opensuse and debian expert can
> > come to
> > > > > > help
> > > > > > > on those issues.
> > > > > > > 2. I'd like to have the credential for bigtop dockerhub so
> that I
> > > > can put
> > > > > > > new OS images on it.
> > > > > > > 3. Have all package in trunk built by new Docker slaves
> > > > > > >
> > > > > > > Maybe Roman can help me out on *#2. :)*
> > > > > > > You can enclose me the credential by my pgp:
> > > > > > > curl http://people.apache.org/~evansye/evansye.pgp
> > > > > > >
> > > > > > > That it. Suggestions are welcome!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Evans
> > > > > > >
> > > > > > > 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > > > > > >
> > > > > > > > Currently all the slaves shut-down: looks like the instances
> > have
> > > > been
> > > > > > > > turned
> > > > > > > > off as I can not relaunch the slaves. Will try to restart
> them
> > > > tonight
> > > > > > > > once I
> > > > > > > > am on machine with AWS keys.
> > > > > > > >
> > > > > > > > Cos
> > > > > > > >
> > > > > > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> > > > > > > > > Guys,
> > > > > > > > >
> > > > > > > > > I want to start a separate thread to track the CI
> > preparations
> > > > for
> > > > > > the
> > > > > > > > release
> > > > > > > > > next month (fingers crossed). Clearly, we can make a
> release
> > > > without
> > > > > > CI,
> > > > > > > > but
> > > > > > > > > it'd way easier to test and create binary artifacts if we
> > have a
> > > > > > working
> > > > > > > > > environment for official validation. Roman has done a lot
> in
> > this
> > > > > > > > direction
> > > > > > > > > (many thanks!), but there are still a few rough edges,
> which
> > > > might be
> > > > > > > > easy to
> > > > > > > > > finish of.
> > > > > > > > >
> > > > > > > > > I want to figure out a couple of things:
> > > > > > > > > - what's the state of CI and how much still needs to be
> done
> > > > (Rvs?
> > > > > > > > Could you
> > > > > > > > > share any first hand feedback?)
> > > > > > > > > - who would be able to help with the CI completion? I can
> > commit
> > > > > > some
> > > > > > > > of my
> > > > > > > > > cycles, but it'd be great to have few more hands on
> that.
> > > > Clearly,
> > > > > > > > some
> > > > > > > > > Jenkins-foo and prior CI skills won't hurt ;)
> > > > > > > > >
> > > > > > > > > Please chime in if you can help. Thanks a lot!
> > > > > > > > > Cos
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
>
--
jay vyas
Re: Bigtop CI [Re: release 1.0.0]
Posted by Roman Shaposhnik <rv...@apache.org>.
Let me get back to you on this tmr. Ok?
On Mon, May 25, 2015 at 1:36 PM, Konstantin Boudnik <co...@apache.org> wrote:
> On Tue, May 26, 2015 at 01:37AM, Evans Ye wrote:
>> Having jenkins master migrated to new EC2 instance would need to have a
>> backup of config.xml and jobs configuration. I can help to setup a new
>> jenkins but a backup configuration is needed. Unfortunately, I don't have
>> credential to login the current master and fetch a backup, do you have any
>> suggestion, cos?
>
> I know that Roman has all backups as he has dealt with the original setup and
> restored it a couple of times after recent crashes. I will ping him off-line
> to make sure he's getting the message ;)
>
> Thanks!
> Cos
>
>>
>> 2015-05-25 18:14 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>>
>> > Hi Evans.
>> >
>> > This is great! Thank you very much for putting this together in time for
>> > 1.0
>> > release! Ignite rc7 should get into the Incubator vote today, so in a
>> > could of
>> > days it will be ready. I will make a patch for the JIRA to fix it as well.
>> >
>> > I think the next step would be to move the master to an instance on new EC2
>> > account, followed by a request to INFRA to setup new ci.bigtop.apache.org
>> > CNAME to point to that instance.
>> >
>> > I guess during that transition we'd need to clean up the old builds. For
>> > Debian and Suse: Olaf and Peter would be the experts in these. Guys, could
>> > you
>> > please chime in?
>> >
>> > I will try to have RC1 this week (pending the Ignite vote) before
>> > switching off
>> > for a couple weeks of vacation.
>> >
>> > Thanks again!
>> > Cos
>> > On Thu, May 21, 2015 at 12:22AM, Evans Ye wrote:
>> > > Folks,
>> > >
>> > > News on our Docker based CI.
>> > >
>> > > We now have nightly build for all the bigtop components available at
>> > > http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/
>> > > And it's all green except the ignite-hadoop build.
>> > >
>> > > Those Docker prefixed jobs will run daily and fetch the newest code from
>> > > master to build.
>> > > This can give us a clear view to the status of Bigtop master code
>> > > repository.
>> > > For example, we can quickly know which commit breaks the build.
>> > >
>> > > What underlying supporting this feature are 4 EMR team donated EC2
>> > > instances and my docker images
>> > > <https://github.com/evans-ye/bigtop-dockerfiles> for each Bigtop 1.0
>> > > targeted OS.
>> > >
>> > > To be notice that we only have centos 6, centos 7, ubuntu 14.04, and
>> > fedora
>> > > 20 builds available now, opensuse 13.2 and debian 8 are still not ready
>> > > yet. As I mentioned in my previous mail, it would be great if someone
>> > > familiar with the OS can help. :)
>> > >
>> > > Let's get back to the bigtop 1.0 release, with most of the CI
>> > > infrastructure(slaves, images) are available and all the JIRAs that need
>> > to
>> > > be fixed in 1.0 are addressed, I believe we're close to the release.
>> > > Now that the only blocker is ignite 1.1 (BIGTOP-1833
>> > > <https://issues.apache.org/jira/browse/BIGTOP-1833>), and the opensuse
>> > 13.2
>> > > and debian 8 images.
>> > >
>> > > The next step of our CI improvement is to setup docker-provisioner and
>> > runs
>> > > smoke-tests to evaluate the system interoperability. I'll continue
>> > working
>> > > down this path.
>> > >
>> > > Anyone interesting in this or has any suggestion is welcomed as always.
>> > >
>> > >
>> > > 2015-04-28 14:15 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>> > >
>> > > > ah, got it. Thanks for the explanations and all the work!
>> > > >
>> > > > On Tue, Apr 28, 2015 at 02:07PM, Evans Ye wrote:
>> > > > > Hey Cos,
>> > > > >
>> > > > > I build new images only if needed.
>> > > > >
>> > > > > In 0.8, we have:
>> > > > >
>> > > > > centos-6.4
>> > > > > fedora-20
>> > > > > ubuntu-12.04
>> > > > > ubuntu-14.04
>> > > > > opensuse-12.3
>> > > > >
>> > > > > Compare with the 1.0 BOM:
>> > > > >
>> > > > > OS:
>> > > > > CentOS6, CentOS7, Fedora 20
>> > > > > SLES 11SP3, SLES12, OpenSUSE 13.2
>> > > > > Ubuntu 14.04 LTS, Debian 8
>> > > > >
>> > > > >
>> > > > > fedora-20 and ubuntu-14.04 are still roman's images in 1.0.
>> > > > >
>> > > > > But centos-6.4 is quite old, hence I build a new one based on
>> > centos-6.6.
>> > > > >
>> > > > > centos-7 and debian are new comers in 1.0, so we have to create them
>> > from
>> > > > > scratch.
>> > > > >
>> > > > > opensuse has been upgraded from 12.3 to 13.2. We need a new image to
>> > be
>> > > > > built as well.
>> > > > >
>> > > > > Evans
>> > > > >
>> > > > > 2015-04-28 3:04 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>> > > > >
>> > > > > > That's great! Thank you Evans!
>> > > > > >
>> > > > > > I am sure Peter Linnell should be able to give you a hand with
>> > *Suse
>> > > > > > platform.
>> > > > > > Olaf, could you please help Evans with Debian?
>> > > > > >
>> > > > > > One question about dockers: any reason you've chosen not to use the
>> > > > images
>> > > > > > that Roman has put together initially? Not like I care one way or
>> > > > another,
>> > > > > > but
>> > > > > > I am using those at the moment and want to make sure that we are
>> > > > building
>> > > > > > in
>> > > > > > the same environment across the board.
>> > > > > >
>> > > > > > Also, per Nate's request I have added S3 policy to the bigtop-ci
>> > group
>> > > > on
>> > > > > > AWS,
>> > > > > > so you guys should be able to use S3 if needed.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Cos
>> > > > > >
>> > > > > > On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote:
>> > > > > > > Hi folks,
>> > > > > > >
>> > > > > > > Allow me to report some progress on our CI settings for 1.0
>> > release.
>> > > > > > > (nate@reactor8.com asked me about the status and would like to
>> > help
>> > > > > > > but sorry I didn't respond you well back then)
>> > > > > > >
>> > > > > > >
>> > > > > > > For 1.0 to release, we need to build bigtop by those supported OS
>> > > > listed
>> > > > > > in
>> > > > > > > BIGTOP-1480.
>> > > > > > >
>> > > > > > > And I plan to use use docker images to build 1.0 release
>> > directly so
>> > > > that
>> > > > > > > we don't need to setup jenkins slaves one-by-one. (the fastest
>> > way I
>> > > > can
>> > > > > > > think of for speeding up 1.0 release)
>> > > > > > >
>> > > > > > > My plan is to prepare bigtop/puppet images by ourselves
>> > > > > > > <https://github.com/evans-ye/bigtop-dockerfiles>and apply
>> > > > > > bigtop_toolchain
>> > > > > > > by jenkins
>> > > > > > > <
>> > http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/
>> > > > >to
>> > > > > > get
>> > > > > > > bigtop/slaves images.
>> > > > > > >
>> > > > > > > I've built and tested some of the OS in bigtop/puppet family
>> > (stored
>> > > > > > > as evansye/puppet
>> > > > > > > <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/
>> > > > > > >temporary),
>> > > > > > > but some of the OS do not work properly since I'm lacking of
>> > > > knowledge on
>> > > > > > > them.
>> > > > > > >
>> > > > > > > ***worked***
>> > > > > > >
>> > > > > > > centos-6
>> > > > > > > centos-7
>> > > > > > > ubuntu-14.04 (unchanged)
>> > > > > > > fedora-20 (unchanged)
>> > > > > > >
>> > > > > > > ***not working***
>> > > > > > > opensuse-13.2
>> > > > > > > ---> I can't have "hostname" package installed, it seems that is
>> > > > needed
>> > > > > > by
>> > > > > > > puppet to determine what node to apply
>> > > > > > > debian 8
>> > > > > > > ---> no wget and curl available
>> > > > > > > [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c
>> > 'apt-get -y
>> > > > > > install
>> > > > > > > curl wget'
>> > > > > > > Reading package lists... Done
>> > > > > > > Building dependency tree... Done
>> > > > > > > E: Unable to locate package curl
>> > > > > > > E: Unable to locate package wget
>> > > > > > >
>> > > > > > >
>> > > > > > > Todo list:
>> > > > > > > 1. It would be a great help if opensuse and debian expert can
>> > come to
>> > > > > > help
>> > > > > > > on those issues.
>> > > > > > > 2. I'd like to have the credential for bigtop dockerhub so that I
>> > > > can put
>> > > > > > > new OS images on it.
>> > > > > > > 3. Have all package in trunk built by new Docker slaves
>> > > > > > >
>> > > > > > > Maybe Roman can help me out on *#2. :)*
>> > > > > > > You can enclose me the credential by my pgp:
>> > > > > > > curl http://people.apache.org/~evansye/evansye.pgp
>> > > > > > >
>> > > > > > > That it. Suggestions are welcome!
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Evans
>> > > > > > >
>> > > > > > > 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>> > > > > > >
>> > > > > > > > Currently all the slaves shut-down: looks like the instances
>> > have
>> > > > been
>> > > > > > > > turned
>> > > > > > > > off as I can not relaunch the slaves. Will try to restart them
>> > > > tonight
>> > > > > > > > once I
>> > > > > > > > am on machine with AWS keys.
>> > > > > > > >
>> > > > > > > > Cos
>> > > > > > > >
>> > > > > > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
>> > > > > > > > > Guys,
>> > > > > > > > >
>> > > > > > > > > I want to start a separate thread to track the CI
>> > preparations
>> > > > for
>> > > > > > the
>> > > > > > > > release
>> > > > > > > > > next month (fingers crossed). Clearly, we can make a release
>> > > > without
>> > > > > > CI,
>> > > > > > > > but
>> > > > > > > > > it'd way easier to test and create binary artifacts if we
>> > have a
>> > > > > > working
>> > > > > > > > > environment for official validation. Roman has done a lot in
>> > this
>> > > > > > > > direction
>> > > > > > > > > (many thanks!), but there are still a few rough edges, which
>> > > > might be
>> > > > > > > > easy to
>> > > > > > > > > finish of.
>> > > > > > > > >
>> > > > > > > > > I want to figure out a couple of things:
>> > > > > > > > > - what's the state of CI and how much still needs to be done
>> > > > (Rvs?
>> > > > > > > > Could you
>> > > > > > > > > share any first hand feedback?)
>> > > > > > > > > - who would be able to help with the CI completion? I can
>> > commit
>> > > > > > some
>> > > > > > > > of my
>> > > > > > > > > cycles, but it'd be great to have few more hands on that.
>> > > > Clearly,
>> > > > > > > > some
>> > > > > > > > > Jenkins-foo and prior CI skills won't hurt ;)
>> > > > > > > > >
>> > > > > > > > > Please chime in if you can help. Thanks a lot!
>> > > > > > > > > Cos
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > >
>> >
Re: Bigtop CI [Re: release 1.0.0]
Posted by Konstantin Boudnik <co...@apache.org>.
On Tue, May 26, 2015 at 01:37AM, Evans Ye wrote:
> Having jenkins master migrated to new EC2 instance would need to have a
> backup of config.xml and jobs configuration. I can help to setup a new
> jenkins but a backup configuration is needed. Unfortunately, I don't have
> credential to login the current master and fetch a backup, do you have any
> suggestion, cos?
I know that Roman has all backups as he has dealt with the original setup and
restored it a couple of times after recent crashes. I will ping him off-line
to make sure he's getting the message ;)
Thanks!
Cos
>
> 2015-05-25 18:14 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>
> > Hi Evans.
> >
> > This is great! Thank you very much for putting this together in time for
> > 1.0
> > release! Ignite rc7 should get into the Incubator vote today, so in a
> > could of
> > days it will be ready. I will make a patch for the JIRA to fix it as well.
> >
> > I think the next step would be to move the master to an instance on new EC2
> > account, followed by a request to INFRA to setup new ci.bigtop.apache.org
> > CNAME to point to that instance.
> >
> > I guess during that transition we'd need to clean up the old builds. For
> > Debian and Suse: Olaf and Peter would be the experts in these. Guys, could
> > you
> > please chime in?
> >
> > I will try to have RC1 this week (pending the Ignite vote) before
> > switching off
> > for a couple weeks of vacation.
> >
> > Thanks again!
> > Cos
> > On Thu, May 21, 2015 at 12:22AM, Evans Ye wrote:
> > > Folks,
> > >
> > > News on our Docker based CI.
> > >
> > > We now have nightly build for all the bigtop components available at
> > > http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/
> > > And it's all green except the ignite-hadoop build.
> > >
> > > Those Docker prefixed jobs will run daily and fetch the newest code from
> > > master to build.
> > > This can give us a clear view to the status of Bigtop master code
> > > repository.
> > > For example, we can quickly know which commit breaks the build.
> > >
> > > What underlying supporting this feature are 4 EMR team donated EC2
> > > instances and my docker images
> > > <https://github.com/evans-ye/bigtop-dockerfiles> for each Bigtop 1.0
> > > targeted OS.
> > >
> > > To be notice that we only have centos 6, centos 7, ubuntu 14.04, and
> > fedora
> > > 20 builds available now, opensuse 13.2 and debian 8 are still not ready
> > > yet. As I mentioned in my previous mail, it would be great if someone
> > > familiar with the OS can help. :)
> > >
> > > Let's get back to the bigtop 1.0 release, with most of the CI
> > > infrastructure(slaves, images) are available and all the JIRAs that need
> > to
> > > be fixed in 1.0 are addressed, I believe we're close to the release.
> > > Now that the only blocker is ignite 1.1 (BIGTOP-1833
> > > <https://issues.apache.org/jira/browse/BIGTOP-1833>), and the opensuse
> > 13.2
> > > and debian 8 images.
> > >
> > > The next step of our CI improvement is to setup docker-provisioner and
> > runs
> > > smoke-tests to evaluate the system interoperability. I'll continue
> > working
> > > down this path.
> > >
> > > Anyone interesting in this or has any suggestion is welcomed as always.
> > >
> > >
> > > 2015-04-28 14:15 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >
> > > > ah, got it. Thanks for the explanations and all the work!
> > > >
> > > > On Tue, Apr 28, 2015 at 02:07PM, Evans Ye wrote:
> > > > > Hey Cos,
> > > > >
> > > > > I build new images only if needed.
> > > > >
> > > > > In 0.8, we have:
> > > > >
> > > > > centos-6.4
> > > > > fedora-20
> > > > > ubuntu-12.04
> > > > > ubuntu-14.04
> > > > > opensuse-12.3
> > > > >
> > > > > Compare with the 1.0 BOM:
> > > > >
> > > > > OS:
> > > > > CentOS6, CentOS7, Fedora 20
> > > > > SLES 11SP3, SLES12, OpenSUSE 13.2
> > > > > Ubuntu 14.04 LTS, Debian 8
> > > > >
> > > > >
> > > > > fedora-20 and ubuntu-14.04 are still roman's images in 1.0.
> > > > >
> > > > > But centos-6.4 is quite old, hence I build a new one based on
> > centos-6.6.
> > > > >
> > > > > centos-7 and debian are new comers in 1.0, so we have to create them
> > from
> > > > > scratch.
> > > > >
> > > > > opensuse has been upgraded from 12.3 to 13.2. We need a new image to
> > be
> > > > > built as well.
> > > > >
> > > > > Evans
> > > > >
> > > > > 2015-04-28 3:04 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > > > >
> > > > > > That's great! Thank you Evans!
> > > > > >
> > > > > > I am sure Peter Linnell should be able to give you a hand with
> > *Suse
> > > > > > platform.
> > > > > > Olaf, could you please help Evans with Debian?
> > > > > >
> > > > > > One question about dockers: any reason you've chosen not to use the
> > > > images
> > > > > > that Roman has put together initially? Not like I care one way or
> > > > another,
> > > > > > but
> > > > > > I am using those at the moment and want to make sure that we are
> > > > building
> > > > > > in
> > > > > > the same environment across the board.
> > > > > >
> > > > > > Also, per Nate's request I have added S3 policy to the bigtop-ci
> > group
> > > > on
> > > > > > AWS,
> > > > > > so you guys should be able to use S3 if needed.
> > > > > >
> > > > > > Thanks,
> > > > > > Cos
> > > > > >
> > > > > > On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote:
> > > > > > > Hi folks,
> > > > > > >
> > > > > > > Allow me to report some progress on our CI settings for 1.0
> > release.
> > > > > > > (nate@reactor8.com asked me about the status and would like to
> > help
> > > > > > > but sorry I didn't respond you well back then)
> > > > > > >
> > > > > > >
> > > > > > > For 1.0 to release, we need to build bigtop by those supported OS
> > > > listed
> > > > > > in
> > > > > > > BIGTOP-1480.
> > > > > > >
> > > > > > > And I plan to use use docker images to build 1.0 release
> > directly so
> > > > that
> > > > > > > we don't need to setup jenkins slaves one-by-one. (the fastest
> > way I
> > > > can
> > > > > > > think of for speeding up 1.0 release)
> > > > > > >
> > > > > > > My plan is to prepare bigtop/puppet images by ourselves
> > > > > > > <https://github.com/evans-ye/bigtop-dockerfiles>and apply
> > > > > > bigtop_toolchain
> > > > > > > by jenkins
> > > > > > > <
> > http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/
> > > > >to
> > > > > > get
> > > > > > > bigtop/slaves images.
> > > > > > >
> > > > > > > I've built and tested some of the OS in bigtop/puppet family
> > (stored
> > > > > > > as evansye/puppet
> > > > > > > <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/
> > > > > > >temporary),
> > > > > > > but some of the OS do not work properly since I'm lacking of
> > > > knowledge on
> > > > > > > them.
> > > > > > >
> > > > > > > ***worked***
> > > > > > >
> > > > > > > centos-6
> > > > > > > centos-7
> > > > > > > ubuntu-14.04 (unchanged)
> > > > > > > fedora-20 (unchanged)
> > > > > > >
> > > > > > > ***not working***
> > > > > > > opensuse-13.2
> > > > > > > ---> I can't have "hostname" package installed, it seems that is
> > > > needed
> > > > > > by
> > > > > > > puppet to determine what node to apply
> > > > > > > debian 8
> > > > > > > ---> no wget and curl available
> > > > > > > [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c
> > 'apt-get -y
> > > > > > install
> > > > > > > curl wget'
> > > > > > > Reading package lists... Done
> > > > > > > Building dependency tree... Done
> > > > > > > E: Unable to locate package curl
> > > > > > > E: Unable to locate package wget
> > > > > > >
> > > > > > >
> > > > > > > Todo list:
> > > > > > > 1. It would be a great help if opensuse and debian expert can
> > come to
> > > > > > help
> > > > > > > on those issues.
> > > > > > > 2. I'd like to have the credential for bigtop dockerhub so that I
> > > > can put
> > > > > > > new OS images on it.
> > > > > > > 3. Have all package in trunk built by new Docker slaves
> > > > > > >
> > > > > > > Maybe Roman can help me out on *#2. :)*
> > > > > > > You can enclose me the credential by my pgp:
> > > > > > > curl http://people.apache.org/~evansye/evansye.pgp
> > > > > > >
> > > > > > > That it. Suggestions are welcome!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Evans
> > > > > > >
> > > > > > > 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > > > > > >
> > > > > > > > Currently all the slaves shut-down: looks like the instances
> > have
> > > > been
> > > > > > > > turned
> > > > > > > > off as I can not relaunch the slaves. Will try to restart them
> > > > tonight
> > > > > > > > once I
> > > > > > > > am on machine with AWS keys.
> > > > > > > >
> > > > > > > > Cos
> > > > > > > >
> > > > > > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> > > > > > > > > Guys,
> > > > > > > > >
> > > > > > > > > I want to start a separate thread to track the CI
> > preparations
> > > > for
> > > > > > the
> > > > > > > > release
> > > > > > > > > next month (fingers crossed). Clearly, we can make a release
> > > > without
> > > > > > CI,
> > > > > > > > but
> > > > > > > > > it'd way easier to test and create binary artifacts if we
> > have a
> > > > > > working
> > > > > > > > > environment for official validation. Roman has done a lot in
> > this
> > > > > > > > direction
> > > > > > > > > (many thanks!), but there are still a few rough edges, which
> > > > might be
> > > > > > > > easy to
> > > > > > > > > finish of.
> > > > > > > > >
> > > > > > > > > I want to figure out a couple of things:
> > > > > > > > > - what's the state of CI and how much still needs to be done
> > > > (Rvs?
> > > > > > > > Could you
> > > > > > > > > share any first hand feedback?)
> > > > > > > > > - who would be able to help with the CI completion? I can
> > commit
> > > > > > some
> > > > > > > > of my
> > > > > > > > > cycles, but it'd be great to have few more hands on that.
> > > > Clearly,
> > > > > > > > some
> > > > > > > > > Jenkins-foo and prior CI skills won't hurt ;)
> > > > > > > > >
> > > > > > > > > Please chime in if you can help. Thanks a lot!
> > > > > > > > > Cos
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
Re: Bigtop CI [Re: release 1.0.0]
Posted by Evans Ye <ev...@apache.org>.
Having jenkins master migrated to new EC2 instance would need to have a
backup of config.xml and jobs configuration. I can help to setup a new
jenkins but a backup configuration is needed. Unfortunately, I don't have
credential to login the current master and fetch a backup, do you have any
suggestion, cos?
2015-05-25 18:14 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> Hi Evans.
>
> This is great! Thank you very much for putting this together in time for
> 1.0
> release! Ignite rc7 should get into the Incubator vote today, so in a
> could of
> days it will be ready. I will make a patch for the JIRA to fix it as well.
>
> I think the next step would be to move the master to an instance on new EC2
> account, followed by a request to INFRA to setup new ci.bigtop.apache.org
> CNAME to point to that instance.
>
> I guess during that transition we'd need to clean up the old builds. For
> Debian and Suse: Olaf and Peter would be the experts in these. Guys, could
> you
> please chime in?
>
> I will try to have RC1 this week (pending the Ignite vote) before
> switching off
> for a couple weeks of vacation.
>
> Thanks again!
> Cos
> On Thu, May 21, 2015 at 12:22AM, Evans Ye wrote:
> > Folks,
> >
> > News on our Docker based CI.
> >
> > We now have nightly build for all the bigtop components available at
> > http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/
> > And it's all green except the ignite-hadoop build.
> >
> > Those Docker prefixed jobs will run daily and fetch the newest code from
> > master to build.
> > This can give us a clear view to the status of Bigtop master code
> > repository.
> > For example, we can quickly know which commit breaks the build.
> >
> > What underlying supporting this feature are 4 EMR team donated EC2
> > instances and my docker images
> > <https://github.com/evans-ye/bigtop-dockerfiles> for each Bigtop 1.0
> > targeted OS.
> >
> > To be notice that we only have centos 6, centos 7, ubuntu 14.04, and
> fedora
> > 20 builds available now, opensuse 13.2 and debian 8 are still not ready
> > yet. As I mentioned in my previous mail, it would be great if someone
> > familiar with the OS can help. :)
> >
> > Let's get back to the bigtop 1.0 release, with most of the CI
> > infrastructure(slaves, images) are available and all the JIRAs that need
> to
> > be fixed in 1.0 are addressed, I believe we're close to the release.
> > Now that the only blocker is ignite 1.1 (BIGTOP-1833
> > <https://issues.apache.org/jira/browse/BIGTOP-1833>), and the opensuse
> 13.2
> > and debian 8 images.
> >
> > The next step of our CI improvement is to setup docker-provisioner and
> runs
> > smoke-tests to evaluate the system interoperability. I'll continue
> working
> > down this path.
> >
> > Anyone interesting in this or has any suggestion is welcomed as always.
> >
> >
> > 2015-04-28 14:15 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> >
> > > ah, got it. Thanks for the explanations and all the work!
> > >
> > > On Tue, Apr 28, 2015 at 02:07PM, Evans Ye wrote:
> > > > Hey Cos,
> > > >
> > > > I build new images only if needed.
> > > >
> > > > In 0.8, we have:
> > > >
> > > > centos-6.4
> > > > fedora-20
> > > > ubuntu-12.04
> > > > ubuntu-14.04
> > > > opensuse-12.3
> > > >
> > > > Compare with the 1.0 BOM:
> > > >
> > > > OS:
> > > > CentOS6, CentOS7, Fedora 20
> > > > SLES 11SP3, SLES12, OpenSUSE 13.2
> > > > Ubuntu 14.04 LTS, Debian 8
> > > >
> > > >
> > > > fedora-20 and ubuntu-14.04 are still roman's images in 1.0.
> > > >
> > > > But centos-6.4 is quite old, hence I build a new one based on
> centos-6.6.
> > > >
> > > > centos-7 and debian are new comers in 1.0, so we have to create them
> from
> > > > scratch.
> > > >
> > > > opensuse has been upgraded from 12.3 to 13.2. We need a new image to
> be
> > > > built as well.
> > > >
> > > > Evans
> > > >
> > > > 2015-04-28 3:04 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > > >
> > > > > That's great! Thank you Evans!
> > > > >
> > > > > I am sure Peter Linnell should be able to give you a hand with
> *Suse
> > > > > platform.
> > > > > Olaf, could you please help Evans with Debian?
> > > > >
> > > > > One question about dockers: any reason you've chosen not to use the
> > > images
> > > > > that Roman has put together initially? Not like I care one way or
> > > another,
> > > > > but
> > > > > I am using those at the moment and want to make sure that we are
> > > building
> > > > > in
> > > > > the same environment across the board.
> > > > >
> > > > > Also, per Nate's request I have added S3 policy to the bigtop-ci
> group
> > > on
> > > > > AWS,
> > > > > so you guys should be able to use S3 if needed.
> > > > >
> > > > > Thanks,
> > > > > Cos
> > > > >
> > > > > On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote:
> > > > > > Hi folks,
> > > > > >
> > > > > > Allow me to report some progress on our CI settings for 1.0
> release.
> > > > > > (nate@reactor8.com asked me about the status and would like to
> help
> > > > > > but sorry I didn't respond you well back then)
> > > > > >
> > > > > >
> > > > > > For 1.0 to release, we need to build bigtop by those supported OS
> > > listed
> > > > > in
> > > > > > BIGTOP-1480.
> > > > > >
> > > > > > And I plan to use use docker images to build 1.0 release
> directly so
> > > that
> > > > > > we don't need to setup jenkins slaves one-by-one. (the fastest
> way I
> > > can
> > > > > > think of for speeding up 1.0 release)
> > > > > >
> > > > > > My plan is to prepare bigtop/puppet images by ourselves
> > > > > > <https://github.com/evans-ye/bigtop-dockerfiles>and apply
> > > > > bigtop_toolchain
> > > > > > by jenkins
> > > > > > <
> http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/
> > > >to
> > > > > get
> > > > > > bigtop/slaves images.
> > > > > >
> > > > > > I've built and tested some of the OS in bigtop/puppet family
> (stored
> > > > > > as evansye/puppet
> > > > > > <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/
> > > > > >temporary),
> > > > > > but some of the OS do not work properly since I'm lacking of
> > > knowledge on
> > > > > > them.
> > > > > >
> > > > > > ***worked***
> > > > > >
> > > > > > centos-6
> > > > > > centos-7
> > > > > > ubuntu-14.04 (unchanged)
> > > > > > fedora-20 (unchanged)
> > > > > >
> > > > > > ***not working***
> > > > > > opensuse-13.2
> > > > > > ---> I can't have "hostname" package installed, it seems that is
> > > needed
> > > > > by
> > > > > > puppet to determine what node to apply
> > > > > > debian 8
> > > > > > ---> no wget and curl available
> > > > > > [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c
> 'apt-get -y
> > > > > install
> > > > > > curl wget'
> > > > > > Reading package lists... Done
> > > > > > Building dependency tree... Done
> > > > > > E: Unable to locate package curl
> > > > > > E: Unable to locate package wget
> > > > > >
> > > > > >
> > > > > > Todo list:
> > > > > > 1. It would be a great help if opensuse and debian expert can
> come to
> > > > > help
> > > > > > on those issues.
> > > > > > 2. I'd like to have the credential for bigtop dockerhub so that I
> > > can put
> > > > > > new OS images on it.
> > > > > > 3. Have all package in trunk built by new Docker slaves
> > > > > >
> > > > > > Maybe Roman can help me out on *#2. :)*
> > > > > > You can enclose me the credential by my pgp:
> > > > > > curl http://people.apache.org/~evansye/evansye.pgp
> > > > > >
> > > > > > That it. Suggestions are welcome!
> > > > > >
> > > > > > Thanks,
> > > > > > Evans
> > > > > >
> > > > > > 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > > > > >
> > > > > > > Currently all the slaves shut-down: looks like the instances
> have
> > > been
> > > > > > > turned
> > > > > > > off as I can not relaunch the slaves. Will try to restart them
> > > tonight
> > > > > > > once I
> > > > > > > am on machine with AWS keys.
> > > > > > >
> > > > > > > Cos
> > > > > > >
> > > > > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> > > > > > > > Guys,
> > > > > > > >
> > > > > > > > I want to start a separate thread to track the CI
> preparations
> > > for
> > > > > the
> > > > > > > release
> > > > > > > > next month (fingers crossed). Clearly, we can make a release
> > > without
> > > > > CI,
> > > > > > > but
> > > > > > > > it'd way easier to test and create binary artifacts if we
> have a
> > > > > working
> > > > > > > > environment for official validation. Roman has done a lot in
> this
> > > > > > > direction
> > > > > > > > (many thanks!), but there are still a few rough edges, which
> > > might be
> > > > > > > easy to
> > > > > > > > finish of.
> > > > > > > >
> > > > > > > > I want to figure out a couple of things:
> > > > > > > > - what's the state of CI and how much still needs to be done
> > > (Rvs?
> > > > > > > Could you
> > > > > > > > share any first hand feedback?)
> > > > > > > > - who would be able to help with the CI completion? I can
> commit
> > > > > some
> > > > > > > of my
> > > > > > > > cycles, but it'd be great to have few more hands on that.
> > > Clearly,
> > > > > > > some
> > > > > > > > Jenkins-foo and prior CI skills won't hurt ;)
> > > > > > > >
> > > > > > > > Please chime in if you can help. Thanks a lot!
> > > > > > > > Cos
> > > > > > > >
> > > > > > >
> > > > >
> > >
>
Re: Bigtop CI [Re: release 1.0.0]
Posted by Konstantin Boudnik <co...@apache.org>.
Hi Evans.
This is great! Thank you very much for putting this together in time for 1.0
release! Ignite rc7 should get into the Incubator vote today, so in a could of
days it will be ready. I will make a patch for the JIRA to fix it as well.
I think the next step would be to move the master to an instance on new EC2
account, followed by a request to INFRA to setup new ci.bigtop.apache.org
CNAME to point to that instance.
I guess during that transition we'd need to clean up the old builds. For
Debian and Suse: Olaf and Peter would be the experts in these. Guys, could you
please chime in?
I will try to have RC1 this week (pending the Ignite vote) before switching off
for a couple weeks of vacation.
Thanks again!
Cos
On Thu, May 21, 2015 at 12:22AM, Evans Ye wrote:
> Folks,
>
> News on our Docker based CI.
>
> We now have nightly build for all the bigtop components available at
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/
> And it's all green except the ignite-hadoop build.
>
> Those Docker prefixed jobs will run daily and fetch the newest code from
> master to build.
> This can give us a clear view to the status of Bigtop master code
> repository.
> For example, we can quickly know which commit breaks the build.
>
> What underlying supporting this feature are 4 EMR team donated EC2
> instances and my docker images
> <https://github.com/evans-ye/bigtop-dockerfiles> for each Bigtop 1.0
> targeted OS.
>
> To be notice that we only have centos 6, centos 7, ubuntu 14.04, and fedora
> 20 builds available now, opensuse 13.2 and debian 8 are still not ready
> yet. As I mentioned in my previous mail, it would be great if someone
> familiar with the OS can help. :)
>
> Let's get back to the bigtop 1.0 release, with most of the CI
> infrastructure(slaves, images) are available and all the JIRAs that need to
> be fixed in 1.0 are addressed, I believe we're close to the release.
> Now that the only blocker is ignite 1.1 (BIGTOP-1833
> <https://issues.apache.org/jira/browse/BIGTOP-1833>), and the opensuse 13.2
> and debian 8 images.
>
> The next step of our CI improvement is to setup docker-provisioner and runs
> smoke-tests to evaluate the system interoperability. I'll continue working
> down this path.
>
> Anyone interesting in this or has any suggestion is welcomed as always.
>
>
> 2015-04-28 14:15 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>
> > ah, got it. Thanks for the explanations and all the work!
> >
> > On Tue, Apr 28, 2015 at 02:07PM, Evans Ye wrote:
> > > Hey Cos,
> > >
> > > I build new images only if needed.
> > >
> > > In 0.8, we have:
> > >
> > > centos-6.4
> > > fedora-20
> > > ubuntu-12.04
> > > ubuntu-14.04
> > > opensuse-12.3
> > >
> > > Compare with the 1.0 BOM:
> > >
> > > OS:
> > > CentOS6, CentOS7, Fedora 20
> > > SLES 11SP3, SLES12, OpenSUSE 13.2
> > > Ubuntu 14.04 LTS, Debian 8
> > >
> > >
> > > fedora-20 and ubuntu-14.04 are still roman's images in 1.0.
> > >
> > > But centos-6.4 is quite old, hence I build a new one based on centos-6.6.
> > >
> > > centos-7 and debian are new comers in 1.0, so we have to create them from
> > > scratch.
> > >
> > > opensuse has been upgraded from 12.3 to 13.2. We need a new image to be
> > > built as well.
> > >
> > > Evans
> > >
> > > 2015-04-28 3:04 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >
> > > > That's great! Thank you Evans!
> > > >
> > > > I am sure Peter Linnell should be able to give you a hand with *Suse
> > > > platform.
> > > > Olaf, could you please help Evans with Debian?
> > > >
> > > > One question about dockers: any reason you've chosen not to use the
> > images
> > > > that Roman has put together initially? Not like I care one way or
> > another,
> > > > but
> > > > I am using those at the moment and want to make sure that we are
> > building
> > > > in
> > > > the same environment across the board.
> > > >
> > > > Also, per Nate's request I have added S3 policy to the bigtop-ci group
> > on
> > > > AWS,
> > > > so you guys should be able to use S3 if needed.
> > > >
> > > > Thanks,
> > > > Cos
> > > >
> > > > On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote:
> > > > > Hi folks,
> > > > >
> > > > > Allow me to report some progress on our CI settings for 1.0 release.
> > > > > (nate@reactor8.com asked me about the status and would like to help
> > > > > but sorry I didn't respond you well back then)
> > > > >
> > > > >
> > > > > For 1.0 to release, we need to build bigtop by those supported OS
> > listed
> > > > in
> > > > > BIGTOP-1480.
> > > > >
> > > > > And I plan to use use docker images to build 1.0 release directly so
> > that
> > > > > we don't need to setup jenkins slaves one-by-one. (the fastest way I
> > can
> > > > > think of for speeding up 1.0 release)
> > > > >
> > > > > My plan is to prepare bigtop/puppet images by ourselves
> > > > > <https://github.com/evans-ye/bigtop-dockerfiles>and apply
> > > > bigtop_toolchain
> > > > > by jenkins
> > > > > <http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/
> > >to
> > > > get
> > > > > bigtop/slaves images.
> > > > >
> > > > > I've built and tested some of the OS in bigtop/puppet family (stored
> > > > > as evansye/puppet
> > > > > <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/
> > > > >temporary),
> > > > > but some of the OS do not work properly since I'm lacking of
> > knowledge on
> > > > > them.
> > > > >
> > > > > ***worked***
> > > > >
> > > > > centos-6
> > > > > centos-7
> > > > > ubuntu-14.04 (unchanged)
> > > > > fedora-20 (unchanged)
> > > > >
> > > > > ***not working***
> > > > > opensuse-13.2
> > > > > ---> I can't have "hostname" package installed, it seems that is
> > needed
> > > > by
> > > > > puppet to determine what node to apply
> > > > > debian 8
> > > > > ---> no wget and curl available
> > > > > [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c 'apt-get -y
> > > > install
> > > > > curl wget'
> > > > > Reading package lists... Done
> > > > > Building dependency tree... Done
> > > > > E: Unable to locate package curl
> > > > > E: Unable to locate package wget
> > > > >
> > > > >
> > > > > Todo list:
> > > > > 1. It would be a great help if opensuse and debian expert can come to
> > > > help
> > > > > on those issues.
> > > > > 2. I'd like to have the credential for bigtop dockerhub so that I
> > can put
> > > > > new OS images on it.
> > > > > 3. Have all package in trunk built by new Docker slaves
> > > > >
> > > > > Maybe Roman can help me out on *#2. :)*
> > > > > You can enclose me the credential by my pgp:
> > > > > curl http://people.apache.org/~evansye/evansye.pgp
> > > > >
> > > > > That it. Suggestions are welcome!
> > > > >
> > > > > Thanks,
> > > > > Evans
> > > > >
> > > > > 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > > > >
> > > > > > Currently all the slaves shut-down: looks like the instances have
> > been
> > > > > > turned
> > > > > > off as I can not relaunch the slaves. Will try to restart them
> > tonight
> > > > > > once I
> > > > > > am on machine with AWS keys.
> > > > > >
> > > > > > Cos
> > > > > >
> > > > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> > > > > > > Guys,
> > > > > > >
> > > > > > > I want to start a separate thread to track the CI preparations
> > for
> > > > the
> > > > > > release
> > > > > > > next month (fingers crossed). Clearly, we can make a release
> > without
> > > > CI,
> > > > > > but
> > > > > > > it'd way easier to test and create binary artifacts if we have a
> > > > working
> > > > > > > environment for official validation. Roman has done a lot in this
> > > > > > direction
> > > > > > > (many thanks!), but there are still a few rough edges, which
> > might be
> > > > > > easy to
> > > > > > > finish of.
> > > > > > >
> > > > > > > I want to figure out a couple of things:
> > > > > > > - what's the state of CI and how much still needs to be done
> > (Rvs?
> > > > > > Could you
> > > > > > > share any first hand feedback?)
> > > > > > > - who would be able to help with the CI completion? I can commit
> > > > some
> > > > > > of my
> > > > > > > cycles, but it'd be great to have few more hands on that.
> > Clearly,
> > > > > > some
> > > > > > > Jenkins-foo and prior CI skills won't hurt ;)
> > > > > > >
> > > > > > > Please chime in if you can help. Thanks a lot!
> > > > > > > Cos
> > > > > > >
> > > > > >
> > > >
> >
Re: Bigtop CI [Re: release 1.0.0]
Posted by Evans Ye <ev...@apache.org>.
Hi Olaf,
Thanks for providing help.
I finally found that the issue happened on debian container is just because
our company's intranet can not resolve debian repo...
root@b3b601f45634:/# apt-get update
Err http://httpredir.debian.org jessie InRelease
I can now successfully build required debian image from debian:jessie.
Awesome!
2015-06-07 5:27 GMT+08:00 Olaf Flebbe <of...@oflebbe.de>:
> Hi Evans,
>
> Sorry for late answer, I was away from fast internet for a few weeks.
> Great to see progress. Please contact me directly what issue you have with
> debian8. Since I have no credentials I cannot help right now.
>
> The official debian release "docker pull debian:jessie“ should be
> sufficient.
>
> Olaf
>
Re: Bigtop CI [Re: release 1.0.0]
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi Evans,
Sorry for late answer, I was away from fast internet for a few weeks. Great to see progress. Please contact me directly what issue you have with debian8. Since I have no credentials I cannot help right now.
The official debian release "docker pull debian:jessie“ should be sufficient.
Olaf
Re: Bigtop CI [Re: release 1.0.0]
Posted by Evans Ye <ev...@apache.org>.
Folks,
News on our Docker based CI.
We now have nightly build for all the bigtop components available at
http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/
And it's all green except the ignite-hadoop build.
Those Docker prefixed jobs will run daily and fetch the newest code from
master to build.
This can give us a clear view to the status of Bigtop master code
repository.
For example, we can quickly know which commit breaks the build.
What underlying supporting this feature are 4 EMR team donated EC2
instances and my docker images
<https://github.com/evans-ye/bigtop-dockerfiles> for each Bigtop 1.0
targeted OS.
To be notice that we only have centos 6, centos 7, ubuntu 14.04, and fedora
20 builds available now, opensuse 13.2 and debian 8 are still not ready
yet. As I mentioned in my previous mail, it would be great if someone
familiar with the OS can help. :)
Let's get back to the bigtop 1.0 release, with most of the CI
infrastructure(slaves, images) are available and all the JIRAs that need to
be fixed in 1.0 are addressed, I believe we're close to the release.
Now that the only blocker is ignite 1.1 (BIGTOP-1833
<https://issues.apache.org/jira/browse/BIGTOP-1833>), and the opensuse 13.2
and debian 8 images.
The next step of our CI improvement is to setup docker-provisioner and runs
smoke-tests to evaluate the system interoperability. I'll continue working
down this path.
Anyone interesting in this or has any suggestion is welcomed as always.
2015-04-28 14:15 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> ah, got it. Thanks for the explanations and all the work!
>
> On Tue, Apr 28, 2015 at 02:07PM, Evans Ye wrote:
> > Hey Cos,
> >
> > I build new images only if needed.
> >
> > In 0.8, we have:
> >
> > centos-6.4
> > fedora-20
> > ubuntu-12.04
> > ubuntu-14.04
> > opensuse-12.3
> >
> > Compare with the 1.0 BOM:
> >
> > OS:
> > CentOS6, CentOS7, Fedora 20
> > SLES 11SP3, SLES12, OpenSUSE 13.2
> > Ubuntu 14.04 LTS, Debian 8
> >
> >
> > fedora-20 and ubuntu-14.04 are still roman's images in 1.0.
> >
> > But centos-6.4 is quite old, hence I build a new one based on centos-6.6.
> >
> > centos-7 and debian are new comers in 1.0, so we have to create them from
> > scratch.
> >
> > opensuse has been upgraded from 12.3 to 13.2. We need a new image to be
> > built as well.
> >
> > Evans
> >
> > 2015-04-28 3:04 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> >
> > > That's great! Thank you Evans!
> > >
> > > I am sure Peter Linnell should be able to give you a hand with *Suse
> > > platform.
> > > Olaf, could you please help Evans with Debian?
> > >
> > > One question about dockers: any reason you've chosen not to use the
> images
> > > that Roman has put together initially? Not like I care one way or
> another,
> > > but
> > > I am using those at the moment and want to make sure that we are
> building
> > > in
> > > the same environment across the board.
> > >
> > > Also, per Nate's request I have added S3 policy to the bigtop-ci group
> on
> > > AWS,
> > > so you guys should be able to use S3 if needed.
> > >
> > > Thanks,
> > > Cos
> > >
> > > On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote:
> > > > Hi folks,
> > > >
> > > > Allow me to report some progress on our CI settings for 1.0 release.
> > > > (nate@reactor8.com asked me about the status and would like to help
> > > > but sorry I didn't respond you well back then)
> > > >
> > > >
> > > > For 1.0 to release, we need to build bigtop by those supported OS
> listed
> > > in
> > > > BIGTOP-1480.
> > > >
> > > > And I plan to use use docker images to build 1.0 release directly so
> that
> > > > we don't need to setup jenkins slaves one-by-one. (the fastest way I
> can
> > > > think of for speeding up 1.0 release)
> > > >
> > > > My plan is to prepare bigtop/puppet images by ourselves
> > > > <https://github.com/evans-ye/bigtop-dockerfiles>and apply
> > > bigtop_toolchain
> > > > by jenkins
> > > > <http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/
> >to
> > > get
> > > > bigtop/slaves images.
> > > >
> > > > I've built and tested some of the OS in bigtop/puppet family (stored
> > > > as evansye/puppet
> > > > <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/
> > > >temporary),
> > > > but some of the OS do not work properly since I'm lacking of
> knowledge on
> > > > them.
> > > >
> > > > ***worked***
> > > >
> > > > centos-6
> > > > centos-7
> > > > ubuntu-14.04 (unchanged)
> > > > fedora-20 (unchanged)
> > > >
> > > > ***not working***
> > > > opensuse-13.2
> > > > ---> I can't have "hostname" package installed, it seems that is
> needed
> > > by
> > > > puppet to determine what node to apply
> > > > debian 8
> > > > ---> no wget and curl available
> > > > [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c 'apt-get -y
> > > install
> > > > curl wget'
> > > > Reading package lists... Done
> > > > Building dependency tree... Done
> > > > E: Unable to locate package curl
> > > > E: Unable to locate package wget
> > > >
> > > >
> > > > Todo list:
> > > > 1. It would be a great help if opensuse and debian expert can come to
> > > help
> > > > on those issues.
> > > > 2. I'd like to have the credential for bigtop dockerhub so that I
> can put
> > > > new OS images on it.
> > > > 3. Have all package in trunk built by new Docker slaves
> > > >
> > > > Maybe Roman can help me out on *#2. :)*
> > > > You can enclose me the credential by my pgp:
> > > > curl http://people.apache.org/~evansye/evansye.pgp
> > > >
> > > > That it. Suggestions are welcome!
> > > >
> > > > Thanks,
> > > > Evans
> > > >
> > > > 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > > >
> > > > > Currently all the slaves shut-down: looks like the instances have
> been
> > > > > turned
> > > > > off as I can not relaunch the slaves. Will try to restart them
> tonight
> > > > > once I
> > > > > am on machine with AWS keys.
> > > > >
> > > > > Cos
> > > > >
> > > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> > > > > > Guys,
> > > > > >
> > > > > > I want to start a separate thread to track the CI preparations
> for
> > > the
> > > > > release
> > > > > > next month (fingers crossed). Clearly, we can make a release
> without
> > > CI,
> > > > > but
> > > > > > it'd way easier to test and create binary artifacts if we have a
> > > working
> > > > > > environment for official validation. Roman has done a lot in this
> > > > > direction
> > > > > > (many thanks!), but there are still a few rough edges, which
> might be
> > > > > easy to
> > > > > > finish of.
> > > > > >
> > > > > > I want to figure out a couple of things:
> > > > > > - what's the state of CI and how much still needs to be done
> (Rvs?
> > > > > Could you
> > > > > > share any first hand feedback?)
> > > > > > - who would be able to help with the CI completion? I can commit
> > > some
> > > > > of my
> > > > > > cycles, but it'd be great to have few more hands on that.
> Clearly,
> > > > > some
> > > > > > Jenkins-foo and prior CI skills won't hurt ;)
> > > > > >
> > > > > > Please chime in if you can help. Thanks a lot!
> > > > > > Cos
> > > > > >
> > > > >
> > >
>
Re: Bigtop CI [Re: release 1.0.0]
Posted by Konstantin Boudnik <co...@apache.org>.
ah, got it. Thanks for the explanations and all the work!
On Tue, Apr 28, 2015 at 02:07PM, Evans Ye wrote:
> Hey Cos,
>
> I build new images only if needed.
>
> In 0.8, we have:
>
> centos-6.4
> fedora-20
> ubuntu-12.04
> ubuntu-14.04
> opensuse-12.3
>
> Compare with the 1.0 BOM:
>
> OS:
> CentOS6, CentOS7, Fedora 20
> SLES 11SP3, SLES12, OpenSUSE 13.2
> Ubuntu 14.04 LTS, Debian 8
>
>
> fedora-20 and ubuntu-14.04 are still roman's images in 1.0.
>
> But centos-6.4 is quite old, hence I build a new one based on centos-6.6.
>
> centos-7 and debian are new comers in 1.0, so we have to create them from
> scratch.
>
> opensuse has been upgraded from 12.3 to 13.2. We need a new image to be
> built as well.
>
> Evans
>
> 2015-04-28 3:04 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>
> > That's great! Thank you Evans!
> >
> > I am sure Peter Linnell should be able to give you a hand with *Suse
> > platform.
> > Olaf, could you please help Evans with Debian?
> >
> > One question about dockers: any reason you've chosen not to use the images
> > that Roman has put together initially? Not like I care one way or another,
> > but
> > I am using those at the moment and want to make sure that we are building
> > in
> > the same environment across the board.
> >
> > Also, per Nate's request I have added S3 policy to the bigtop-ci group on
> > AWS,
> > so you guys should be able to use S3 if needed.
> >
> > Thanks,
> > Cos
> >
> > On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote:
> > > Hi folks,
> > >
> > > Allow me to report some progress on our CI settings for 1.0 release.
> > > (nate@reactor8.com asked me about the status and would like to help
> > > but sorry I didn't respond you well back then)
> > >
> > >
> > > For 1.0 to release, we need to build bigtop by those supported OS listed
> > in
> > > BIGTOP-1480.
> > >
> > > And I plan to use use docker images to build 1.0 release directly so that
> > > we don't need to setup jenkins slaves one-by-one. (the fastest way I can
> > > think of for speeding up 1.0 release)
> > >
> > > My plan is to prepare bigtop/puppet images by ourselves
> > > <https://github.com/evans-ye/bigtop-dockerfiles>and apply
> > bigtop_toolchain
> > > by jenkins
> > > <http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/>to
> > get
> > > bigtop/slaves images.
> > >
> > > I've built and tested some of the OS in bigtop/puppet family (stored
> > > as evansye/puppet
> > > <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/
> > >temporary),
> > > but some of the OS do not work properly since I'm lacking of knowledge on
> > > them.
> > >
> > > ***worked***
> > >
> > > centos-6
> > > centos-7
> > > ubuntu-14.04 (unchanged)
> > > fedora-20 (unchanged)
> > >
> > > ***not working***
> > > opensuse-13.2
> > > ---> I can't have "hostname" package installed, it seems that is needed
> > by
> > > puppet to determine what node to apply
> > > debian 8
> > > ---> no wget and curl available
> > > [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c 'apt-get -y
> > install
> > > curl wget'
> > > Reading package lists... Done
> > > Building dependency tree... Done
> > > E: Unable to locate package curl
> > > E: Unable to locate package wget
> > >
> > >
> > > Todo list:
> > > 1. It would be a great help if opensuse and debian expert can come to
> > help
> > > on those issues.
> > > 2. I'd like to have the credential for bigtop dockerhub so that I can put
> > > new OS images on it.
> > > 3. Have all package in trunk built by new Docker slaves
> > >
> > > Maybe Roman can help me out on *#2. :)*
> > > You can enclose me the credential by my pgp:
> > > curl http://people.apache.org/~evansye/evansye.pgp
> > >
> > > That it. Suggestions are welcome!
> > >
> > > Thanks,
> > > Evans
> > >
> > > 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >
> > > > Currently all the slaves shut-down: looks like the instances have been
> > > > turned
> > > > off as I can not relaunch the slaves. Will try to restart them tonight
> > > > once I
> > > > am on machine with AWS keys.
> > > >
> > > > Cos
> > > >
> > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> > > > > Guys,
> > > > >
> > > > > I want to start a separate thread to track the CI preparations for
> > the
> > > > release
> > > > > next month (fingers crossed). Clearly, we can make a release without
> > CI,
> > > > but
> > > > > it'd way easier to test and create binary artifacts if we have a
> > working
> > > > > environment for official validation. Roman has done a lot in this
> > > > direction
> > > > > (many thanks!), but there are still a few rough edges, which might be
> > > > easy to
> > > > > finish of.
> > > > >
> > > > > I want to figure out a couple of things:
> > > > > - what's the state of CI and how much still needs to be done (Rvs?
> > > > Could you
> > > > > share any first hand feedback?)
> > > > > - who would be able to help with the CI completion? I can commit
> > some
> > > > of my
> > > > > cycles, but it'd be great to have few more hands on that. Clearly,
> > > > some
> > > > > Jenkins-foo and prior CI skills won't hurt ;)
> > > > >
> > > > > Please chime in if you can help. Thanks a lot!
> > > > > Cos
> > > > >
> > > >
> >
Re: Bigtop CI [Re: release 1.0.0]
Posted by Evans Ye <ev...@apache.org>.
Hey Cos,
I build new images only if needed.
In 0.8, we have:
centos-6.4
fedora-20
ubuntu-12.04
ubuntu-14.04
opensuse-12.3
Compare with the 1.0 BOM:
OS:
CentOS6, CentOS7, Fedora 20
SLES 11SP3, SLES12, OpenSUSE 13.2
Ubuntu 14.04 LTS, Debian 8
fedora-20 and ubuntu-14.04 are still roman's images in 1.0.
But centos-6.4 is quite old, hence I build a new one based on centos-6.6.
centos-7 and debian are new comers in 1.0, so we have to create them from
scratch.
opensuse has been upgraded from 12.3 to 13.2. We need a new image to be
built as well.
Evans
2015-04-28 3:04 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> That's great! Thank you Evans!
>
> I am sure Peter Linnell should be able to give you a hand with *Suse
> platform.
> Olaf, could you please help Evans with Debian?
>
> One question about dockers: any reason you've chosen not to use the images
> that Roman has put together initially? Not like I care one way or another,
> but
> I am using those at the moment and want to make sure that we are building
> in
> the same environment across the board.
>
> Also, per Nate's request I have added S3 policy to the bigtop-ci group on
> AWS,
> so you guys should be able to use S3 if needed.
>
> Thanks,
> Cos
>
> On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote:
> > Hi folks,
> >
> > Allow me to report some progress on our CI settings for 1.0 release.
> > (nate@reactor8.com asked me about the status and would like to help
> > but sorry I didn't respond you well back then)
> >
> >
> > For 1.0 to release, we need to build bigtop by those supported OS listed
> in
> > BIGTOP-1480.
> >
> > And I plan to use use docker images to build 1.0 release directly so that
> > we don't need to setup jenkins slaves one-by-one. (the fastest way I can
> > think of for speeding up 1.0 release)
> >
> > My plan is to prepare bigtop/puppet images by ourselves
> > <https://github.com/evans-ye/bigtop-dockerfiles>and apply
> bigtop_toolchain
> > by jenkins
> > <http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/>to
> get
> > bigtop/slaves images.
> >
> > I've built and tested some of the OS in bigtop/puppet family (stored
> > as evansye/puppet
> > <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/
> >temporary),
> > but some of the OS do not work properly since I'm lacking of knowledge on
> > them.
> >
> > ***worked***
> >
> > centos-6
> > centos-7
> > ubuntu-14.04 (unchanged)
> > fedora-20 (unchanged)
> >
> > ***not working***
> > opensuse-13.2
> > ---> I can't have "hostname" package installed, it seems that is needed
> by
> > puppet to determine what node to apply
> > debian 8
> > ---> no wget and curl available
> > [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c 'apt-get -y
> install
> > curl wget'
> > Reading package lists... Done
> > Building dependency tree... Done
> > E: Unable to locate package curl
> > E: Unable to locate package wget
> >
> >
> > Todo list:
> > 1. It would be a great help if opensuse and debian expert can come to
> help
> > on those issues.
> > 2. I'd like to have the credential for bigtop dockerhub so that I can put
> > new OS images on it.
> > 3. Have all package in trunk built by new Docker slaves
> >
> > Maybe Roman can help me out on *#2. :)*
> > You can enclose me the credential by my pgp:
> > curl http://people.apache.org/~evansye/evansye.pgp
> >
> > That it. Suggestions are welcome!
> >
> > Thanks,
> > Evans
> >
> > 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> >
> > > Currently all the slaves shut-down: looks like the instances have been
> > > turned
> > > off as I can not relaunch the slaves. Will try to restart them tonight
> > > once I
> > > am on machine with AWS keys.
> > >
> > > Cos
> > >
> > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> > > > Guys,
> > > >
> > > > I want to start a separate thread to track the CI preparations for
> the
> > > release
> > > > next month (fingers crossed). Clearly, we can make a release without
> CI,
> > > but
> > > > it'd way easier to test and create binary artifacts if we have a
> working
> > > > environment for official validation. Roman has done a lot in this
> > > direction
> > > > (many thanks!), but there are still a few rough edges, which might be
> > > easy to
> > > > finish of.
> > > >
> > > > I want to figure out a couple of things:
> > > > - what's the state of CI and how much still needs to be done (Rvs?
> > > Could you
> > > > share any first hand feedback?)
> > > > - who would be able to help with the CI completion? I can commit
> some
> > > of my
> > > > cycles, but it'd be great to have few more hands on that. Clearly,
> > > some
> > > > Jenkins-foo and prior CI skills won't hurt ;)
> > > >
> > > > Please chime in if you can help. Thanks a lot!
> > > > Cos
> > > >
> > >
>
Re: Bigtop CI [Re: release 1.0.0]
Posted by Konstantin Boudnik <co...@apache.org>.
That's great! Thank you Evans!
I am sure Peter Linnell should be able to give you a hand with *Suse platform.
Olaf, could you please help Evans with Debian?
One question about dockers: any reason you've chosen not to use the images
that Roman has put together initially? Not like I care one way or another, but
I am using those at the moment and want to make sure that we are building in
the same environment across the board.
Also, per Nate's request I have added S3 policy to the bigtop-ci group on AWS,
so you guys should be able to use S3 if needed.
Thanks,
Cos
On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote:
> Hi folks,
>
> Allow me to report some progress on our CI settings for 1.0 release.
> (nate@reactor8.com asked me about the status and would like to help
> but sorry I didn't respond you well back then)
>
>
> For 1.0 to release, we need to build bigtop by those supported OS listed in
> BIGTOP-1480.
>
> And I plan to use use docker images to build 1.0 release directly so that
> we don't need to setup jenkins slaves one-by-one. (the fastest way I can
> think of for speeding up 1.0 release)
>
> My plan is to prepare bigtop/puppet images by ourselves
> <https://github.com/evans-ye/bigtop-dockerfiles>and apply bigtop_toolchain
> by jenkins
> <http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/>to get
> bigtop/slaves images.
>
> I've built and tested some of the OS in bigtop/puppet family (stored
> as evansye/puppet
> <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/>temporary),
> but some of the OS do not work properly since I'm lacking of knowledge on
> them.
>
> ***worked***
>
> centos-6
> centos-7
> ubuntu-14.04 (unchanged)
> fedora-20 (unchanged)
>
> ***not working***
> opensuse-13.2
> ---> I can't have "hostname" package installed, it seems that is needed by
> puppet to determine what node to apply
> debian 8
> ---> no wget and curl available
> [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c 'apt-get -y install
> curl wget'
> Reading package lists... Done
> Building dependency tree... Done
> E: Unable to locate package curl
> E: Unable to locate package wget
>
>
> Todo list:
> 1. It would be a great help if opensuse and debian expert can come to help
> on those issues.
> 2. I'd like to have the credential for bigtop dockerhub so that I can put
> new OS images on it.
> 3. Have all package in trunk built by new Docker slaves
>
> Maybe Roman can help me out on *#2. :)*
> You can enclose me the credential by my pgp:
> curl http://people.apache.org/~evansye/evansye.pgp
>
> That it. Suggestions are welcome!
>
> Thanks,
> Evans
>
> 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>
> > Currently all the slaves shut-down: looks like the instances have been
> > turned
> > off as I can not relaunch the slaves. Will try to restart them tonight
> > once I
> > am on machine with AWS keys.
> >
> > Cos
> >
> > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote:
> > > Guys,
> > >
> > > I want to start a separate thread to track the CI preparations for the
> > release
> > > next month (fingers crossed). Clearly, we can make a release without CI,
> > but
> > > it'd way easier to test and create binary artifacts if we have a working
> > > environment for official validation. Roman has done a lot in this
> > direction
> > > (many thanks!), but there are still a few rough edges, which might be
> > easy to
> > > finish of.
> > >
> > > I want to figure out a couple of things:
> > > - what's the state of CI and how much still needs to be done (Rvs?
> > Could you
> > > share any first hand feedback?)
> > > - who would be able to help with the CI completion? I can commit some
> > of my
> > > cycles, but it'd be great to have few more hands on that. Clearly,
> > some
> > > Jenkins-foo and prior CI skills won't hurt ;)
> > >
> > > Please chime in if you can help. Thanks a lot!
> > > Cos
> > >
> >