You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by Jay Vyas <ja...@gmail.com> on 2013/07/06 16:33:09 UTC
permanant builds on jenkins
Hi bigtop:
Are there any permanant builds saved on jenkins (for the VM matrix)?
If not it would be nice to add them for certain known well tested, working
disk images .
(for context, I'm currently running Mr2 build of the KVM box and it appears
to have some intermittent write issues on the DataNode path, and also, my
namenode appears to really like being in safe mode. these could just be
due to VM setup though, as im changing some things like adding static IPs
and data node write paths... so nothing to be alarmed about.)
--
Jay Vyas
http://jayunit100.blogspot.com
Re: permanent builds on jenkins
Posted by Konstantin Boudnik <co...@apache.org>.
And also it is 'permanent'.
On Sun, Jul 07, 2013 at 02:37PM, Konstantin Boudnik wrote:
> [Moving user@ to Bcc: and cross-posting to dev@ This is a more dev discussion,
> let's keep it there]
>
> I think there's a confusion of some sort here. Apache normally doesn't provide
> binary releases. A release - in ASF - is a source code. While binaries can be
> provided as a convenience artifacts it is most often isn't a case.
>
> Now, a Bigtop release is a fixed BOM file that states what versions of what
> components were built and tested to work against each other. If you are
> looking for particular system packages - it is very easy to get by
> make rpm|deb
> from a particular release tag. I believe we do publish binary packages for
> each release, but you should understand that these are stored on s3 that costs
> money. Right now, BT build infrastructure is running on the funds donated by
> commercial vendors (Cloudera is the only one so far, others are chiming in as
> we speak).
>
> Same goes for a VM: if you need a VM with particular release bits in it: you
> just check out the tag and run boxgrinder that will create identical VM each
> time you run the build. The VM in question isn't meant for development of BT.
> This is a mere test-bench for people who want to try the latest release of
> Hadoop based stack and have no skills of going through the whole process by
> themselves.
>
> I wish we could keep a copy of VMs for all stable builds, but this is a simple
> matter of the costs vs. convenience.
>
> As for #3 below: this is pretty much how our Jenkins infra works. There's a
> bunch of slave VMs that are used for package building/installation/testing. If
> you have some improvements in mind - feel free to open a JIRA tickets and
> contribute a patch/design document/etc. Any help is highly welcome.
>
> Cos
>
> On Sun, Jul 07, 2013 at 12:03PM, Jay Vyas wrote:
> > Hi bruno:
> >
> > 1) I've noticed that https://builds.apache.org/view/A-F/view/Bigtop/ is not
> > valid but "https://builds.apache.org/job/Bigtop-trunk/" seems to work. Is
> > this an error on the website?
> >
> > 2) http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/ Seems to build
> > direct from HEAD.
> >
> > Are there actual bigtop releases which aren't on jenkins? Everything seems
> > to be centered around the build server, rather than actual frozen,
> > downloadable releases.
> >
> > 3) Regarding testing the VMs. Maybe to start some simple virt-install
> > scripts could help confirm that the KVM builds, at least, are working. If
> > we could get them working with static IPs then we could even clone down the
> > bigtop source code and run bigtop hadoop tests on the VMs after
> > construction as a validation step.
> >
> >
> >
> >
> > On Sat, Jul 6, 2013 at 8:58 PM, Bruno MahИ <bm...@apache.org> wrote:
> >
> > > On 07/06/2013 07:33 AM, Jay Vyas wrote:
> > >
> > >> Hi bigtop:
> > >>
> > >> Are there any permanant builds saved on jenkins (for the VM matrix)?
> > >>
> > >> If not it would be nice to add them for certain known well tested,
> > >> working disk images .
> > >>
> > >> (for context, I'm currently running Mr2 build of the KVM box and it
> > >> appears to have some intermittent write issues on the DataNode path, and
> > >> also, my namenode appears to really like being in safe mode. these
> > >> could just be due to VM setup though, as im changing some things like
> > >> adding static IPs and data node write paths... so nothing to be alarmed
> > >> about.)
> > >>
> > >> --
> > >> Jay Vyas
> > >> http://jayunit100.blogspot.com
> > >>
> > >
> > > Hi Jay,
> > >
> > > Could you defined "permanent build" ?
> > > I am not sure if this fits your requirement, but jenkins has a link to the
> > > latest successful build (ex: http://bigtop01.cloudera.org:**
> > > 8080/job/Bigtop-VM-matrix/BR=**master,KIND=kvm,label=**
> > > fedora16/lastSuccessfulBuild/**artifact/bigtop-vm-kvm-master.**tar.gz<http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/BR=master,KIND=kvm,label=fedora16/lastSuccessfulBuild/artifact/bigtop-vm-kvm-master.tar.gz>)
> > >
> > > We do not store convenient artifacts of VMs since no one has asked about
> > > it before.
> > > So ideally, known well tested working disk images would be the ones from
> > > Apache Bigtop releases. But right now, there is not much testing of our
> > > VMs. But any help on that front would be welcome!
> > >
> > > Note also that I added that VM more as a base VM for an Apache Hadoop
> > > cloud image than a developer VM. That's why there is not much in it as well
> > > as no desktop pre-configured.
> > > So depending on your needs, we may want to add a new VM or enhance the
> > > current one (also, boxgrinder enables inheritance between appliances).
> > >
> > > Also Boxgrinder is apparently not being maintained anymore. So we may want
> > > to look into other VM builders (Oz, etc.)
> > >
> > >
> > > Thanks,
> > > Bruno
> > >
> >
> >
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
Re: permanant builds on jenkins
Posted by Jay Vyas <ja...@gmail.com>.
Okay thanks for the feedback.
On Sun, Jul 7, 2013 at 5:37 PM, Konstantin Boudnik <co...@apache.org> wrote:
> [Moving user@ to Bcc: and cross-posting to dev@ This is a more dev
> discussion,
> let's keep it there]
>
> I think there's a confusion of some sort here. Apache normally doesn't
> provide
> binary releases. A release - in ASF - is a source code. While binaries can
> be
> provided as a convenience artifacts it is most often isn't a case.
>
> Now, a Bigtop release is a fixed BOM file that states what versions of what
> components were built and tested to work against each other. If you are
> looking for particular system packages - it is very easy to get by
> make rpm|deb
> from a particular release tag. I believe we do publish binary packages for
> each release, but you should understand that these are stored on s3 that
> costs
> money. Right now, BT build infrastructure is running on the funds donated
> by
> commercial vendors (Cloudera is the only one so far, others are chiming in
> as
> we speak).
>
> Same goes for a VM: if you need a VM with particular release bits in it:
> you
> just check out the tag and run boxgrinder that will create identical VM
> each
> time you run the build. The VM in question isn't meant for development of
> BT.
> This is a mere test-bench for people who want to try the latest release of
> Hadoop based stack and have no skills of going through the whole process by
> themselves.
>
> I wish we could keep a copy of VMs for all stable builds, but this is a
> simple
> matter of the costs vs. convenience.
>
> As for #3 below: this is pretty much how our Jenkins infra works. There's a
> bunch of slave VMs that are used for package
> building/installation/testing. If
> you have some improvements in mind - feel free to open a JIRA tickets and
> contribute a patch/design document/etc. Any help is highly welcome.
>
> Cos
>
> On Sun, Jul 07, 2013 at 12:03PM, Jay Vyas wrote:
> > Hi bruno:
> >
> > 1) I've noticed that https://builds.apache.org/view/A-F/view/Bigtop/ is
> not
> > valid but "https://builds.apache.org/job/Bigtop-trunk/" seems to work.
> Is
> > this an error on the website?
> >
> > 2) http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/ Seems to
> build
> > direct from HEAD.
> >
> > Are there actual bigtop releases which aren't on jenkins? Everything
> seems
> > to be centered around the build server, rather than actual frozen,
> > downloadable releases.
> >
> > 3) Regarding testing the VMs. Maybe to start some simple virt-install
> > scripts could help confirm that the KVM builds, at least, are working.
> If
> > we could get them working with static IPs then we could even clone down
> the
> > bigtop source code and run bigtop hadoop tests on the VMs after
> > construction as a validation step.
> >
> >
> >
> >
> > On Sat, Jul 6, 2013 at 8:58 PM, Bruno MahИ <bm...@apache.org> wrote:
> >
> > > On 07/06/2013 07:33 AM, Jay Vyas wrote:
> > >
> > >> Hi bigtop:
> > >>
> > >> Are there any permanant builds saved on jenkins (for the VM matrix)?
> > >>
> > >> If not it would be nice to add them for certain known well tested,
> > >> working disk images .
> > >>
> > >> (for context, I'm currently running Mr2 build of the KVM box and it
> > >> appears to have some intermittent write issues on the DataNode path,
> and
> > >> also, my namenode appears to really like being in safe mode. these
> > >> could just be due to VM setup though, as im changing some things like
> > >> adding static IPs and data node write paths... so nothing to be
> alarmed
> > >> about.)
> > >>
> > >> --
> > >> Jay Vyas
> > >> http://jayunit100.blogspot.com
> > >>
> > >
> > > Hi Jay,
> > >
> > > Could you defined "permanent build" ?
> > > I am not sure if this fits your requirement, but jenkins has a link to
> the
> > > latest successful build (ex: http://bigtop01.cloudera.org:**
> > > 8080/job/Bigtop-VM-matrix/BR=**master,KIND=kvm,label=**
> > > fedora16/lastSuccessfulBuild/**artifact/bigtop-vm-kvm-master.**tar.gz<
> http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/BR=master,KIND=kvm,label=fedora16/lastSuccessfulBuild/artifact/bigtop-vm-kvm-master.tar.gz
> >)
> > >
> > > We do not store convenient artifacts of VMs since no one has asked
> about
> > > it before.
> > > So ideally, known well tested working disk images would be the ones
> from
> > > Apache Bigtop releases. But right now, there is not much testing of our
> > > VMs. But any help on that front would be welcome!
> > >
> > > Note also that I added that VM more as a base VM for an Apache Hadoop
> > > cloud image than a developer VM. That's why there is not much in it as
> well
> > > as no desktop pre-configured.
> > > So depending on your needs, we may want to add a new VM or enhance the
> > > current one (also, boxgrinder enables inheritance between appliances).
> > >
> > > Also Boxgrinder is apparently not being maintained anymore. So we may
> want
> > > to look into other VM builders (Oz, etc.)
> > >
> > >
> > > Thanks,
> > > Bruno
> > >
> >
> >
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: permanant builds on jenkins
Posted by Konstantin Boudnik <co...@apache.org>.
[Moving user@ to Bcc: and cross-posting to dev@ This is a more dev discussion,
let's keep it there]
I think there's a confusion of some sort here. Apache normally doesn't provide
binary releases. A release - in ASF - is a source code. While binaries can be
provided as a convenience artifacts it is most often isn't a case.
Now, a Bigtop release is a fixed BOM file that states what versions of what
components were built and tested to work against each other. If you are
looking for particular system packages - it is very easy to get by
make rpm|deb
from a particular release tag. I believe we do publish binary packages for
each release, but you should understand that these are stored on s3 that costs
money. Right now, BT build infrastructure is running on the funds donated by
commercial vendors (Cloudera is the only one so far, others are chiming in as
we speak).
Same goes for a VM: if you need a VM with particular release bits in it: you
just check out the tag and run boxgrinder that will create identical VM each
time you run the build. The VM in question isn't meant for development of BT.
This is a mere test-bench for people who want to try the latest release of
Hadoop based stack and have no skills of going through the whole process by
themselves.
I wish we could keep a copy of VMs for all stable builds, but this is a simple
matter of the costs vs. convenience.
As for #3 below: this is pretty much how our Jenkins infra works. There's a
bunch of slave VMs that are used for package building/installation/testing. If
you have some improvements in mind - feel free to open a JIRA tickets and
contribute a patch/design document/etc. Any help is highly welcome.
Cos
On Sun, Jul 07, 2013 at 12:03PM, Jay Vyas wrote:
> Hi bruno:
>
> 1) I've noticed that https://builds.apache.org/view/A-F/view/Bigtop/ is not
> valid but "https://builds.apache.org/job/Bigtop-trunk/" seems to work. Is
> this an error on the website?
>
> 2) http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/ Seems to build
> direct from HEAD.
>
> Are there actual bigtop releases which aren't on jenkins? Everything seems
> to be centered around the build server, rather than actual frozen,
> downloadable releases.
>
> 3) Regarding testing the VMs. Maybe to start some simple virt-install
> scripts could help confirm that the KVM builds, at least, are working. If
> we could get them working with static IPs then we could even clone down the
> bigtop source code and run bigtop hadoop tests on the VMs after
> construction as a validation step.
>
>
>
>
> On Sat, Jul 6, 2013 at 8:58 PM, Bruno MahИ <bm...@apache.org> wrote:
>
> > On 07/06/2013 07:33 AM, Jay Vyas wrote:
> >
> >> Hi bigtop:
> >>
> >> Are there any permanant builds saved on jenkins (for the VM matrix)?
> >>
> >> If not it would be nice to add them for certain known well tested,
> >> working disk images .
> >>
> >> (for context, I'm currently running Mr2 build of the KVM box and it
> >> appears to have some intermittent write issues on the DataNode path, and
> >> also, my namenode appears to really like being in safe mode. these
> >> could just be due to VM setup though, as im changing some things like
> >> adding static IPs and data node write paths... so nothing to be alarmed
> >> about.)
> >>
> >> --
> >> Jay Vyas
> >> http://jayunit100.blogspot.com
> >>
> >
> > Hi Jay,
> >
> > Could you defined "permanent build" ?
> > I am not sure if this fits your requirement, but jenkins has a link to the
> > latest successful build (ex: http://bigtop01.cloudera.org:**
> > 8080/job/Bigtop-VM-matrix/BR=**master,KIND=kvm,label=**
> > fedora16/lastSuccessfulBuild/**artifact/bigtop-vm-kvm-master.**tar.gz<http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/BR=master,KIND=kvm,label=fedora16/lastSuccessfulBuild/artifact/bigtop-vm-kvm-master.tar.gz>)
> >
> > We do not store convenient artifacts of VMs since no one has asked about
> > it before.
> > So ideally, known well tested working disk images would be the ones from
> > Apache Bigtop releases. But right now, there is not much testing of our
> > VMs. But any help on that front would be welcome!
> >
> > Note also that I added that VM more as a base VM for an Apache Hadoop
> > cloud image than a developer VM. That's why there is not much in it as well
> > as no desktop pre-configured.
> > So depending on your needs, we may want to add a new VM or enhance the
> > current one (also, boxgrinder enables inheritance between appliances).
> >
> > Also Boxgrinder is apparently not being maintained anymore. So we may want
> > to look into other VM builders (Oz, etc.)
> >
> >
> > Thanks,
> > Bruno
> >
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: permanant builds on jenkins
Posted by Konstantin Boudnik <co...@apache.org>.
[Moving user@ to Bcc: and cross-posting to dev@ This is a more dev discussion,
let's keep it there]
I think there's a confusion of some sort here. Apache normally doesn't provide
binary releases. A release - in ASF - is a source code. While binaries can be
provided as a convenience artifacts it is most often isn't a case.
Now, a Bigtop release is a fixed BOM file that states what versions of what
components were built and tested to work against each other. If you are
looking for particular system packages - it is very easy to get by
make rpm|deb
from a particular release tag. I believe we do publish binary packages for
each release, but you should understand that these are stored on s3 that costs
money. Right now, BT build infrastructure is running on the funds donated by
commercial vendors (Cloudera is the only one so far, others are chiming in as
we speak).
Same goes for a VM: if you need a VM with particular release bits in it: you
just check out the tag and run boxgrinder that will create identical VM each
time you run the build. The VM in question isn't meant for development of BT.
This is a mere test-bench for people who want to try the latest release of
Hadoop based stack and have no skills of going through the whole process by
themselves.
I wish we could keep a copy of VMs for all stable builds, but this is a simple
matter of the costs vs. convenience.
As for #3 below: this is pretty much how our Jenkins infra works. There's a
bunch of slave VMs that are used for package building/installation/testing. If
you have some improvements in mind - feel free to open a JIRA tickets and
contribute a patch/design document/etc. Any help is highly welcome.
Cos
On Sun, Jul 07, 2013 at 12:03PM, Jay Vyas wrote:
> Hi bruno:
>
> 1) I've noticed that https://builds.apache.org/view/A-F/view/Bigtop/ is not
> valid but "https://builds.apache.org/job/Bigtop-trunk/" seems to work. Is
> this an error on the website?
>
> 2) http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/ Seems to build
> direct from HEAD.
>
> Are there actual bigtop releases which aren't on jenkins? Everything seems
> to be centered around the build server, rather than actual frozen,
> downloadable releases.
>
> 3) Regarding testing the VMs. Maybe to start some simple virt-install
> scripts could help confirm that the KVM builds, at least, are working. If
> we could get them working with static IPs then we could even clone down the
> bigtop source code and run bigtop hadoop tests on the VMs after
> construction as a validation step.
>
>
>
>
> On Sat, Jul 6, 2013 at 8:58 PM, Bruno MahИ <bm...@apache.org> wrote:
>
> > On 07/06/2013 07:33 AM, Jay Vyas wrote:
> >
> >> Hi bigtop:
> >>
> >> Are there any permanant builds saved on jenkins (for the VM matrix)?
> >>
> >> If not it would be nice to add them for certain known well tested,
> >> working disk images .
> >>
> >> (for context, I'm currently running Mr2 build of the KVM box and it
> >> appears to have some intermittent write issues on the DataNode path, and
> >> also, my namenode appears to really like being in safe mode. these
> >> could just be due to VM setup though, as im changing some things like
> >> adding static IPs and data node write paths... so nothing to be alarmed
> >> about.)
> >>
> >> --
> >> Jay Vyas
> >> http://jayunit100.blogspot.com
> >>
> >
> > Hi Jay,
> >
> > Could you defined "permanent build" ?
> > I am not sure if this fits your requirement, but jenkins has a link to the
> > latest successful build (ex: http://bigtop01.cloudera.org:**
> > 8080/job/Bigtop-VM-matrix/BR=**master,KIND=kvm,label=**
> > fedora16/lastSuccessfulBuild/**artifact/bigtop-vm-kvm-master.**tar.gz<http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/BR=master,KIND=kvm,label=fedora16/lastSuccessfulBuild/artifact/bigtop-vm-kvm-master.tar.gz>)
> >
> > We do not store convenient artifacts of VMs since no one has asked about
> > it before.
> > So ideally, known well tested working disk images would be the ones from
> > Apache Bigtop releases. But right now, there is not much testing of our
> > VMs. But any help on that front would be welcome!
> >
> > Note also that I added that VM more as a base VM for an Apache Hadoop
> > cloud image than a developer VM. That's why there is not much in it as well
> > as no desktop pre-configured.
> > So depending on your needs, we may want to add a new VM or enhance the
> > current one (also, boxgrinder enables inheritance between appliances).
> >
> > Also Boxgrinder is apparently not being maintained anymore. So we may want
> > to look into other VM builders (Oz, etc.)
> >
> >
> > Thanks,
> > Bruno
> >
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: permanant builds on jenkins
Posted by Jay Vyas <ja...@gmail.com>.
Hi bruno:
1) I've noticed that https://builds.apache.org/view/A-F/view/Bigtop/ is not
valid but "https://builds.apache.org/job/Bigtop-trunk/" seems to work. Is
this an error on the website?
2) http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/ Seems to build
direct from HEAD.
Are there actual bigtop releases which aren't on jenkins? Everything seems
to be centered around the build server, rather than actual frozen,
downloadable releases.
3) Regarding testing the VMs. Maybe to start some simple virt-install
scripts could help confirm that the KVM builds, at least, are working. If
we could get them working with static IPs then we could even clone down the
bigtop source code and run bigtop hadoop tests on the VMs after
construction as a validation step.
On Sat, Jul 6, 2013 at 8:58 PM, Bruno Mahé <bm...@apache.org> wrote:
> On 07/06/2013 07:33 AM, Jay Vyas wrote:
>
>> Hi bigtop:
>>
>> Are there any permanant builds saved on jenkins (for the VM matrix)?
>>
>> If not it would be nice to add them for certain known well tested,
>> working disk images .
>>
>> (for context, I'm currently running Mr2 build of the KVM box and it
>> appears to have some intermittent write issues on the DataNode path, and
>> also, my namenode appears to really like being in safe mode. these
>> could just be due to VM setup though, as im changing some things like
>> adding static IPs and data node write paths... so nothing to be alarmed
>> about.)
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>>
>
> Hi Jay,
>
> Could you defined "permanent build" ?
> I am not sure if this fits your requirement, but jenkins has a link to the
> latest successful build (ex: http://bigtop01.cloudera.org:**
> 8080/job/Bigtop-VM-matrix/BR=**master,KIND=kvm,label=**
> fedora16/lastSuccessfulBuild/**artifact/bigtop-vm-kvm-master.**tar.gz<http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/BR=master,KIND=kvm,label=fedora16/lastSuccessfulBuild/artifact/bigtop-vm-kvm-master.tar.gz>)
>
> We do not store convenient artifacts of VMs since no one has asked about
> it before.
> So ideally, known well tested working disk images would be the ones from
> Apache Bigtop releases. But right now, there is not much testing of our
> VMs. But any help on that front would be welcome!
>
> Note also that I added that VM more as a base VM for an Apache Hadoop
> cloud image than a developer VM. That's why there is not much in it as well
> as no desktop pre-configured.
> So depending on your needs, we may want to add a new VM or enhance the
> current one (also, boxgrinder enables inheritance between appliances).
>
> Also Boxgrinder is apparently not being maintained anymore. So we may want
> to look into other VM builders (Oz, etc.)
>
>
> Thanks,
> Bruno
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: permanant builds on jenkins
Posted by Bruno Mahé <bm...@apache.org>.
On 07/06/2013 07:33 AM, Jay Vyas wrote:
> Hi bigtop:
>
> Are there any permanant builds saved on jenkins (for the VM matrix)?
>
> If not it would be nice to add them for certain known well tested,
> working disk images .
>
> (for context, I'm currently running Mr2 build of the KVM box and it
> appears to have some intermittent write issues on the DataNode path, and
> also, my namenode appears to really like being in safe mode. these
> could just be due to VM setup though, as im changing some things like
> adding static IPs and data node write paths... so nothing to be alarmed
> about.)
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Hi Jay,
Could you defined "permanent build" ?
I am not sure if this fits your requirement, but jenkins has a link to
the latest successful build (ex:
http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/BR=master,KIND=kvm,label=fedora16/lastSuccessfulBuild/artifact/bigtop-vm-kvm-master.tar.gz
)
We do not store convenient artifacts of VMs since no one has asked about
it before.
So ideally, known well tested working disk images would be the ones from
Apache Bigtop releases. But right now, there is not much testing of our
VMs. But any help on that front would be welcome!
Note also that I added that VM more as a base VM for an Apache Hadoop
cloud image than a developer VM. That's why there is not much in it as
well as no desktop pre-configured.
So depending on your needs, we may want to add a new VM or enhance the
current one (also, boxgrinder enables inheritance between appliances).
Also Boxgrinder is apparently not being maintained anymore. So we may
want to look into other VM builders (Oz, etc.)
Thanks,
Bruno