You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Konstantinos Tsakalozos <ko...@canonical.com> on 2016/06/07 07:19:25 UTC
Deploy bleeding edge?
Hi everyone,
Is there a way to deploy the bleeding edge of the components Bigtop
packages? This would greatly benefit Apache project developers as well as
users that want to try out the latest features.
Do we have a mirror with deb packages nightly builds that install what is
on the head of the Apache projects offered by Bigtop? We would also need to
grab the head of Bigtop on github, right?
Thank you,
Konstantinos
Re: Deploy bleeding edge?
Posted by Konstantinos Tsakalozos <ko...@canonical.com>.
Thank you for you prompt reply Olaf
On Tue, Jun 7, 2016 at 11:33 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
> Hi,
>
> you can download the last sucessfuly builds of unsigned packages directly
> from the CI Build Bigtop-trunk.
>
> For instance ubuntu
>
>
> https://ci.bigtop.apache.org/job/Bigtop-trunk/BUILD_ENVIRONMENTS=ubuntu-14.04,label=docker-slave/lastSuccessfulBuild/artifact/output/apt/
>
> Please properly urlencode the = char if you automate it.
>
> I would suggest to download as an archive if you want to do regression
> tests. You do not need to compile Bigtop yourself.
> It would be possible to use a dummy signature and sign packages, if needed.
>
> ubuntu-16.04 will follow shortly. Have to add it to the jobs first.
>
> Olaf
>
>
> > Am 07.06.2016 um 09:19 schrieb Konstantinos Tsakalozos <
> kos.tsakalozos@canonical.com>:
> >
> > Hi everyone,
> >
> > Is there a way to deploy the bleeding edge of the components Bigtop
> > packages? This would greatly benefit Apache project developers as well as
> > users that want to try out the latest features.
> >
> > Do we have a mirror with deb packages nightly builds that install what is
> > on the head of the Apache projects offered by Bigtop? We would also need
> to
> > grab the head of Bigtop on github, right?
> >
> > Thank you,
> > Konstantinos
>
>
Re: Deploy bleeding edge?
Posted by Konstantin Boudnik <co...@apache.org>.
I guess this is the best and the only option right now. It would be great to
have a way of actually producing the repos for nightly builds, but considering
own docker-based build environment it will require some thought-through way of
managing docker volumes, so we can collect the artifacts in an easy fashion.
Unfortunately, I don't have any decent ideas on that front. Thoughts?
Cos
On Tue, Jun 07, 2016 at 10:33AM, Olaf Flebbe wrote:
> Hi,
>
> you can download the last sucessfuly builds of unsigned packages directly
> from the CI Build Bigtop-trunk.
>
> For instance ubuntu
>
> https://ci.bigtop.apache.org/job/Bigtop-trunk/BUILD_ENVIRONMENTS=ubuntu-14.04,label=docker-slave/lastSuccessfulBuild/artifact/output/apt/
>
> Please properly urlencode the = char if you automate it.
>
> I would suggest to download as an archive if you want to do regression
> tests. You do not need to compile Bigtop yourself. It would be possible to
> use a dummy signature and sign packages, if needed.
>
> ubuntu-16.04 will follow shortly. Have to add it to the jobs first.
>
> Olaf
>
>
> > Am 07.06.2016 um 09:19 schrieb Konstantinos Tsakalozos <ko...@canonical.com>:
> >
> > Hi everyone,
> >
> > Is there a way to deploy the bleeding edge of the components Bigtop
> > packages? This would greatly benefit Apache project developers as well as
> > users that want to try out the latest features.
> >
> > Do we have a mirror with deb packages nightly builds that install what is
> > on the head of the Apache projects offered by Bigtop? We would also need to
> > grab the head of Bigtop on github, right?
> >
> > Thank you,
> > Konstantinos
>
Re: Deploy bleeding edge?
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi,
you can download the last sucessfuly builds of unsigned packages directly from the CI Build Bigtop-trunk.
For instance ubuntu
https://ci.bigtop.apache.org/job/Bigtop-trunk/BUILD_ENVIRONMENTS=ubuntu-14.04,label=docker-slave/lastSuccessfulBuild/artifact/output/apt/
Please properly urlencode the = char if you automate it.
I would suggest to download as an archive if you want to do regression tests. You do not need to compile Bigtop yourself.
It would be possible to use a dummy signature and sign packages, if needed.
ubuntu-16.04 will follow shortly. Have to add it to the jobs first.
Olaf
> Am 07.06.2016 um 09:19 schrieb Konstantinos Tsakalozos <ko...@canonical.com>:
>
> Hi everyone,
>
> Is there a way to deploy the bleeding edge of the components Bigtop
> packages? This would greatly benefit Apache project developers as well as
> users that want to try out the latest features.
>
> Do we have a mirror with deb packages nightly builds that install what is
> on the head of the Apache projects offered by Bigtop? We would also need to
> grab the head of Bigtop on github, right?
>
> Thank you,
> Konstantinos
Re: Deploy bleeding edge?
Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, Jun 08, 2016 at 05:31PM, Olaf Flebbe wrote:
> Hi,
>
> I like to have Continous Testing/Deployment for bigtop. So I am ++1 to enhance/enable it.
>
> we already procduce new repos as part of our Bigtop-trunk Jenkins job every night. We archive the artifacts generated.
>
> We have issues, right now:
>
> * gradle.org repository is rather unstable breaking 50% of our builds.
> I already addressed this with BIGTOP-2474, since we do not need to access gradle.org from our build infra.
>
> * The repositories are not signed. This can be fixed with a dummy signature.
>
> Right now, we have two different Compile job:
>
> Bigtop-trunk: Compiling all the stack, very unstable.
> Bigtop-packages-trunk: Compiling all packages as a single compile. Does not
> generate a repository. This is intended as an test compile only.
IIRC I've been using packages produced by Bigtop-packages-trunk for releases,
as it gets all of them in the same place. Hence it is easy to collect.
Cos
> I would recommend to stick at the Bigtop-trunk repositories for now.
>
> olaf
>
>
> > Am 08.06.2016 um 07:22 schrieb Konstantin Boudnik <co...@apache.org>:
> >
> > Makes all the sense to me. To me the only question is what we can do in the CI
> > to start producing the repos on each nightly run.
> >
> > Cos
> >
> > On Tue, Jun 07, 2016 at 04:44PM, Antonio Rosales wrote:
> >> On Tuesday, June 7, 2016, Jonathan Kelly <jo...@gmail.com> wrote:
> >>
> >>> Oh, sorry, yes, I misunderstood. Nightly builds of Bigtop trunk would
> >>> certainly make a lot more sense than any sort of nightly build of the
> >>> trunks of each supported application. :)
> >>
> >>
> >> The intent here is to enable charms to deploy a stable release or the
> >> latest Bigtop "bleeding edge." Ideally we would like to run benchmarks and
> >> integration tests on nightly builds to identify issues earlier in the dev
> >> pipeline.
> >>
> >> Would folks find that helpful?
> >>
> >> -Antonio
> >>
> >>
> >>> On Tue, Jun 7, 2016 at 3:24 PM Konstantin Boudnik <cos@apache.org
> >>> <javascript:;>> wrote:
> >>>
> >>>> On Tue, Jun 07, 2016 at 09:08PM, Jonathan Kelly wrote:
> >>>>> I'm not sure this will work very well, since many apps depend upon
> >>> other
> >>>>> apps but have not yet upgraded to support the latest version of their
> >>>>> dependencies.
> >>>>>
> >>>>> For example, Spark 2.0 is going to be released in the next month or so,
> >>>> but
> >>>>> Hive, Mahout, Oozie, Zeppelin and possibly more do not yet support
> >>> Spark
> >>>>> 2.0. As another example, Hive 2.1 will be released somewhat soon too,
> >>> but
> >>>>> some apps might not yet support Hive 2.x, let alone Hive 2.1. (Bigtop
> >>> is
> >>>>> still only on Hive 1.2.1.)
> >>>>
> >>>> I believe Konstantinos' question is about the "bleeding state" of Bigtop
> >>>> stack. Where all you said is completely true - we don't simply leap
> >>> forward
> >>>> and switch a version of a component that is unsupported by the rest of
> >>> the
> >>>> ecosystem.
> >>>>
> >>>> Yet, our trunk has a number of advances compared to any previous release
> >>>> and
> >>>> for some people such deployments might be very desirable.
> >>>>
> >>>> Cos
> >>>>
> >>>>> ~ Jonathan
> >>>>>
> >>>>> On Tue, Jun 7, 2016 at 12:19 AM Konstantinos Tsakalozos <
> >>>>> kos.tsakalozos@canonical.com <javascript:;>> wrote:
> >>>>>
> >>>>>> Hi everyone,
> >>>>>>
> >>>>>> Is there a way to deploy the bleeding edge of the components Bigtop
> >>>>>> packages? This would greatly benefit Apache project developers as
> >>> well
> >>>> as
> >>>>>> users that want to try out the latest features.
> >>>>>>
> >>>>>> Do we have a mirror with deb packages nightly builds that install
> >>> what
> >>>> is
> >>>>>> on the head of the Apache projects offered by Bigtop? We would also
> >>>> need to
> >>>>>> grab the head of Bigtop on github, right?
> >>>>>>
> >>>>>> Thank you,
> >>>>>> Konstantinos
> >>>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> -Thanks
> >> Antonio
>
Re: Deploy bleeding edge?
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi,
I like to have Continous Testing/Deployment for bigtop. So I am ++1 to enhance/enable it.
we already procduce new repos as part of our Bigtop-trunk Jenkins job every night. We archive the artifacts generated.
We have issues, right now:
* gradle.org repository is rather unstable breaking 50% of our builds.
I already addressed this with BIGTOP-2474, since we do not need to access gradle.org from our build infra.
* The repositories are not signed. This can be fixed with a dummy signature.
Right now, we have two different Compile job:
Bigtop-trunk: Compiling all the stack, very unstable.
Bigtop-packages-trunk: Compiling all packages as a single compile. Does not generate a repository. This is intended as an test compile only.
I would recommend to stick at the Bigtop-trunk repositories for now.
olaf
> Am 08.06.2016 um 07:22 schrieb Konstantin Boudnik <co...@apache.org>:
>
> Makes all the sense to me. To me the only question is what we can do in the CI
> to start producing the repos on each nightly run.
>
> Cos
>
> On Tue, Jun 07, 2016 at 04:44PM, Antonio Rosales wrote:
>> On Tuesday, June 7, 2016, Jonathan Kelly <jo...@gmail.com> wrote:
>>
>>> Oh, sorry, yes, I misunderstood. Nightly builds of Bigtop trunk would
>>> certainly make a lot more sense than any sort of nightly build of the
>>> trunks of each supported application. :)
>>
>>
>> The intent here is to enable charms to deploy a stable release or the
>> latest Bigtop "bleeding edge." Ideally we would like to run benchmarks and
>> integration tests on nightly builds to identify issues earlier in the dev
>> pipeline.
>>
>> Would folks find that helpful?
>>
>> -Antonio
>>
>>
>>> On Tue, Jun 7, 2016 at 3:24 PM Konstantin Boudnik <cos@apache.org
>>> <javascript:;>> wrote:
>>>
>>>> On Tue, Jun 07, 2016 at 09:08PM, Jonathan Kelly wrote:
>>>>> I'm not sure this will work very well, since many apps depend upon
>>> other
>>>>> apps but have not yet upgraded to support the latest version of their
>>>>> dependencies.
>>>>>
>>>>> For example, Spark 2.0 is going to be released in the next month or so,
>>>> but
>>>>> Hive, Mahout, Oozie, Zeppelin and possibly more do not yet support
>>> Spark
>>>>> 2.0. As another example, Hive 2.1 will be released somewhat soon too,
>>> but
>>>>> some apps might not yet support Hive 2.x, let alone Hive 2.1. (Bigtop
>>> is
>>>>> still only on Hive 1.2.1.)
>>>>
>>>> I believe Konstantinos' question is about the "bleeding state" of Bigtop
>>>> stack. Where all you said is completely true - we don't simply leap
>>> forward
>>>> and switch a version of a component that is unsupported by the rest of
>>> the
>>>> ecosystem.
>>>>
>>>> Yet, our trunk has a number of advances compared to any previous release
>>>> and
>>>> for some people such deployments might be very desirable.
>>>>
>>>> Cos
>>>>
>>>>> ~ Jonathan
>>>>>
>>>>> On Tue, Jun 7, 2016 at 12:19 AM Konstantinos Tsakalozos <
>>>>> kos.tsakalozos@canonical.com <javascript:;>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Is there a way to deploy the bleeding edge of the components Bigtop
>>>>>> packages? This would greatly benefit Apache project developers as
>>> well
>>>> as
>>>>>> users that want to try out the latest features.
>>>>>>
>>>>>> Do we have a mirror with deb packages nightly builds that install
>>> what
>>>> is
>>>>>> on the head of the Apache projects offered by Bigtop? We would also
>>>> need to
>>>>>> grab the head of Bigtop on github, right?
>>>>>>
>>>>>> Thank you,
>>>>>> Konstantinos
>>>>>>
>>>>
>>>
>>
>>
>> --
>> -Thanks
>> Antonio
Re: Deploy bleeding edge?
Posted by Konstantin Boudnik <co...@apache.org>.
Makes all the sense to me. To me the only question is what we can do in the CI
to start producing the repos on each nightly run.
Cos
On Tue, Jun 07, 2016 at 04:44PM, Antonio Rosales wrote:
> On Tuesday, June 7, 2016, Jonathan Kelly <jo...@gmail.com> wrote:
>
> > Oh, sorry, yes, I misunderstood. Nightly builds of Bigtop trunk would
> > certainly make a lot more sense than any sort of nightly build of the
> > trunks of each supported application. :)
>
>
> The intent here is to enable charms to deploy a stable release or the
> latest Bigtop "bleeding edge." Ideally we would like to run benchmarks and
> integration tests on nightly builds to identify issues earlier in the dev
> pipeline.
>
> Would folks find that helpful?
>
> -Antonio
>
>
> > On Tue, Jun 7, 2016 at 3:24 PM Konstantin Boudnik <cos@apache.org
> > <javascript:;>> wrote:
> >
> > > On Tue, Jun 07, 2016 at 09:08PM, Jonathan Kelly wrote:
> > > > I'm not sure this will work very well, since many apps depend upon
> > other
> > > > apps but have not yet upgraded to support the latest version of their
> > > > dependencies.
> > > >
> > > > For example, Spark 2.0 is going to be released in the next month or so,
> > > but
> > > > Hive, Mahout, Oozie, Zeppelin and possibly more do not yet support
> > Spark
> > > > 2.0. As another example, Hive 2.1 will be released somewhat soon too,
> > but
> > > > some apps might not yet support Hive 2.x, let alone Hive 2.1. (Bigtop
> > is
> > > > still only on Hive 1.2.1.)
> > >
> > > I believe Konstantinos' question is about the "bleeding state" of Bigtop
> > > stack. Where all you said is completely true - we don't simply leap
> > forward
> > > and switch a version of a component that is unsupported by the rest of
> > the
> > > ecosystem.
> > >
> > > Yet, our trunk has a number of advances compared to any previous release
> > > and
> > > for some people such deployments might be very desirable.
> > >
> > > Cos
> > >
> > > > ~ Jonathan
> > > >
> > > > On Tue, Jun 7, 2016 at 12:19 AM Konstantinos Tsakalozos <
> > > > kos.tsakalozos@canonical.com <javascript:;>> wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Is there a way to deploy the bleeding edge of the components Bigtop
> > > > > packages? This would greatly benefit Apache project developers as
> > well
> > > as
> > > > > users that want to try out the latest features.
> > > > >
> > > > > Do we have a mirror with deb packages nightly builds that install
> > what
> > > is
> > > > > on the head of the Apache projects offered by Bigtop? We would also
> > > need to
> > > > > grab the head of Bigtop on github, right?
> > > > >
> > > > > Thank you,
> > > > > Konstantinos
> > > > >
> > >
> >
>
>
> --
> -Thanks
> Antonio
Re: Deploy bleeding edge?
Posted by Antonio Rosales <an...@canonical.com>.
On Tuesday, June 7, 2016, Jonathan Kelly <jo...@gmail.com> wrote:
> Oh, sorry, yes, I misunderstood. Nightly builds of Bigtop trunk would
> certainly make a lot more sense than any sort of nightly build of the
> trunks of each supported application. :)
The intent here is to enable charms to deploy a stable release or the
latest Bigtop "bleeding edge." Ideally we would like to run benchmarks and
integration tests on nightly builds to identify issues earlier in the dev
pipeline.
Would folks find that helpful?
-Antonio
> On Tue, Jun 7, 2016 at 3:24 PM Konstantin Boudnik <cos@apache.org
> <javascript:;>> wrote:
>
> > On Tue, Jun 07, 2016 at 09:08PM, Jonathan Kelly wrote:
> > > I'm not sure this will work very well, since many apps depend upon
> other
> > > apps but have not yet upgraded to support the latest version of their
> > > dependencies.
> > >
> > > For example, Spark 2.0 is going to be released in the next month or so,
> > but
> > > Hive, Mahout, Oozie, Zeppelin and possibly more do not yet support
> Spark
> > > 2.0. As another example, Hive 2.1 will be released somewhat soon too,
> but
> > > some apps might not yet support Hive 2.x, let alone Hive 2.1. (Bigtop
> is
> > > still only on Hive 1.2.1.)
> >
> > I believe Konstantinos' question is about the "bleeding state" of Bigtop
> > stack. Where all you said is completely true - we don't simply leap
> forward
> > and switch a version of a component that is unsupported by the rest of
> the
> > ecosystem.
> >
> > Yet, our trunk has a number of advances compared to any previous release
> > and
> > for some people such deployments might be very desirable.
> >
> > Cos
> >
> > > ~ Jonathan
> > >
> > > On Tue, Jun 7, 2016 at 12:19 AM Konstantinos Tsakalozos <
> > > kos.tsakalozos@canonical.com <javascript:;>> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Is there a way to deploy the bleeding edge of the components Bigtop
> > > > packages? This would greatly benefit Apache project developers as
> well
> > as
> > > > users that want to try out the latest features.
> > > >
> > > > Do we have a mirror with deb packages nightly builds that install
> what
> > is
> > > > on the head of the Apache projects offered by Bigtop? We would also
> > need to
> > > > grab the head of Bigtop on github, right?
> > > >
> > > > Thank you,
> > > > Konstantinos
> > > >
> >
>
--
-Thanks
Antonio
Re: Deploy bleeding edge?
Posted by Jonathan Kelly <jo...@gmail.com>.
Oh, sorry, yes, I misunderstood. Nightly builds of Bigtop trunk would
certainly make a lot more sense than any sort of nightly build of the
trunks of each supported application. :)
On Tue, Jun 7, 2016 at 3:24 PM Konstantin Boudnik <co...@apache.org> wrote:
> On Tue, Jun 07, 2016 at 09:08PM, Jonathan Kelly wrote:
> > I'm not sure this will work very well, since many apps depend upon other
> > apps but have not yet upgraded to support the latest version of their
> > dependencies.
> >
> > For example, Spark 2.0 is going to be released in the next month or so,
> but
> > Hive, Mahout, Oozie, Zeppelin and possibly more do not yet support Spark
> > 2.0. As another example, Hive 2.1 will be released somewhat soon too, but
> > some apps might not yet support Hive 2.x, let alone Hive 2.1. (Bigtop is
> > still only on Hive 1.2.1.)
>
> I believe Konstantinos' question is about the "bleeding state" of Bigtop
> stack. Where all you said is completely true - we don't simply leap forward
> and switch a version of a component that is unsupported by the rest of the
> ecosystem.
>
> Yet, our trunk has a number of advances compared to any previous release
> and
> for some people such deployments might be very desirable.
>
> Cos
>
> > ~ Jonathan
> >
> > On Tue, Jun 7, 2016 at 12:19 AM Konstantinos Tsakalozos <
> > kos.tsakalozos@canonical.com> wrote:
> >
> > > Hi everyone,
> > >
> > > Is there a way to deploy the bleeding edge of the components Bigtop
> > > packages? This would greatly benefit Apache project developers as well
> as
> > > users that want to try out the latest features.
> > >
> > > Do we have a mirror with deb packages nightly builds that install what
> is
> > > on the head of the Apache projects offered by Bigtop? We would also
> need to
> > > grab the head of Bigtop on github, right?
> > >
> > > Thank you,
> > > Konstantinos
> > >
>
Re: Deploy bleeding edge?
Posted by Konstantin Boudnik <co...@apache.org>.
On Tue, Jun 07, 2016 at 09:08PM, Jonathan Kelly wrote:
> I'm not sure this will work very well, since many apps depend upon other
> apps but have not yet upgraded to support the latest version of their
> dependencies.
>
> For example, Spark 2.0 is going to be released in the next month or so, but
> Hive, Mahout, Oozie, Zeppelin and possibly more do not yet support Spark
> 2.0. As another example, Hive 2.1 will be released somewhat soon too, but
> some apps might not yet support Hive 2.x, let alone Hive 2.1. (Bigtop is
> still only on Hive 1.2.1.)
I believe Konstantinos' question is about the "bleeding state" of Bigtop
stack. Where all you said is completely true - we don't simply leap forward
and switch a version of a component that is unsupported by the rest of the
ecosystem.
Yet, our trunk has a number of advances compared to any previous release and
for some people such deployments might be very desirable.
Cos
> ~ Jonathan
>
> On Tue, Jun 7, 2016 at 12:19 AM Konstantinos Tsakalozos <
> kos.tsakalozos@canonical.com> wrote:
>
> > Hi everyone,
> >
> > Is there a way to deploy the bleeding edge of the components Bigtop
> > packages? This would greatly benefit Apache project developers as well as
> > users that want to try out the latest features.
> >
> > Do we have a mirror with deb packages nightly builds that install what is
> > on the head of the Apache projects offered by Bigtop? We would also need to
> > grab the head of Bigtop on github, right?
> >
> > Thank you,
> > Konstantinos
> >
Re: Deploy bleeding edge?
Posted by Jonathan Kelly <jo...@gmail.com>.
I'm not sure this will work very well, since many apps depend upon other
apps but have not yet upgraded to support the latest version of their
dependencies.
For example, Spark 2.0 is going to be released in the next month or so, but
Hive, Mahout, Oozie, Zeppelin and possibly more do not yet support Spark
2.0. As another example, Hive 2.1 will be released somewhat soon too, but
some apps might not yet support Hive 2.x, let alone Hive 2.1. (Bigtop is
still only on Hive 1.2.1.)
~ Jonathan
On Tue, Jun 7, 2016 at 12:19 AM Konstantinos Tsakalozos <
kos.tsakalozos@canonical.com> wrote:
> Hi everyone,
>
> Is there a way to deploy the bleeding edge of the components Bigtop
> packages? This would greatly benefit Apache project developers as well as
> users that want to try out the latest features.
>
> Do we have a mirror with deb packages nightly builds that install what is
> on the head of the Apache projects offered by Bigtop? We would also need to
> grab the head of Bigtop on github, right?
>
> Thank you,
> Konstantinos
>