You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by Olaf Flebbe <of...@oflebbe.de> on 2015/03/03 09:57:29 UTC
Re: Update on meetup?
Hi
Too bad that I missed meetup and start of discussion, because I only monitored dev list...
We deploy our testcluster on a single machine as pure lxc containers. Using a hiera supported puppet michel weiser is trying to merge with bigtop.
Imho one solution does not fits all.
In productiom:On a cluster of machines you would deploy bigtop on bare metal, not sacrifing runtime performance for anything.
In a longer running testcluster on a maschine:
We run a traditional os in container (i.e. We run systemd formely known as init in it). This has the big advantage that we can test long running things like kerberos tickets expiring happening on a real bare metal cluster. Right now we have journalling, with tez on hive including hue and are happy right now when we can switch on kerberos authentication.
For smoke tests the vagrant/docker scripts are really cool, i looked at it yesterday the first time. Too bad they are deeply bound to centos, I had to change many thing to get it run on debian.
Docker is more centered
I looked into the vm puppet scripts and
Sent with SMTP
Am 30 Jan 2015 um 20:10 schrieb jay vyas <jayunit100.apache@gmail.com <ma...@gmail.com>>:
> Roman youre right, a real docker container doesn't do things the way we currently do in bigtop.
>
> If we embrace docker, we will be able to really drop alot of weight.
>
> Maybe the experimental bigtop 1x can deploy pure tarballs from apache distributions on apache mesos for the orchestration ?
>
> That would be verrrrrrrryyyyyyyyyyyy interesting .....
>
> On Fri, Jan 30, 2015 at 12:45 PM, Roman Shaposhnik <roman@shaposhnik.org <ma...@shaposhnik.org>> wrote:
> On Thu, Jan 29, 2015 at 7:09 AM, Evans Ye <evansye@apache.org <ma...@apache.org>> wrote:
> > I really like the idea to deploy hadoop stack in docker containers.
> > Just want to catch up with you guys, are you planning to ship docker images
> > as a drop and run cluster deployment? or no? Anyhow that was just my guess,
> > I'm interesting in the challenge of this cool stuff :)
>
> That's a $64k question. On one hand, a kosher docker container has just
> one single service inside of it. We can do that very easily. Then the real
> questions kick in: how do you orchestrate those, how do you provide
> them with configuration, etc. Seems like you still need some kind of
> a system like Amabari, etc.
>
> Thanks,
> Roman.
>
>
>
> --
> jay vyas
Re: Update on meetup?
Posted by jay vyas <ja...@gmail.com>.
absolutely supportive of olaf's ideas to make the docker support more
platform agnostic. We don't want to tie folks down - bigtop's mission is
to be open to solutions from all vendors !
On Tue, Mar 3, 2015 at 12:16 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Olaf,
>
> that's really pity that our docker stuff is CentOS specific, which is I
> guess
> expected when RedHat guys are working on them ;) Do you think they can be
> made
> more generic? After all, we use the same Puppet for any supported
> platform, so
> I don't see why we shouldn't be able to have docker env. that works for any
> containerised OS.
>
> Cos
>
> On Tue, Mar 03, 2015 at 09:57AM, Olaf Flebbe wrote:
> > Hi
> >
> > Too bad that I missed meetup and start of discussion, because I only
> monitored dev list...
> >
> > We deploy our testcluster on a single machine as pure lxc containers.
> Using
> > a hiera supported puppet michel weiser is trying to merge with bigtop.
> >
> > Imho one solution does not fits all.
> >
> > In productiom: On a cluster of machines you would deploy bigtop on bare
> > metal, not sacrifing runtime performance for anything.
> >
> > In a longer running testcluster on a maschine:
> > We run a traditional os in container (i.e. We run systemd formely known
> as
> > init in it). This has the big advantage that we can test long running
> things
> > like kerberos tickets expiring happening on a real bare metal cluster.
> Right
> > now we have journalling, with tez on hive including hue and are happy
> right
> > now when we can switch on kerberos authentication.
> >
> > For smoke tests the vagrant/docker scripts are really cool, i looked at
> it
> > yesterday the first time. Too bad they are deeply bound to centos, I had
> to
> > change many thing to get it run on debian.
> >
> >
> > Docker is more centered
> >
> > I looked into the vm puppet scripts and
> >
> >
> > Sent with SMTP
> >
> > Am 30 Jan 2015 um 20:10 schrieb jay vyas <jayunit100.apache@gmail.com
> <ma...@gmail.com>>:
> >
> > > Roman youre right, a real docker container doesn't do things the way
> we currently do in bigtop.
> > >
> > > If we embrace docker, we will be able to really drop alot of weight.
> > >
> > > Maybe the experimental bigtop 1x can deploy pure tarballs from apache
> distributions on apache mesos for the orchestration ?
> > >
> > > That would be verrrrrrrryyyyyyyyyyyy interesting .....
> > >
> > > On Fri, Jan 30, 2015 at 12:45 PM, Roman Shaposhnik <
> roman@shaposhnik.org <ma...@shaposhnik.org>> wrote:
> > > On Thu, Jan 29, 2015 at 7:09 AM, Evans Ye <evansye@apache.org <mailto:
> evansye@apache.org>> wrote:
> > > > I really like the idea to deploy hadoop stack in docker containers.
> > > > Just want to catch up with you guys, are you planning to ship docker
> images
> > > > as a drop and run cluster deployment? or no? Anyhow that was just my
> guess,
> > > > I'm interesting in the challenge of this cool stuff :)
> > >
> > > That's a $64k question. On one hand, a kosher docker container has just
> > > one single service inside of it. We can do that very easily. Then the
> real
> > > questions kick in: how do you orchestrate those, how do you provide
> > > them with configuration, etc. Seems like you still need some kind of
> > > a system like Amabari, etc.
> > >
> > > Thanks,
> > > Roman.
> > >
> > >
> > >
> > > --
> > > jay vyas
>
>
--
jay vyas
Re: Update on meetup?
Posted by Konstantin Boudnik <co...@apache.org>.
Olaf,
that's really pity that our docker stuff is CentOS specific, which is I guess
expected when RedHat guys are working on them ;) Do you think they can be made
more generic? After all, we use the same Puppet for any supported platform, so
I don't see why we shouldn't be able to have docker env. that works for any
containerised OS.
Cos
On Tue, Mar 03, 2015 at 09:57AM, Olaf Flebbe wrote:
> Hi
>
> Too bad that I missed meetup and start of discussion, because I only monitored dev list...
>
> We deploy our testcluster on a single machine as pure lxc containers. Using
> a hiera supported puppet michel weiser is trying to merge with bigtop.
>
> Imho one solution does not fits all.
>
> In productiom: On a cluster of machines you would deploy bigtop on bare
> metal, not sacrifing runtime performance for anything.
>
> In a longer running testcluster on a maschine:
> We run a traditional os in container (i.e. We run systemd formely known as
> init in it). This has the big advantage that we can test long running things
> like kerberos tickets expiring happening on a real bare metal cluster. Right
> now we have journalling, with tez on hive including hue and are happy right
> now when we can switch on kerberos authentication.
>
> For smoke tests the vagrant/docker scripts are really cool, i looked at it
> yesterday the first time. Too bad they are deeply bound to centos, I had to
> change many thing to get it run on debian.
>
>
> Docker is more centered
>
> I looked into the vm puppet scripts and
>
>
> Sent with SMTP
>
> Am 30 Jan 2015 um 20:10 schrieb jay vyas <jayunit100.apache@gmail.com <ma...@gmail.com>>:
>
> > Roman youre right, a real docker container doesn't do things the way we currently do in bigtop.
> >
> > If we embrace docker, we will be able to really drop alot of weight.
> >
> > Maybe the experimental bigtop 1x can deploy pure tarballs from apache distributions on apache mesos for the orchestration ?
> >
> > That would be verrrrrrrryyyyyyyyyyyy interesting .....
> >
> > On Fri, Jan 30, 2015 at 12:45 PM, Roman Shaposhnik <roman@shaposhnik.org <ma...@shaposhnik.org>> wrote:
> > On Thu, Jan 29, 2015 at 7:09 AM, Evans Ye <evansye@apache.org <ma...@apache.org>> wrote:
> > > I really like the idea to deploy hadoop stack in docker containers.
> > > Just want to catch up with you guys, are you planning to ship docker images
> > > as a drop and run cluster deployment? or no? Anyhow that was just my guess,
> > > I'm interesting in the challenge of this cool stuff :)
> >
> > That's a $64k question. On one hand, a kosher docker container has just
> > one single service inside of it. We can do that very easily. Then the real
> > questions kick in: how do you orchestrate those, how do you provide
> > them with configuration, etc. Seems like you still need some kind of
> > a system like Amabari, etc.
> >
> > Thanks,
> > Roman.
> >
> >
> >
> > --
> > jay vyas