You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by Antoine DUBOIS <an...@cc.in2p3.fr> on 2021/03/30 09:25:18 UTC
Bigtop Use case
Hello,
I've been working to deploy multiple Apache big data products (Yarn, Spark, Zeppelin, Hive, Ranger) not HDFS
So far I've been deploying it manually and it works.
I'm now aiming at a production infrastructure with automated deployement.
I'm wondering if Apache bigtop can help us to deploy such cluster and if possible how ?
Is it simply a way to build RPM with choosen version or can it help us deploy the full cluster with proper configuration depending on server roles ?
It's a bit confusing in my head what is exactly Apache bigtop.
Have a very nice day.
Antoine
Re: Bigtop Use case
Posted by Antoine DUBOIS <an...@cc.in2p3.fr>.
Thank you Evans and Cos,
nice answers.
I'm currently working with traditional VMs and have not access to an easy way of doing stuff.
I'm also a bit surprise that bigtop do not implement ranger or knox.
We're working with puppet though, I might give it a shot as there already a ton of work put in BigTop.
Thank you and have a nice day.
Antoine
De: "Evans Ye" <ev...@apache.org>
À: user@bigtop.apache.org
Envoyé: Mardi 30 Mars 2021 19:56:13
Objet: Re: Bigtop Use case
As Cos said, the docker provisioner gives you the idea of how bigtop deployment works. The Bigtop Puppet deployment solution can be applied to a real cluster as well.
If you'd like to set up a cluster with advanced features such as NameNode HA and Kerberos. You can start editing the configuration and testing via docker provisioner to verify your configuration.
- Evans
Konstantin Boudnik < [ mailto:cos@apache.org | cos@apache.org ] > 於 2021年3月30日 週二 下午9:20寫道：
Hey Antoine.
Indeed, Bigtop has all the tools for automatic cluster deployment
according to the roles of the nodes. In fact, this is exactly how we
test Bigtop in our CI/CD environment with docker provisioner. Same would
apply to actual hardware based cluster.
Please check bigtop-deploy/puppet/README.md and other content in that
directory for more information.
There's also a way to deploy clusters with Ubuntu Juju, but I am not
sure how up-to-date that solution is.
Cheers.
--
With regards,
Cos
On 30.03.2021 12:25, Antoine DUBOIS wrote:
> Hello,
>
> I've been working to deploy multiple Apache big data products (Yarn,
> Spark, Zeppelin, Hive, Ranger) not HDFS
> So far I've been deploying it manually and it works.
> I'm now aiming at a production infrastructure with automated deployement.
> I'm wondering if Apache bigtop can help us to deploy such cluster and if
> possible how ?
> Is it simply a way to build RPM with choosen version or can it help us
> deploy the full cluster with proper configuration depending on server
> roles ?
> It's a bit confusing in my head what is exactly Apache bigtop.
>
> Have a very nice day.
>
> Antoine
Re: Bigtop Use case
Posted by Evans Ye <ev...@apache.org>.
As Cos said, the docker provisioner gives you the idea of how bigtop
deployment works. The Bigtop Puppet deployment solution can be applied to a
real cluster as well.
If you'd like to set up a cluster with advanced features such as NameNode
HA and Kerberos. You can start editing the configuration and testing via
docker provisioner to verify your configuration.
- Evans
Konstantin Boudnik <co...@apache.org> 於 2021年3月30日 週二 下午9:20寫道：
> Hey Antoine.
>
> Indeed, Bigtop has all the tools for automatic cluster deployment
> according to the roles of the nodes. In fact, this is exactly how we
> test Bigtop in our CI/CD environment with docker provisioner. Same would
> apply to actual hardware based cluster.
>
> Please check bigtop-deploy/puppet/README.md and other content in that
> directory for more information.
>
> There's also a way to deploy clusters with Ubuntu Juju, but I am not
> sure how up-to-date that solution is.
>
> Cheers.
> --
> With regards,
> Cos
>
> On 30.03.2021 12:25, Antoine DUBOIS wrote:
> > Hello,
> >
> > I've been working to deploy multiple Apache big data products (Yarn,
> > Spark, Zeppelin, Hive, Ranger) not HDFS
> > So far I've been deploying it manually and it works.
> > I'm now aiming at a production infrastructure with automated deployement.
> > I'm wondering if Apache bigtop can help us to deploy such cluster and if
> > possible how ?
> > Is it simply a way to build RPM with choosen version or can it help us
> > deploy the full cluster with proper configuration depending on server
> > roles ?
> > It's a bit confusing in my head what is exactly Apache bigtop.
> >
> > Have a very nice day.
> >
> > Antoine
>
Re: Bigtop Use case
Posted by Konstantin Boudnik <co...@apache.org>.
Hey Antoine.
Indeed, Bigtop has all the tools for automatic cluster deployment
according to the roles of the nodes. In fact, this is exactly how we
test Bigtop in our CI/CD environment with docker provisioner. Same would
apply to actual hardware based cluster.
Please check bigtop-deploy/puppet/README.md and other content in that
directory for more information.
There's also a way to deploy clusters with Ubuntu Juju, but I am not
sure how up-to-date that solution is.
Cheers.
--
With regards,
Cos
On 30.03.2021 12:25, Antoine DUBOIS wrote:
> Hello,
>
> I've been working to deploy multiple Apache big data products (Yarn,
> Spark, Zeppelin, Hive, Ranger) not HDFS
> So far I've been deploying it manually and it works.
> I'm now aiming at a production infrastructure with automated deployement.
> I'm wondering if Apache bigtop can help us to deploy such cluster and if
> possible how ?
> Is it simply a way to build RPM with choosen version or can it help us
> deploy the full cluster with proper configuration depending on server
> roles ?
> It's a bit confusing in my head what is exactly Apache bigtop.
>
> Have a very nice day.
>
> Antoine