You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Kevin Monroe <ke...@canonical.com> on 2016/05/10 01:35:13 UTC
Bigtop and Juju
Hi Bigtop Devs!
We’ve been working on an Open Source way to model distributed systems like
Big Data. We initially built Apache packages (like Hadoop and Spark) from
scratch, but quickly realized that leveraging Bigtop’s experience and
packaging was a smarter approach. Using Bigtop has drastically improved
the development, maintainability, and feature set of our project, so thanks
for the solid work done here in the community :-)
The project we’re working on is Juju[0], and more specifically, Juju
charms[1] for Bigtop components. Charms are essentially operational code
that wrap big software like Hadoop. We are excited to share our recent work
where we focus on using Bigtop as the foundation of the Juju big data
charms. We’ve bundled a mapreduce solution (along with ganglia monitoring)
here:
https://jujucharms.com/u/bigdata-dev/bigtop-processing-mapreduce
With charms being service orientated and repeatable, we have also developed
an interesting way to get insights on how Bigtop-based solutions are
performing across a given set of clouds or local containers. Check out
example runs of a “cloud weather report” for the Bigtop bundle on GCE and
AWS: https://goo.gl/TPbXaj
These pages are static HTML generated by:
https://github.com/juju-solutions/cloud-weather-report/
We’d like to propose making these reports available alongside the Bigtop CI
(with Canonical providing the cloud resources), as in:
https://ci.bigtop.apache.org/<solution-name>
In addition, we would like contribute the charms and bundles per the
following PR:
https://github.com/apache/bigtop/pull/108
We are excited to be part of the Bigtop community and appreciate the
consistency displayed and effort involved in packaging something as complex
as the big data ecosystem. If you would like to discuss this more
in-person and you are attending Apache Big Data in Vancouver, we will be
meeting at 5pm Tuesday (May 10) in Prince of Wales room, 3rd floor at the
Hyatt past the bathrooms.
We look forward to discussing this further, either on-list or in person
this week. Thanks!
-Kevin Monroe
[0] - Juju is an open-source modeling language that captures operational
intelligence for (big) software like Big Data -- things like life cycle,
configuration, scaling, and intra-service relationships for a given piece
of software.
[1] - Charms are at the heart of Juju modeling. They encapsulate
software’s operational code in repeatably deployable packages.
Re: Bigtop and Juju
Posted by Konstantin Boudnik <co...@apache.org>.
On Mon, May 23, 2016 at 09:29PM, Kevin Monroe wrote:
> > Regarding license:
> >
> > The copyright issues have to be resolved. We can only bring things in,
> > when the RAT Test ("gradle rat") does no complain about licenses any more.
> >
>
> The issues that we introduced have been resolved in PR 108 (note the
> careful wording there)... I do see an unapproved license when I run
> gradlew rat:
>
> /home/ubuntu/bigtop/docker/bigtop-slaves/ubuntu-14.04-aarch64/Dockerfile
>
> However, we didn't introduce this Dockerfile with this PR, so I'm not sure
> if we should "fix" it. Guidance would be much appreciated!
>
Yup, it has been broken earlier and is reported in BIGTOP-2412. But clearly
someone doesn't give a damn about bugs assigned :/
Cos
Re: Bigtop and Juju
Posted by Konstantinos Tsakalozos <ko...@canonical.com>.
Hi Alexander,
We certainly have plans for charming and bundling other Bigtop services
(including Spark, Kafka, etc). We are interested in feedback to help us
prioritize our development. We know Spark is a big one, but are there other
services you (or anyone else from the community) would like to see charmed
first?
Your input is much appreciated. Thanks,
Konstantinos
On Tue, May 24, 2016 at 8:37 AM, Alexander Bezzubov <bz...@apache.org> wrote:
> Hi,
>
> great to see this initiative going!
>
> I'm sorry if my question is something that is not particularly relevant to
> the subject of this discussion, although I assume it might be, so are there
> any plans or existing bundles for apache spark, using hadoop-processing
> bundle introduced in BIGTOP-2435 [1]?
>
> 1. https://issues.apache.org/jira/browse/BIGTOP-2435
>
>
> --
> Kind regards,
> Alexander
>
>
> On Tue, May 24, 2016 at 11:29 AM, Kevin Monroe <kevin.monroe@canonical.com
> >
> wrote:
>
> > Hi Olaf,
> >
> > I've replied to your points below, but I'm also good with moving the
> > discussion to the jira (
> https://issues.apache.org/jira/browse/BIGTOP-2435)
> > if that make more sense.
> >
> > On Mon, May 16, 2016 at 11:04 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
> >
> > > Hi,
> > >
> > > nice to have you all on board. Sorry I only do not know enough about
> > juju.
> > >
> > > Regarding the structure:
> > >
> > > We like to have JIRA's referencing only one git commit object , with
> the
> > > commit message mirroring the JIRA.
> > >
> > > Since your contribution is quite large I like to have it split in a few
> > > independent contributions. Maybe the bigtop-packages/src/charm for
> > instance
> > > into one JIRA .
> > >
> >
> > We've updated the PR to address concerns after a bit of feedback in
> > #bigtop. As of now, the proposal still spans 3 main subdirs (with a
> touch
> > on NOTICE, build.gradle, and pom.xml):
> >
> > bigtop-deploy/juju
> > bigtop-packages/src/charm
> > bigtop-tests/cloud-weather-report
> >
> > While the PR is large, it is fairly compartmentalized into an overall
> "juju
> > enablement" goal, and should have no effect on sibling sources. That
> said,
> > let me know if splitting the patch by subdir with a JIRA for each
> affected
> > component would still be more preferable.
> >
> > Is there any specific reason why you placed charm under the
> bigtop-packages
> > > directory ? It looks to me more like some orchestration data rather
> than
> > > packaging info.
> > >
> >
> > Charms are the "packages" in Juju parlance. They're comprised of
> software
> > (stuff you would apt install) plus operational code (how that software
> > should scale, react to other software, configure itself based on
> hardware,
> > etc). This isn't quite the same as orchestration data because a charm
> > (like a deb or rpm) has no idea where it should live nor what its host
> will
> > look like ahead of time -- it only knows how to react once an
> orchestrator
> > puts it somewhere.
> >
> > We do have a simple way to bundle collections of charms into deployment
> > models. We cleverly call these "bundles", an example of which is
> committed
> > to ./bigtop-deploy/juju (https://jujucharms.com/hadoop-processing/). To
> > round out the PR, we also chose ./bigtop-tests/cloud-weather-report as
> the
> > location for test plans that would drive a simple red/green UI to check
> the
> > health of bundle deployments across clouds (
> > http://status.juju.solutions/recent).
> >
> > Hopefully that helps explain the reasoning behind these locations, but
> I'm
> > happy to address concerns with any of our choices!
> >
> >
> > > Regarding license:
> > >
> > > The copyright issues have to be resolved. We can only bring things in,
> > > when the RAT Test ("gradle rat") does no complain about licenses any
> > more.
> > >
> >
> > The issues that we introduced have been resolved in PR 108 (note the
> > careful wording there)... I do see an unapproved license when I run
> > gradlew rat:
> >
> > /home/ubuntu/bigtop/docker/bigtop-slaves/ubuntu-14.04-aarch64/Dockerfile
> >
> > However, we didn't introduce this Dockerfile with this PR, so I'm not
> sure
> > if we should "fix" it. Guidance would be much appreciated!
> >
> > Thanks,
> > -Kevin Monroe
> >
> >
> > >
> > >
> > > Olaf
> > >
> > >
> > >
> > >
> > >
> > >
> > > > Am 16.05.2016 um 17:00 schrieb Cory Johns <cory.johns@canonical.com
> >:
> > > >
> > > > Hey, another one of the Juju charm developers here.
> > > >
> > > > We're happy to answer any Juju questions here, as well as on IRC in
> > #juju
> > > > or #bigtop on Freenode, or wherever. :)
> > > >
> > > > We also already realized that we should move the charm code under
> > > > bigtop-deploy and I believe Kevin has already updated the PR on
> GitHub.
> > > > Please let us know if the current structure there matches what you
> were
> > > > thinking. He was also trying to get the JIRA ticket created but I
> > guess
> > > > the system was down when he tried last week, but he'll get that in
> ASAP
> > > if
> > > > he hasn't already (though he's currently travelling).
> > > >
> > > > The license and copyright issues are side effects of these originally
> > > being
> > > > multiple different repositories and they just need to be cleaned
> up. I
> > > > believe the intent is to bring it fully in line with the rest of the
> > > Bigtop
> > > > code base, which I assume means assigning the copyright to the ASF,
> > but I
> > > > have to double-check that to confirm.
> > > >
> > > > The weather report is actually a tool that can be used with the
> charms
> > > and
> > > > bundles to easily run the tests and benchmarks on multiple different
> > > > clouds. I believe the intention was for Canonical to provide the
> cloud
> > > > infrastructure to run it periodically on several different clouds and
> > > > provide the results in a way that is easy to integrate into the
> Bigtop
> > > CI /
> > > > reporting. The idea being that it would enable the community to see
> > if a
> > > > given change or release of Bigtop introduced any issues (or
> > improvements)
> > > > on a particular cloud, architecture, etc, that might not have been
> > > apparent
> > > > from running the CI in a single environment.
> > > >
> > > > As for deploying Bigtop using the charms, I think we'll want to add a
> > > > README under bigtop-deploy/juju and possibly a helper script, but you
> > can
> > > > build and deploy the charms directly from the Bigtop repo. Of
> course,
> > we
> > > > also intend for those built charms to be published to the Juju charm
> > > store
> > > > (they're currently published under the bigdata-dev namespace, e.g.
> > > > https://jujucharms.com/u/bigdata-dev/bigtop-processing-mapreduce; I
> > > think I
> > > > heard someone put forth the idea of an Apache namespace and / or
> > landing
> > > > page at https://jujucharms.com/apache/ but that doesn't exist yet).
> > > > Publishing them to the charm store is to make it easy for Juju users
> to
> > > > discover and deploy them without building them themselves, but we
> still
> > > > intend for the charm code in the Bigtop repository to be the official
> > > home
> > > > of the charms, with the charm store pages linking back to the Bigtop
> > > > repository.
> > > >
> > > > On Sat, May 14, 2016 at 8:08 PM, Roman Shaposhnik <
> > roman@shaposhnik.org>
> > > > wrote:
> > > >
> > > >> Kevin, first of all -- it was great meeting you at ApacheCON last
> > week!
> > > >>
> > > >> To echo Cos -- I too am super excited about Ubuntu charm definitions
> > > >> becoming part of Bigtop. I think that will allow a much tighter
> > > evolution
> > > >> of stacks corresponding to different versions of Bigtop.
> > > >>
> > > >> Btw, does it mean I can now ask n00b Juju questions on dev@bigtop?
> > ;-)
> > > >>
> > > >> On Fri, May 13, 2016 at 2:31 PM, Konstantin Boudnik <cos@apache.org
> >
> > > >> wrote:
> > > >>> Hey Kevin.
> > > >>>
> > > >>> Welcome! And sorry for not answering earlier on the list, although
> we
> > > >> have
> > > >>> chatted off-line during the conference. I like everything I read
> > here,
> > > >> to be
> > > >>> honest. And to recap that conversation:
> > > >>> - perhaps it would make more sense to move the charms under
> > > >> bigtop-deploy?
> > > >>> While charms aren't exactly the means deployment in this case, but
> > > >> they can
> > > >>> be if the logic is written into charms themselves instead of
> getting
> > > >>> externalized to puppet, as in this case
> > > >>
> > > >> +1 on that!
> > > >>
> > > >> Also, I've noticed that the bundle is called hadoop-mapreduce.
> That's
> > > >> pretty
> > > >> 2014 if you know what I mean ;-) Why not call it hadoop-yarn?
> > > >>
> > > >> Another question: why are there so many copies of Apache License
> > > >> throughout the repo? This is certainly not required by ASF, so I
> guess
> > > >> that is a requirement on Canonical's end?
> > > >>
> > > >> And speaking of Canonical:
> > > >> $ grep -ilr "Copyright.*Canonical"
> > > >> bigtop-deploy/juju//hadoop-mapreduce/copyright
> > > >> bigtop-packages/src/charm//hadoop/layer-hadoop-datanode/copyright
> > > >> bigtop-packages/src/charm//hadoop/layer-hadoop-namenode/copyright
> > > >>
> bigtop-packages/src/charm//hadoop/layer-hadoop-nodemanager/copyright
> > > >> bigtop-packages/src/charm//hadoop/layer-hadoop-plugin/copyright
> > > >>
> > >
> bigtop-packages/src/charm//hadoop/layer-hadoop-resourcemanager/copyright
> > > >>
> > > >> We'd have to change that to Copyright XXXX, The Apache Software
> > > Foundation
> > > >> while at the same time adding:
> > > >> Portions Copyright 2015-2016 Canonical Ltd.
> > > >> to the NOTICE
> > > >>
> > > >>> - weather report benchmarking is great. As you said. it would
> > involve a
> > > >> set of
> > > >>> compute resource to be ran. I guess we need to figure out a way
> for
> > > >> adding
> > > >>> those to ci.bigtop.apache.org Jenkins system, right?
> > > >>
> > > >> Wait! I'm confused. Isn't this weather report a service offered by
> > > >> Canonical?
> > > >>
> > > >>> - last but not least: how do you envision people be using charms
> from
> > > the
> > > >>> bigtop's release/git? Will it be possible to point juju to
> something
> > > >> like
> > > >>>
> > > >>
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=tree;f=bigtop-packages/charms
> > > >>> or
> > > >>> https://www.apache.org/dist/bigtop/bigtop-1.1.0/charms
> > > >>> in order to deploy ?
> > > >>
> > > >> That's my biggest question as well! Also, if we plan to "release"
> > > >> those charms to
> > > >> other channels (like the official place where they need to reside)
> we
> > > >> need to figure
> > > >> that one out
> > > >>
> > > >> Thanks a million for your contribution!
> > > >>
> > > >> Roman.
> > > >>
> > >
> > >
> >
>
--
Konstantinos Tsakalozos
Re: Bigtop and Juju
Posted by Alexander Bezzubov <bz...@apache.org>.
Hi,
great to see this initiative going!
I'm sorry if my question is something that is not particularly relevant to
the subject of this discussion, although I assume it might be, so are there
any plans or existing bundles for apache spark, using hadoop-processing
bundle introduced in BIGTOP-2435 [1]?
1. https://issues.apache.org/jira/browse/BIGTOP-2435
--
Kind regards,
Alexander
On Tue, May 24, 2016 at 11:29 AM, Kevin Monroe <ke...@canonical.com>
wrote:
> Hi Olaf,
>
> I've replied to your points below, but I'm also good with moving the
> discussion to the jira (https://issues.apache.org/jira/browse/BIGTOP-2435)
> if that make more sense.
>
> On Mon, May 16, 2016 at 11:04 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
>
> > Hi,
> >
> > nice to have you all on board. Sorry I only do not know enough about
> juju.
> >
> > Regarding the structure:
> >
> > We like to have JIRA's referencing only one git commit object , with the
> > commit message mirroring the JIRA.
> >
> > Since your contribution is quite large I like to have it split in a few
> > independent contributions. Maybe the bigtop-packages/src/charm for
> instance
> > into one JIRA .
> >
>
> We've updated the PR to address concerns after a bit of feedback in
> #bigtop. As of now, the proposal still spans 3 main subdirs (with a touch
> on NOTICE, build.gradle, and pom.xml):
>
> bigtop-deploy/juju
> bigtop-packages/src/charm
> bigtop-tests/cloud-weather-report
>
> While the PR is large, it is fairly compartmentalized into an overall "juju
> enablement" goal, and should have no effect on sibling sources. That said,
> let me know if splitting the patch by subdir with a JIRA for each affected
> component would still be more preferable.
>
> Is there any specific reason why you placed charm under the bigtop-packages
> > directory ? It looks to me more like some orchestration data rather than
> > packaging info.
> >
>
> Charms are the "packages" in Juju parlance. They're comprised of software
> (stuff you would apt install) plus operational code (how that software
> should scale, react to other software, configure itself based on hardware,
> etc). This isn't quite the same as orchestration data because a charm
> (like a deb or rpm) has no idea where it should live nor what its host will
> look like ahead of time -- it only knows how to react once an orchestrator
> puts it somewhere.
>
> We do have a simple way to bundle collections of charms into deployment
> models. We cleverly call these "bundles", an example of which is committed
> to ./bigtop-deploy/juju (https://jujucharms.com/hadoop-processing/). To
> round out the PR, we also chose ./bigtop-tests/cloud-weather-report as the
> location for test plans that would drive a simple red/green UI to check the
> health of bundle deployments across clouds (
> http://status.juju.solutions/recent).
>
> Hopefully that helps explain the reasoning behind these locations, but I'm
> happy to address concerns with any of our choices!
>
>
> > Regarding license:
> >
> > The copyright issues have to be resolved. We can only bring things in,
> > when the RAT Test ("gradle rat") does no complain about licenses any
> more.
> >
>
> The issues that we introduced have been resolved in PR 108 (note the
> careful wording there)... I do see an unapproved license when I run
> gradlew rat:
>
> /home/ubuntu/bigtop/docker/bigtop-slaves/ubuntu-14.04-aarch64/Dockerfile
>
> However, we didn't introduce this Dockerfile with this PR, so I'm not sure
> if we should "fix" it. Guidance would be much appreciated!
>
> Thanks,
> -Kevin Monroe
>
>
> >
> >
> > Olaf
> >
> >
> >
> >
> >
> >
> > > Am 16.05.2016 um 17:00 schrieb Cory Johns <co...@canonical.com>:
> > >
> > > Hey, another one of the Juju charm developers here.
> > >
> > > We're happy to answer any Juju questions here, as well as on IRC in
> #juju
> > > or #bigtop on Freenode, or wherever. :)
> > >
> > > We also already realized that we should move the charm code under
> > > bigtop-deploy and I believe Kevin has already updated the PR on GitHub.
> > > Please let us know if the current structure there matches what you were
> > > thinking. He was also trying to get the JIRA ticket created but I
> guess
> > > the system was down when he tried last week, but he'll get that in ASAP
> > if
> > > he hasn't already (though he's currently travelling).
> > >
> > > The license and copyright issues are side effects of these originally
> > being
> > > multiple different repositories and they just need to be cleaned up. I
> > > believe the intent is to bring it fully in line with the rest of the
> > Bigtop
> > > code base, which I assume means assigning the copyright to the ASF,
> but I
> > > have to double-check that to confirm.
> > >
> > > The weather report is actually a tool that can be used with the charms
> > and
> > > bundles to easily run the tests and benchmarks on multiple different
> > > clouds. I believe the intention was for Canonical to provide the cloud
> > > infrastructure to run it periodically on several different clouds and
> > > provide the results in a way that is easy to integrate into the Bigtop
> > CI /
> > > reporting. The idea being that it would enable the community to see
> if a
> > > given change or release of Bigtop introduced any issues (or
> improvements)
> > > on a particular cloud, architecture, etc, that might not have been
> > apparent
> > > from running the CI in a single environment.
> > >
> > > As for deploying Bigtop using the charms, I think we'll want to add a
> > > README under bigtop-deploy/juju and possibly a helper script, but you
> can
> > > build and deploy the charms directly from the Bigtop repo. Of course,
> we
> > > also intend for those built charms to be published to the Juju charm
> > store
> > > (they're currently published under the bigdata-dev namespace, e.g.
> > > https://jujucharms.com/u/bigdata-dev/bigtop-processing-mapreduce; I
> > think I
> > > heard someone put forth the idea of an Apache namespace and / or
> landing
> > > page at https://jujucharms.com/apache/ but that doesn't exist yet).
> > > Publishing them to the charm store is to make it easy for Juju users to
> > > discover and deploy them without building them themselves, but we still
> > > intend for the charm code in the Bigtop repository to be the official
> > home
> > > of the charms, with the charm store pages linking back to the Bigtop
> > > repository.
> > >
> > > On Sat, May 14, 2016 at 8:08 PM, Roman Shaposhnik <
> roman@shaposhnik.org>
> > > wrote:
> > >
> > >> Kevin, first of all -- it was great meeting you at ApacheCON last
> week!
> > >>
> > >> To echo Cos -- I too am super excited about Ubuntu charm definitions
> > >> becoming part of Bigtop. I think that will allow a much tighter
> > evolution
> > >> of stacks corresponding to different versions of Bigtop.
> > >>
> > >> Btw, does it mean I can now ask n00b Juju questions on dev@bigtop?
> ;-)
> > >>
> > >> On Fri, May 13, 2016 at 2:31 PM, Konstantin Boudnik <co...@apache.org>
> > >> wrote:
> > >>> Hey Kevin.
> > >>>
> > >>> Welcome! And sorry for not answering earlier on the list, although we
> > >> have
> > >>> chatted off-line during the conference. I like everything I read
> here,
> > >> to be
> > >>> honest. And to recap that conversation:
> > >>> - perhaps it would make more sense to move the charms under
> > >> bigtop-deploy?
> > >>> While charms aren't exactly the means deployment in this case, but
> > >> they can
> > >>> be if the logic is written into charms themselves instead of getting
> > >>> externalized to puppet, as in this case
> > >>
> > >> +1 on that!
> > >>
> > >> Also, I've noticed that the bundle is called hadoop-mapreduce. That's
> > >> pretty
> > >> 2014 if you know what I mean ;-) Why not call it hadoop-yarn?
> > >>
> > >> Another question: why are there so many copies of Apache License
> > >> throughout the repo? This is certainly not required by ASF, so I guess
> > >> that is a requirement on Canonical's end?
> > >>
> > >> And speaking of Canonical:
> > >> $ grep -ilr "Copyright.*Canonical"
> > >> bigtop-deploy/juju//hadoop-mapreduce/copyright
> > >> bigtop-packages/src/charm//hadoop/layer-hadoop-datanode/copyright
> > >> bigtop-packages/src/charm//hadoop/layer-hadoop-namenode/copyright
> > >> bigtop-packages/src/charm//hadoop/layer-hadoop-nodemanager/copyright
> > >> bigtop-packages/src/charm//hadoop/layer-hadoop-plugin/copyright
> > >>
> > bigtop-packages/src/charm//hadoop/layer-hadoop-resourcemanager/copyright
> > >>
> > >> We'd have to change that to Copyright XXXX, The Apache Software
> > Foundation
> > >> while at the same time adding:
> > >> Portions Copyright 2015-2016 Canonical Ltd.
> > >> to the NOTICE
> > >>
> > >>> - weather report benchmarking is great. As you said. it would
> involve a
> > >> set of
> > >>> compute resource to be ran. I guess we need to figure out a way for
> > >> adding
> > >>> those to ci.bigtop.apache.org Jenkins system, right?
> > >>
> > >> Wait! I'm confused. Isn't this weather report a service offered by
> > >> Canonical?
> > >>
> > >>> - last but not least: how do you envision people be using charms from
> > the
> > >>> bigtop's release/git? Will it be possible to point juju to something
> > >> like
> > >>>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=tree;f=bigtop-packages/charms
> > >>> or
> > >>> https://www.apache.org/dist/bigtop/bigtop-1.1.0/charms
> > >>> in order to deploy ?
> > >>
> > >> That's my biggest question as well! Also, if we plan to "release"
> > >> those charms to
> > >> other channels (like the official place where they need to reside) we
> > >> need to figure
> > >> that one out
> > >>
> > >> Thanks a million for your contribution!
> > >>
> > >> Roman.
> > >>
> >
> >
>
Re: Bigtop and Juju
Posted by Kevin Monroe <ke...@canonical.com>.
Hi Olaf,
I've replied to your points below, but I'm also good with moving the
discussion to the jira (https://issues.apache.org/jira/browse/BIGTOP-2435)
if that make more sense.
On Mon, May 16, 2016 at 11:04 AM, Olaf Flebbe <of...@oflebbe.de> wrote:
> Hi,
>
> nice to have you all on board. Sorry I only do not know enough about juju.
>
> Regarding the structure:
>
> We like to have JIRA's referencing only one git commit object , with the
> commit message mirroring the JIRA.
>
> Since your contribution is quite large I like to have it split in a few
> independent contributions. Maybe the bigtop-packages/src/charm for instance
> into one JIRA .
>
We've updated the PR to address concerns after a bit of feedback in
#bigtop. As of now, the proposal still spans 3 main subdirs (with a touch
on NOTICE, build.gradle, and pom.xml):
bigtop-deploy/juju
bigtop-packages/src/charm
bigtop-tests/cloud-weather-report
While the PR is large, it is fairly compartmentalized into an overall "juju
enablement" goal, and should have no effect on sibling sources. That said,
let me know if splitting the patch by subdir with a JIRA for each affected
component would still be more preferable.
Is there any specific reason why you placed charm under the bigtop-packages
> directory ? It looks to me more like some orchestration data rather than
> packaging info.
>
Charms are the "packages" in Juju parlance. They're comprised of software
(stuff you would apt install) plus operational code (how that software
should scale, react to other software, configure itself based on hardware,
etc). This isn't quite the same as orchestration data because a charm
(like a deb or rpm) has no idea where it should live nor what its host will
look like ahead of time -- it only knows how to react once an orchestrator
puts it somewhere.
We do have a simple way to bundle collections of charms into deployment
models. We cleverly call these "bundles", an example of which is committed
to ./bigtop-deploy/juju (https://jujucharms.com/hadoop-processing/). To
round out the PR, we also chose ./bigtop-tests/cloud-weather-report as the
location for test plans that would drive a simple red/green UI to check the
health of bundle deployments across clouds (
http://status.juju.solutions/recent).
Hopefully that helps explain the reasoning behind these locations, but I'm
happy to address concerns with any of our choices!
> Regarding license:
>
> The copyright issues have to be resolved. We can only bring things in,
> when the RAT Test ("gradle rat") does no complain about licenses any more.
>
The issues that we introduced have been resolved in PR 108 (note the
careful wording there)... I do see an unapproved license when I run
gradlew rat:
/home/ubuntu/bigtop/docker/bigtop-slaves/ubuntu-14.04-aarch64/Dockerfile
However, we didn't introduce this Dockerfile with this PR, so I'm not sure
if we should "fix" it. Guidance would be much appreciated!
Thanks,
-Kevin Monroe
>
>
> Olaf
>
>
>
>
>
>
> > Am 16.05.2016 um 17:00 schrieb Cory Johns <co...@canonical.com>:
> >
> > Hey, another one of the Juju charm developers here.
> >
> > We're happy to answer any Juju questions here, as well as on IRC in #juju
> > or #bigtop on Freenode, or wherever. :)
> >
> > We also already realized that we should move the charm code under
> > bigtop-deploy and I believe Kevin has already updated the PR on GitHub.
> > Please let us know if the current structure there matches what you were
> > thinking. He was also trying to get the JIRA ticket created but I guess
> > the system was down when he tried last week, but he'll get that in ASAP
> if
> > he hasn't already (though he's currently travelling).
> >
> > The license and copyright issues are side effects of these originally
> being
> > multiple different repositories and they just need to be cleaned up. I
> > believe the intent is to bring it fully in line with the rest of the
> Bigtop
> > code base, which I assume means assigning the copyright to the ASF, but I
> > have to double-check that to confirm.
> >
> > The weather report is actually a tool that can be used with the charms
> and
> > bundles to easily run the tests and benchmarks on multiple different
> > clouds. I believe the intention was for Canonical to provide the cloud
> > infrastructure to run it periodically on several different clouds and
> > provide the results in a way that is easy to integrate into the Bigtop
> CI /
> > reporting. The idea being that it would enable the community to see if a
> > given change or release of Bigtop introduced any issues (or improvements)
> > on a particular cloud, architecture, etc, that might not have been
> apparent
> > from running the CI in a single environment.
> >
> > As for deploying Bigtop using the charms, I think we'll want to add a
> > README under bigtop-deploy/juju and possibly a helper script, but you can
> > build and deploy the charms directly from the Bigtop repo. Of course, we
> > also intend for those built charms to be published to the Juju charm
> store
> > (they're currently published under the bigdata-dev namespace, e.g.
> > https://jujucharms.com/u/bigdata-dev/bigtop-processing-mapreduce; I
> think I
> > heard someone put forth the idea of an Apache namespace and / or landing
> > page at https://jujucharms.com/apache/ but that doesn't exist yet).
> > Publishing them to the charm store is to make it easy for Juju users to
> > discover and deploy them without building them themselves, but we still
> > intend for the charm code in the Bigtop repository to be the official
> home
> > of the charms, with the charm store pages linking back to the Bigtop
> > repository.
> >
> > On Sat, May 14, 2016 at 8:08 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> > wrote:
> >
> >> Kevin, first of all -- it was great meeting you at ApacheCON last week!
> >>
> >> To echo Cos -- I too am super excited about Ubuntu charm definitions
> >> becoming part of Bigtop. I think that will allow a much tighter
> evolution
> >> of stacks corresponding to different versions of Bigtop.
> >>
> >> Btw, does it mean I can now ask n00b Juju questions on dev@bigtop? ;-)
> >>
> >> On Fri, May 13, 2016 at 2:31 PM, Konstantin Boudnik <co...@apache.org>
> >> wrote:
> >>> Hey Kevin.
> >>>
> >>> Welcome! And sorry for not answering earlier on the list, although we
> >> have
> >>> chatted off-line during the conference. I like everything I read here,
> >> to be
> >>> honest. And to recap that conversation:
> >>> - perhaps it would make more sense to move the charms under
> >> bigtop-deploy?
> >>> While charms aren't exactly the means deployment in this case, but
> >> they can
> >>> be if the logic is written into charms themselves instead of getting
> >>> externalized to puppet, as in this case
> >>
> >> +1 on that!
> >>
> >> Also, I've noticed that the bundle is called hadoop-mapreduce. That's
> >> pretty
> >> 2014 if you know what I mean ;-) Why not call it hadoop-yarn?
> >>
> >> Another question: why are there so many copies of Apache License
> >> throughout the repo? This is certainly not required by ASF, so I guess
> >> that is a requirement on Canonical's end?
> >>
> >> And speaking of Canonical:
> >> $ grep -ilr "Copyright.*Canonical"
> >> bigtop-deploy/juju//hadoop-mapreduce/copyright
> >> bigtop-packages/src/charm//hadoop/layer-hadoop-datanode/copyright
> >> bigtop-packages/src/charm//hadoop/layer-hadoop-namenode/copyright
> >> bigtop-packages/src/charm//hadoop/layer-hadoop-nodemanager/copyright
> >> bigtop-packages/src/charm//hadoop/layer-hadoop-plugin/copyright
> >>
> bigtop-packages/src/charm//hadoop/layer-hadoop-resourcemanager/copyright
> >>
> >> We'd have to change that to Copyright XXXX, The Apache Software
> Foundation
> >> while at the same time adding:
> >> Portions Copyright 2015-2016 Canonical Ltd.
> >> to the NOTICE
> >>
> >>> - weather report benchmarking is great. As you said. it would involve a
> >> set of
> >>> compute resource to be ran. I guess we need to figure out a way for
> >> adding
> >>> those to ci.bigtop.apache.org Jenkins system, right?
> >>
> >> Wait! I'm confused. Isn't this weather report a service offered by
> >> Canonical?
> >>
> >>> - last but not least: how do you envision people be using charms from
> the
> >>> bigtop's release/git? Will it be possible to point juju to something
> >> like
> >>>
> >>
> https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=tree;f=bigtop-packages/charms
> >>> or
> >>> https://www.apache.org/dist/bigtop/bigtop-1.1.0/charms
> >>> in order to deploy ?
> >>
> >> That's my biggest question as well! Also, if we plan to "release"
> >> those charms to
> >> other channels (like the official place where they need to reside) we
> >> need to figure
> >> that one out
> >>
> >> Thanks a million for your contribution!
> >>
> >> Roman.
> >>
>
>
Re: Bigtop and Juju
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi,
nice to have you all on board. Sorry I only do not know enough about juju.
Regarding the structure:
We like to have JIRA's referencing only one git commit object , with the commit message mirroring the JIRA.
Since your contribution is quite large I like to have it split in a few independent contributions. Maybe the bigtop-packages/src/charm for instance into one JIRA .
Is there any specific reason why you placed charm under the bigtop-packages directory ? It looks to me more like some orchestration data rather than packaging info.
Regarding license:
The copyright issues have to be resolved. We can only bring things in, when the RAT Test ("gradle rat") does no complain about licenses any more.
Olaf
> Am 16.05.2016 um 17:00 schrieb Cory Johns <co...@canonical.com>:
>
> Hey, another one of the Juju charm developers here.
>
> We're happy to answer any Juju questions here, as well as on IRC in #juju
> or #bigtop on Freenode, or wherever. :)
>
> We also already realized that we should move the charm code under
> bigtop-deploy and I believe Kevin has already updated the PR on GitHub.
> Please let us know if the current structure there matches what you were
> thinking. He was also trying to get the JIRA ticket created but I guess
> the system was down when he tried last week, but he'll get that in ASAP if
> he hasn't already (though he's currently travelling).
>
> The license and copyright issues are side effects of these originally being
> multiple different repositories and they just need to be cleaned up. I
> believe the intent is to bring it fully in line with the rest of the Bigtop
> code base, which I assume means assigning the copyright to the ASF, but I
> have to double-check that to confirm.
>
> The weather report is actually a tool that can be used with the charms and
> bundles to easily run the tests and benchmarks on multiple different
> clouds. I believe the intention was for Canonical to provide the cloud
> infrastructure to run it periodically on several different clouds and
> provide the results in a way that is easy to integrate into the Bigtop CI /
> reporting. The idea being that it would enable the community to see if a
> given change or release of Bigtop introduced any issues (or improvements)
> on a particular cloud, architecture, etc, that might not have been apparent
> from running the CI in a single environment.
>
> As for deploying Bigtop using the charms, I think we'll want to add a
> README under bigtop-deploy/juju and possibly a helper script, but you can
> build and deploy the charms directly from the Bigtop repo. Of course, we
> also intend for those built charms to be published to the Juju charm store
> (they're currently published under the bigdata-dev namespace, e.g.
> https://jujucharms.com/u/bigdata-dev/bigtop-processing-mapreduce; I think I
> heard someone put forth the idea of an Apache namespace and / or landing
> page at https://jujucharms.com/apache/ but that doesn't exist yet).
> Publishing them to the charm store is to make it easy for Juju users to
> discover and deploy them without building them themselves, but we still
> intend for the charm code in the Bigtop repository to be the official home
> of the charms, with the charm store pages linking back to the Bigtop
> repository.
>
> On Sat, May 14, 2016 at 8:08 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> wrote:
>
>> Kevin, first of all -- it was great meeting you at ApacheCON last week!
>>
>> To echo Cos -- I too am super excited about Ubuntu charm definitions
>> becoming part of Bigtop. I think that will allow a much tighter evolution
>> of stacks corresponding to different versions of Bigtop.
>>
>> Btw, does it mean I can now ask n00b Juju questions on dev@bigtop? ;-)
>>
>> On Fri, May 13, 2016 at 2:31 PM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>>> Hey Kevin.
>>>
>>> Welcome! And sorry for not answering earlier on the list, although we
>> have
>>> chatted off-line during the conference. I like everything I read here,
>> to be
>>> honest. And to recap that conversation:
>>> - perhaps it would make more sense to move the charms under
>> bigtop-deploy?
>>> While charms aren't exactly the means deployment in this case, but
>> they can
>>> be if the logic is written into charms themselves instead of getting
>>> externalized to puppet, as in this case
>>
>> +1 on that!
>>
>> Also, I've noticed that the bundle is called hadoop-mapreduce. That's
>> pretty
>> 2014 if you know what I mean ;-) Why not call it hadoop-yarn?
>>
>> Another question: why are there so many copies of Apache License
>> throughout the repo? This is certainly not required by ASF, so I guess
>> that is a requirement on Canonical's end?
>>
>> And speaking of Canonical:
>> $ grep -ilr "Copyright.*Canonical"
>> bigtop-deploy/juju//hadoop-mapreduce/copyright
>> bigtop-packages/src/charm//hadoop/layer-hadoop-datanode/copyright
>> bigtop-packages/src/charm//hadoop/layer-hadoop-namenode/copyright
>> bigtop-packages/src/charm//hadoop/layer-hadoop-nodemanager/copyright
>> bigtop-packages/src/charm//hadoop/layer-hadoop-plugin/copyright
>> bigtop-packages/src/charm//hadoop/layer-hadoop-resourcemanager/copyright
>>
>> We'd have to change that to Copyright XXXX, The Apache Software Foundation
>> while at the same time adding:
>> Portions Copyright 2015-2016 Canonical Ltd.
>> to the NOTICE
>>
>>> - weather report benchmarking is great. As you said. it would involve a
>> set of
>>> compute resource to be ran. I guess we need to figure out a way for
>> adding
>>> those to ci.bigtop.apache.org Jenkins system, right?
>>
>> Wait! I'm confused. Isn't this weather report a service offered by
>> Canonical?
>>
>>> - last but not least: how do you envision people be using charms from the
>>> bigtop's release/git? Will it be possible to point juju to something
>> like
>>>
>> https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=tree;f=bigtop-packages/charms
>>> or
>>> https://www.apache.org/dist/bigtop/bigtop-1.1.0/charms
>>> in order to deploy ?
>>
>> That's my biggest question as well! Also, if we plan to "release"
>> those charms to
>> other channels (like the official place where they need to reside) we
>> need to figure
>> that one out
>>
>> Thanks a million for your contribution!
>>
>> Roman.
>>
Re: Bigtop and Juju
Posted by Cory Johns <co...@canonical.com>.
Hey, another one of the Juju charm developers here.
We're happy to answer any Juju questions here, as well as on IRC in #juju
or #bigtop on Freenode, or wherever. :)
We also already realized that we should move the charm code under
bigtop-deploy and I believe Kevin has already updated the PR on GitHub.
Please let us know if the current structure there matches what you were
thinking. He was also trying to get the JIRA ticket created but I guess
the system was down when he tried last week, but he'll get that in ASAP if
he hasn't already (though he's currently travelling).
The license and copyright issues are side effects of these originally being
multiple different repositories and they just need to be cleaned up. I
believe the intent is to bring it fully in line with the rest of the Bigtop
code base, which I assume means assigning the copyright to the ASF, but I
have to double-check that to confirm.
The weather report is actually a tool that can be used with the charms and
bundles to easily run the tests and benchmarks on multiple different
clouds. I believe the intention was for Canonical to provide the cloud
infrastructure to run it periodically on several different clouds and
provide the results in a way that is easy to integrate into the Bigtop CI /
reporting. The idea being that it would enable the community to see if a
given change or release of Bigtop introduced any issues (or improvements)
on a particular cloud, architecture, etc, that might not have been apparent
from running the CI in a single environment.
As for deploying Bigtop using the charms, I think we'll want to add a
README under bigtop-deploy/juju and possibly a helper script, but you can
build and deploy the charms directly from the Bigtop repo. Of course, we
also intend for those built charms to be published to the Juju charm store
(they're currently published under the bigdata-dev namespace, e.g.
https://jujucharms.com/u/bigdata-dev/bigtop-processing-mapreduce; I think I
heard someone put forth the idea of an Apache namespace and / or landing
page at https://jujucharms.com/apache/ but that doesn't exist yet).
Publishing them to the charm store is to make it easy for Juju users to
discover and deploy them without building them themselves, but we still
intend for the charm code in the Bigtop repository to be the official home
of the charms, with the charm store pages linking back to the Bigtop
repository.
On Sat, May 14, 2016 at 8:08 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> Kevin, first of all -- it was great meeting you at ApacheCON last week!
>
> To echo Cos -- I too am super excited about Ubuntu charm definitions
> becoming part of Bigtop. I think that will allow a much tighter evolution
> of stacks corresponding to different versions of Bigtop.
>
> Btw, does it mean I can now ask n00b Juju questions on dev@bigtop? ;-)
>
> On Fri, May 13, 2016 at 2:31 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> > Hey Kevin.
> >
> > Welcome! And sorry for not answering earlier on the list, although we
> have
> > chatted off-line during the conference. I like everything I read here,
> to be
> > honest. And to recap that conversation:
> > - perhaps it would make more sense to move the charms under
> bigtop-deploy?
> > While charms aren't exactly the means deployment in this case, but
> they can
> > be if the logic is written into charms themselves instead of getting
> > externalized to puppet, as in this case
>
> +1 on that!
>
> Also, I've noticed that the bundle is called hadoop-mapreduce. That's
> pretty
> 2014 if you know what I mean ;-) Why not call it hadoop-yarn?
>
> Another question: why are there so many copies of Apache License
> throughout the repo? This is certainly not required by ASF, so I guess
> that is a requirement on Canonical's end?
>
> And speaking of Canonical:
> $ grep -ilr "Copyright.*Canonical"
> bigtop-deploy/juju//hadoop-mapreduce/copyright
> bigtop-packages/src/charm//hadoop/layer-hadoop-datanode/copyright
> bigtop-packages/src/charm//hadoop/layer-hadoop-namenode/copyright
> bigtop-packages/src/charm//hadoop/layer-hadoop-nodemanager/copyright
> bigtop-packages/src/charm//hadoop/layer-hadoop-plugin/copyright
> bigtop-packages/src/charm//hadoop/layer-hadoop-resourcemanager/copyright
>
> We'd have to change that to Copyright XXXX, The Apache Software Foundation
> while at the same time adding:
> Portions Copyright 2015-2016 Canonical Ltd.
> to the NOTICE
>
> > - weather report benchmarking is great. As you said. it would involve a
> set of
> > compute resource to be ran. I guess we need to figure out a way for
> adding
> > those to ci.bigtop.apache.org Jenkins system, right?
>
> Wait! I'm confused. Isn't this weather report a service offered by
> Canonical?
>
> > - last but not least: how do you envision people be using charms from the
> > bigtop's release/git? Will it be possible to point juju to something
> like
> >
> https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=tree;f=bigtop-packages/charms
> > or
> > https://www.apache.org/dist/bigtop/bigtop-1.1.0/charms
> > in order to deploy ?
>
> That's my biggest question as well! Also, if we plan to "release"
> those charms to
> other channels (like the official place where they need to reside) we
> need to figure
> that one out
>
> Thanks a million for your contribution!
>
> Roman.
>
Re: Bigtop and Juju
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
Kevin, first of all -- it was great meeting you at ApacheCON last week!
To echo Cos -- I too am super excited about Ubuntu charm definitions
becoming part of Bigtop. I think that will allow a much tighter evolution
of stacks corresponding to different versions of Bigtop.
Btw, does it mean I can now ask n00b Juju questions on dev@bigtop? ;-)
On Fri, May 13, 2016 at 2:31 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Hey Kevin.
>
> Welcome! And sorry for not answering earlier on the list, although we have
> chatted off-line during the conference. I like everything I read here, to be
> honest. And to recap that conversation:
> - perhaps it would make more sense to move the charms under bigtop-deploy?
> While charms aren't exactly the means deployment in this case, but they can
> be if the logic is written into charms themselves instead of getting
> externalized to puppet, as in this case
+1 on that!
Also, I've noticed that the bundle is called hadoop-mapreduce. That's pretty
2014 if you know what I mean ;-) Why not call it hadoop-yarn?
Another question: why are there so many copies of Apache License
throughout the repo? This is certainly not required by ASF, so I guess
that is a requirement on Canonical's end?
And speaking of Canonical:
$ grep -ilr "Copyright.*Canonical"
bigtop-deploy/juju//hadoop-mapreduce/copyright
bigtop-packages/src/charm//hadoop/layer-hadoop-datanode/copyright
bigtop-packages/src/charm//hadoop/layer-hadoop-namenode/copyright
bigtop-packages/src/charm//hadoop/layer-hadoop-nodemanager/copyright
bigtop-packages/src/charm//hadoop/layer-hadoop-plugin/copyright
bigtop-packages/src/charm//hadoop/layer-hadoop-resourcemanager/copyright
We'd have to change that to Copyright XXXX, The Apache Software Foundation
while at the same time adding:
Portions Copyright 2015-2016 Canonical Ltd.
to the NOTICE
> - weather report benchmarking is great. As you said. it would involve a set of
> compute resource to be ran. I guess we need to figure out a way for adding
> those to ci.bigtop.apache.org Jenkins system, right?
Wait! I'm confused. Isn't this weather report a service offered by Canonical?
> - last but not least: how do you envision people be using charms from the
> bigtop's release/git? Will it be possible to point juju to something like
> https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=tree;f=bigtop-packages/charms
> or
> https://www.apache.org/dist/bigtop/bigtop-1.1.0/charms
> in order to deploy ?
That's my biggest question as well! Also, if we plan to "release"
those charms to
other channels (like the official place where they need to reside) we
need to figure
that one out
Thanks a million for your contribution!
Roman.
Re: Bigtop and Juju
Posted by Konstantin Boudnik <co...@apache.org>.
Hey Kevin.
Welcome! And sorry for not answering earlier on the list, although we have
chatted off-line during the conference. I like everything I read here, to be
honest. And to recap that conversation:
- perhaps it would make more sense to move the charms under bigtop-deploy?
While charms aren't exactly the means deployment in this case, but they can
be if the logic is written into charms themselves instead of getting
externalized to puppet, as in this case
- weather report benchmarking is great. As you said. it would involve a set of
compute resource to be ran. I guess we need to figure out a way for adding
those to ci.bigtop.apache.org Jenkins system, right?
- last but not least: how do you envision people be using charms from the
bigtop's release/git? Will it be possible to point juju to something like
https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=tree;f=bigtop-packages/charms
or
https://www.apache.org/dist/bigtop/bigtop-1.1.0/charms
in order to deploy ?
Thanks for your contributions!
Cos
On Mon, May 09, 2016 at 06:35PM, Kevin Monroe wrote:
> Hi Bigtop Devs!
>
> We’ve been working on an Open Source way to model distributed systems like
> Big Data. We initially built Apache packages (like Hadoop and Spark) from
> scratch, but quickly realized that leveraging Bigtop’s experience and
> packaging was a smarter approach. Using Bigtop has drastically improved
> the development, maintainability, and feature set of our project, so thanks
> for the solid work done here in the community :-)
>
> The project we’re working on is Juju[0], and more specifically, Juju
> charms[1] for Bigtop components. Charms are essentially operational code
> that wrap big software like Hadoop. We are excited to share our recent work
> where we focus on using Bigtop as the foundation of the Juju big data
> charms. We’ve bundled a mapreduce solution (along with ganglia monitoring)
> here:
>
> https://jujucharms.com/u/bigdata-dev/bigtop-processing-mapreduce
>
> With charms being service orientated and repeatable, we have also developed
> an interesting way to get insights on how Bigtop-based solutions are
> performing across a given set of clouds or local containers. Check out
> example runs of a “cloud weather report” for the Bigtop bundle on GCE and
> AWS: https://goo.gl/TPbXaj
>
> These pages are static HTML generated by:
>
> https://github.com/juju-solutions/cloud-weather-report/
>
> We’d like to propose making these reports available alongside the Bigtop CI
> (with Canonical providing the cloud resources), as in:
>
> https://ci.bigtop.apache.org/<solution-name>
>
> In addition, we would like contribute the charms and bundles per the
> following PR:
>
> https://github.com/apache/bigtop/pull/108
>
> We are excited to be part of the Bigtop community and appreciate the
> consistency displayed and effort involved in packaging something as complex
> as the big data ecosystem. If you would like to discuss this more
> in-person and you are attending Apache Big Data in Vancouver, we will be
> meeting at 5pm Tuesday (May 10) in Prince of Wales room, 3rd floor at the
> Hyatt past the bathrooms.
>
>
> We look forward to discussing this further, either on-list or in person
> this week. Thanks!
> -Kevin Monroe
>
>
> [0] - Juju is an open-source modeling language that captures operational
> intelligence for (big) software like Big Data -- things like life cycle,
> configuration, scaling, and intra-service relationships for a given piece
> of software.
>
> [1] - Charms are at the heart of Juju modeling. They encapsulate
> software’s operational code in repeatably deployable packages.