Proposal: shared Mesos framework hosting and registry

[Dave Lester <da...@gmail.com>]

As the number of Mesos frameworks grows (and now, a module system), I think
it's time to create a community-maintained registry with the goal of making
frameworks and modules easier to discover, contribute to, and install.

There's already a JIRA ticket tracking this (MESOS-1759) and I've chatted
with several folks (thanks in particular Victor Vieux, Tom Arnfeld, Vinod
Kone, Timothy St Clair, and Joe Stein). I'd like to advance the
conversation by offering a proposal on the public mailing list.

I imagine two initiatives to achieve this:

1) Shared hosting via a GitHub org. I'm not sure if you're familiar with
how Jenkins maintains their plugins on GitHub [1], but they allow
individual plugins to have their own repo within their GH org. Plugins are
repos with a specific naming convention (in their case, they've appended
"-plugin" to the repo name but this isn't the same approach we'd need to
take). Having a shared GH org creates greater visibility to what people are
doing, and encourages community participation and ownership.

In the case of Mesos, not all frameworks will jump at the chance to have
community hosting which is fine. But particularly for smaller frameworks, I
think this is a good option given the success Jenkins has had. We could
potentially use the existing /mesos github org, or create a new org
/mesos-extensions or something of the like. It seems like the only
potential snag here is that we'll want to be explicit that these aren't
Apache-blessed plugins and instead maintained by the larger ecosystem, but
I believe we can achieve that by offering a notice in the GH org
description.

2) A registry that allows framework writers to publish new versions of
their frameworks to a central repository that could be programmatically
accessed and rendered online. The v1 could be incredibly simple, but I
think this is a foundational piece that can grow with the project in the
future. Since this is a little bit more-involved, I've created a separate
Google Doc [2] to begin drafting the scope for this:
https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing,
and have intentionally punted on some of the implementation details until
the scope is finalized and I gauge interest from users and potential
collaborators.

What do you think? If folks are on board I will assign the JIRA issue to
myself and get to work; I'd also welcome collaborators to help get this off
the ground since I think it will be a boost for the community. Feedback is
welcome!

Thanks,
Dave

[1] https://github.com/jenkinsci/
[2]
https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing

Re: Proposal: shared Mesos framework hosting and registry

[Billy Bones <ga...@gmail.com>]

Hi Dave, As I understand your proposal, you want to create a nexus like
repository isn't it?

If so, your proposal is really interesting as I'm working hard on Mesos's
advocate and the most recurrent demand is about a feature of that kind when
I'm facing Enterprise customers.

I think that using GitHub as an always on repository for the frameworks
registry would be a really good idea as it allow everyone to edit it as
soon as one of the listed framework disappear/evolve or URLs changes.

I deeply think that Mesos need something like that to gain much more
traction on Old style Enterprises.

I've updated your docs with two or three ideas mainly focused on WebUI and
Mesos integration.

Let us informed of any update regarding this topic!

2014-12-01 2:32 GMT+01:00 Dave Lester <da...@gmail.com>:

> As the number of Mesos frameworks grows (and now, a module system), I
> think it's time to create a community-maintained registry with the goal of
> making frameworks and modules easier to discover, contribute to, and
> install.
>
> There's already a JIRA ticket tracking this (MESOS-1759) and I've chatted
> with several folks (thanks in particular Victor Vieux, Tom Arnfeld, Vinod
> Kone, Timothy St Clair, and Joe Stein). I'd like to advance the
> conversation by offering a proposal on the public mailing list.
>
> I imagine two initiatives to achieve this:
>
> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with
> how Jenkins maintains their plugins on GitHub [1], but they allow
> individual plugins to have their own repo within their GH org. Plugins are
> repos with a specific naming convention (in their case, they've appended
> "-plugin" to the repo name but this isn't the same approach we'd need to
> take). Having a shared GH org creates greater visibility to what people are
> doing, and encourages community participation and ownership.
>
> In the case of Mesos, not all frameworks will jump at the chance to have
> community hosting which is fine. But particularly for smaller frameworks, I
> think this is a good option given the success Jenkins has had. We could
> potentially use the existing /mesos github org, or create a new org
> /mesos-extensions or something of the like. It seems like the only
> potential snag here is that we'll want to be explicit that these aren't
> Apache-blessed plugins and instead maintained by the larger ecosystem, but
> I believe we can achieve that by offering a notice in the GH org
> description.
>
> 2) A registry that allows framework writers to publish new versions of
> their frameworks to a central repository that could be programmatically
> accessed and rendered online. The v1 could be incredibly simple, but I
> think this is a foundational piece that can grow with the project in the
> future. Since this is a little bit more-involved, I've created a separate
> Google Doc [2] to begin drafting the scope for this:
> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing,
> and have intentionally punted on some of the implementation details until
> the scope is finalized and I gauge interest from users and potential
> collaborators.
>
> What do you think? If folks are on board I will assign the JIRA issue to
> myself and get to work; I'd also welcome collaborators to help get this off
> the ground since I think it will be a boost for the community. Feedback is
> welcome!
>
> Thanks,
> Dave
>
> [1] https://github.com/jenkinsci/
> [2]
> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing
>

Re: Proposal: shared Mesos framework hosting and registry

[Connor Doyle <co...@gmail.com>]

Hi Dave,

Your rephrasing is accurate, and you do bring up some interesting
points around management.
Sorry for the response delay; I and the rest of the Mesosphere folks
will be back in SF following our Q4 offsite by mid-week.

--
Connor

On Fri, Dec 12, 2014 at 12:39 PM, Dave Lester <da...@gmail.com> wrote:
> Hey Connor,
>
> It would be great to hear back about this and get the ball rolling on a
> shared registry. Let me know what you think!
>
> Thanks,
> Dave
>
> On Wed, Dec 3, 2014 at 1:11 PM, Dave Lester <da...@gmail.com> wrote:
>>
>> Hi Connor,
>>
>> Thanks for sharing the work you've been doing with Universe. Sounds
>> complimentary and pretty exciting! I thought I'd take a stab at making some
>> clarifications.
>>
>> To rephrase your proposal by using a comparison, you're proposing the
>> "homebrew model" of package management here in terms of how the registry is
>> maintained, correct? A registry of metadata that's hosted via a single
>> GitHub repository, that community members submit pull requests to when they
>> want to have a release included in the registry.
>>
>> In this case of Homebrew, they manage "formulas"; in our case it would be
>> package.json files (which is how NodeJS handles package info).
>>
>> Architecture
>> The only way Universe diverges from my original thinking is that you're
>> advocating decentralization of package metadata, whereas my original thought
>> was to have it much more centralized. There's no reason it couldn't be
>> brought together through a single interface (command-line, or searchable web
>> interface) so I don't think it's incompatible technically, although I do
>> think it creates some potential social/governance overhead (more below).
>>
>> Metadata
>> The package.json file is minimal and similar to what you see in most
>> package management systems. Where you're taking it a step further is a JSON
>> manifest for how to actually run the program, which is outside of the scope
>> of what I was hoping to accomplish but it looks awesome.
>>
>> Managing The Registry
>> One part of your proposal that's unclear to me is when information is
>> pushed to the registry repository, and how often (for each release, or just
>> when the package is first added). Additionally, what information is pushed
>> here -- just a pointer to the git repo? It's my current understanding that
>> the package.json and framework.json files are both within the actual
>> package's repo, not the registry. I would love to hear how you're thinking
>> about this.
>>
>> If it's just a pointer to another repo, it would make it possible for any
>> package already vetted to become part of the registry to do their own
>> updates without requiring someone to accept a PR. I think this is ideal. If
>> it's the latter, it brings up some interesting questions related to who
>> would manage this and under what guidelines they would do this vetting; we'd
>> essentially get a PR for every release within the community at that point.
>>
>> The Registry "Home"
>> I feel strongly that this should be a community-driven effort, and I see
>> your proposal along with my own as very compatible. Would love to work
>> together, and if this works as I hope it will be a huge driver of activity
>> in the ecosystem.
>>
>> Dave
>>
>> On Tue, Dec 2, 2014 at 1:43 AM, Jing Dong <ji...@qubitdigital.com> wrote:
>>>
>>> I like the idea of having a single UI for installing frameworks. One Web
>>> UI endpoint for managing setting and view state of multiple installed
>>> frameworks.
>>>
>>>
>>> On 2 Dec 2014, at 09:26, Billy Bones <ga...@gmail.com> wrote:
>>>
>>> Wooo mesosphere is making such a good work pushing forward Mesos
>>> technologies!
>>>
>>> I deeply think that Mesos should implement registry as a core feature.
>>>
>>> Let me explain that, Installing manually frameworks / modules / other
>>> third party adds-on is quite cool when you're beginner and want to learn how
>>> the things works, but once you're in production, you've to be fast and react
>>> quickly.
>>>
>>> Let said that I'm a system engineer that have customers requests like
>>> "Hum... Could you had this " insert Framework/module/other name here" on our
>>> cluster region?", I'll have to add it as soon and quick as possible on the
>>> targeted environment, I'll be OK to do it twice, but I wont make my job a
>>> request nightmare and I'll probably search for a way to provide to my
>>> customers a restrictive access (WebUI) to the cluster manager and allow them
>>> to deploy those new add-on by themself.
>>>
>>> With this study case in mind, I'll be happy to have a registry which
>>> allow a filtering mechanism to hide or show frameworks/modules/etc by status
>>> (Official Apache Content / Public Content / Private Content).
>>>
>>> Here is my thoughts about the registry articulation:
>>>
>>> Mesos Registry is an integrated part of Mesos master services.
>>> Mesos Registry is an endpoint available to WebUI and CLI.
>>> Mesos Registry is nothing but a metadata registry.
>>> Mesos Registry save its configs and metadata on a key/value store
>>> (zookeeper?).
>>> Mesos Registry is empty at the first launch.
>>> Mesos Registry as three views: Official Apache Content | Publicly
>>> Maintained Content | Private Maintained Content
>>> Mesos Registry views content is:
>>>
>>> Official Apache Content == Link to existing Apache add-ons hosted on
>>> Github/Git repository + metadata (Like those proposed by Dave and Connor).
>>> Publicly Maintained Content == Links to existing repositories (Github /
>>> Git / other) + metadata (Like those proposed by Dave and Connor).
>>> Private Content == Links to existing private (GIT/GITHUB/Other
>>> repositories) + metadata (Like those proposed by Dave and Connor), this
>>> repository is kinda special as it is hosted and created by the cluster
>>> operators and could be a mixed content of locally maintained repository (GIT
>>> repos on a HDFS or TAR on HDFS) and public content repository cloned from
>>> the public content URL/Metadata.
>>>
>>> I don't know if I'm really clear, so if I'm not, let me know it, I'll do
>>> some sketches :D
>>>
>>>
>>>
>>> 2014-12-02 1:53 GMT+01:00 Connor Doyle <co...@mesosphere.io>:
>>>>
>>>> Hi Dave,
>>>>
>>>> This is a timely topic, since we have been prototyping and mocking up
>>>> something similar at Mesosphere.  We created a new public GitHub repository
>>>> for it about three weeks ago called "universe"
>>>> (http://github.com/mesosphere/universe).
>>>>
>>>> Although we have added some informal specs, it's very malleable at this
>>>> point.  We're very much interested in making our "universe" compatible with,
>>>> or the same as, the registry you're proposing.  Without delving into
>>>> implementation details, some of the goals we have in mind are outlined
>>>> below.
>>>>
>>>> Data Source:
>>>>
>>>> The package repository should be easily consumable by third-party
>>>> command-line and other programs.  There should be a condensed “index”
>>>> representation of the package repository available.
>>>>
>>>> Packages within the repository should be versioned.
>>>>
>>>> The package repository format itself should be versioned.
>>>>
>>>> Decentralization and Composability:
>>>>
>>>> The package metadata should be hosted in a public place (we like GitHub)
>>>> so that additional packages can be added by the community by simply
>>>> submitting pull requests.  We have added some rudimentary commit hooks and
>>>> automated validation to protect the repo against breaking changes.
>>>>
>>>> It’s important that no single entity “owns the keys” to the universe,
>>>> and that the spec and implementation remain public.  It should be easy and
>>>> free for organizations to maintain a private package repository.
>>>>
>>>> A corollary is that it should be easy for consumers to pull from a
>>>> hierarchy of upstream repositories.  One setup we have in mind is that an
>>>> organization might have staging and production repositories running
>>>> internally.  Packages are pushed to staging where integration testing can
>>>> run before “deployment” to production.  If a package isn’t in the local
>>>> repository it might be looked up and installed from upstream.
>>>>
>>>> <hierarchy.png>
>>>>
>>>>
>>>> Repositories should be able to be proxied and cached in this way.
>>>> Organizations should be able to isolate their datacenter but also
>>>> selectively add external packages for experimentation. The system should be
>>>> sufficiently portable and extensible to accomodate these and similar use
>>>> cases.
>>>>
>>>> Meta-Framework Descriptors:
>>>>
>>>> Our conception of the package repository is a bit more expansive than
>>>> just Mesos frameworks; it includes descriptions of how to install any piece
>>>> of server software on a Mesos cluster.  Frameworks and non-frameworks alike
>>>> may be installed using some other meta-framework that’s responsible for
>>>> starting all other cluster services.  Likely candidates for this role are
>>>> the long-lived frameworks: Aurora, Marathon, Singularity, and eventually
>>>> Kubernetes.  In any case, the repository spec should not be prescriptive
>>>> with respect to this choice.
>>>>
>>>> The package repository metadata should make it easy for Mesos framework
>>>> authors (and authors of non-Mesos-aware programs) to describe how to install
>>>> their software on a Mesos cluster.  To this end, our prototype package spec
>>>> allows for Meta-framework descriptor files for each package in the
>>>> repository.  For example for a given package we might see a `marathon.json`
>>>> file as well as a `my-app.aurora` file.
>>>>
>>>> An obvious concern is how to specify site-specific arguments upon
>>>> installation.  Here packages should describe data that must be marshalled
>>>> from the environment (e.g. by prompting a user) and combined with the raw
>>>> meta-framework descriptor to launch the app.  These configuration parameters
>>>> should be agnostic of the supported meta-frameworks.  More concretely, in
>>>> our prototype we describe configuration data in terms of a JSON-Schema.
>>>>
>>>> CLI Integration:
>>>>
>>>> Part of our proposed package format is an optional descriptor for how to
>>>> fetch and install the command-line tools for interacting with the
>>>> application.  For now, we only have one implementation of this, which is to
>>>> fetch a python egg from PyPI.
>>>>
>>>> Governance:
>>>>
>>>> All in all, we think that making this effort more community driven is a
>>>> healthy way to proceed.  Any input is very welcome.  For example, if others
>>>> think that what we have is a good starting point we could transfer ownership
>>>> of the repository to the mesos organization on GitHub.
>>>>
>>>> Cheers,
>>>> --
>>>> Connor Doyle
>>>> http://mesosphere.com
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Nov 30, 2014, at 17:32, Dave Lester <da...@gmail.com> wrote:
>>>>
>>>> As the number of Mesos frameworks grows (and now, a module system), I
>>>> think it's time to create a community-maintained registry with the goal of
>>>> making frameworks and modules easier to discover, contribute to, and
>>>> install.
>>>>
>>>> There's already a JIRA ticket tracking this (MESOS-1759) and I've
>>>> chatted with several folks (thanks in particular Victor Vieux, Tom Arnfeld,
>>>> Vinod Kone, Timothy St Clair, and Joe Stein). I'd like to advance the
>>>> conversation by offering a proposal on the public mailing list.
>>>>
>>>> I imagine two initiatives to achieve this:
>>>>
>>>> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with
>>>> how Jenkins maintains their plugins on GitHub [1], but they allow individual
>>>> plugins to have their own repo within their GH org. Plugins are repos with a
>>>> specific naming convention (in their case, they've appended "-plugin" to the
>>>> repo name but this isn't the same approach we'd need to take). Having a
>>>> shared GH org creates greater visibility to what people are doing, and
>>>> encourages community participation and ownership.
>>>>
>>>> In the case of Mesos, not all frameworks will jump at the chance to have
>>>> community hosting which is fine. But particularly for smaller frameworks, I
>>>> think this is a good option given the success Jenkins has had. We could
>>>> potentially use the existing /mesos github org, or create a new org
>>>> /mesos-extensions or something of the like. It seems like the only potential
>>>> snag here is that we'll want to be explicit that these aren't Apache-blessed
>>>> plugins and instead maintained by the larger ecosystem, but I believe we can
>>>> achieve that by offering a notice in the GH org description.
>>>>
>>>> 2) A registry that allows framework writers to publish new versions of
>>>> their frameworks to a central repository that could be programmatically
>>>> accessed and rendered online. The v1 could be incredibly simple, but I think
>>>> this is a foundational piece that can grow with the project in the future.
>>>> Since this is a little bit more-involved, I've created a separate Google Doc
>>>> [2] to begin drafting the scope for this:
>>>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing,
>>>> and have intentionally punted on some of the implementation details until
>>>> the scope is finalized and I gauge interest from users and potential
>>>> collaborators.
>>>>
>>>> What do you think? If folks are on board I will assign the JIRA issue to
>>>> myself and get to work; I'd also welcome collaborators to help get this off
>>>> the ground since I think it will be a boost for the community. Feedback is
>>>> welcome!
>>>>
>>>> Thanks,
>>>> Dave
>>>>
>>>> [1] https://github.com/jenkinsci/
>>>> [2]
>>>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing
>>>>
>>>>
>>>
>>
>



-- 
connor

Re: Proposal: shared Mesos framework hosting and registry

[Dave Lester <da...@gmail.com>]

Hey Connor,

It would be great to hear back about this and get the ball rolling on a
shared registry. Let me know what you think!

Thanks,
Dave

On Wed, Dec 3, 2014 at 1:11 PM, Dave Lester <da...@gmail.com> wrote:
>
> Hi Connor,
>
> Thanks for sharing the work you've been doing with Universe. Sounds
> complimentary and pretty exciting! I thought I'd take a stab at making some
> clarifications.
>
> To rephrase your proposal by using a comparison, you're proposing the
> "homebrew model" of package management here in terms of how the registry is
> maintained, correct? A registry of metadata that's hosted via a single
> GitHub repository, that community members submit pull requests to when they
> want to have a release included in the registry.
>
> In this case of Homebrew, they manage "formulas"; in our case it would be
> package.json files (which is how NodeJS handles package info).
>
> *Architecture*
> The only way Universe diverges from my original thinking is that you're
> advocating decentralization of package metadata, whereas my original
> thought was to have it much more centralized. There's no reason it couldn't
> be brought together through a single interface (command-line, or searchable
> web interface) so I don't think it's incompatible technically, although I
> do think it creates some potential social/governance overhead (more below).
>
> *Metadata*
> The package.json file is minimal and similar to what you see in most
> package management systems. Where you're taking it a step further is a JSON
> manifest for how to actually run the program, which is outside of the scope
> of what I was hoping to accomplish but it looks awesome.
>
> *Managing The Registry*
> One part of your proposal that's unclear to me is when information is
> pushed to the registry repository, and how often (for each release, or just
> when the package is first added). Additionally, what information is pushed
> here -- just a pointer to the git repo? It's my current understanding that
> the package.json and framework.json files are both within the actual
> package's repo, not the registry. I would love to hear how you're thinking
> about this.
>
> If it's just a pointer to another repo, it would make it possible for any
> package already vetted to become part of the registry to do their own
> updates without requiring someone to accept a PR. I think this is ideal. If
> it's the latter, it brings up some interesting questions related to who
> would manage this and under what guidelines they would do this vetting;
> we'd essentially get a PR for every release within the community at that
> point.
>
> *The Registry "Home"*
> I feel strongly that this should be a community-driven effort, and I see
> your proposal along with my own as very compatible. Would love to work
> together, and if this works as I hope it will be a huge driver of activity
> in the ecosystem.
>
> Dave
>
> On Tue, Dec 2, 2014 at 1:43 AM, Jing Dong <ji...@qubitdigital.com> wrote:
>
>> I like the idea of having a single UI for installing frameworks. One Web
>> UI endpoint for managing setting and view state of multiple installed
>> frameworks.
>>
>>
>> On 2 Dec 2014, at 09:26, Billy Bones <ga...@gmail.com> wrote:
>>
>> Wooo mesosphere is making such a good work pushing forward Mesos
>> technologies!
>>
>> I deeply think that Mesos should implement registry as a core feature.
>>
>> Let me explain that, Installing manually frameworks / modules / other
>> third party adds-on is quite cool when you're beginner and want to learn
>> how the things works, but once you're in production, you've to be fast and
>> react quickly.
>>
>> Let said that I'm a system engineer that have customers requests like
>> "Hum... Could you had this " insert Framework/module/other name here" on
>> our cluster region?", I'll have to add it as soon and quick as possible on
>> the targeted environment, I'll be OK to do it twice, but I wont make my job
>> a request nightmare and I'll probably search for a way to provide to my
>> customers a restrictive access (WebUI) to the cluster manager and allow
>> them to deploy those new add-on by themself.
>>
>> With this study case in mind, I'll be happy to have a registry which
>> allow a filtering mechanism to hide or show frameworks/modules/etc by
>> status (Official Apache Content / Public Content / Private Content).
>>
>> Here is my thoughts about the registry articulation:
>>
>>    - Mesos Registry is an integrated part of Mesos master services.
>>    - Mesos Registry is an endpoint available to WebUI and CLI.
>>    - Mesos Registry is nothing but a metadata registry.
>>    - Mesos Registry save its configs and metadata on a key/value store
>>    (zookeeper?).
>>    - Mesos Registry is empty at the first launch.
>>    - Mesos Registry as three views: Official Apache Content | Publicly
>>    Maintained Content | Private Maintained Content
>>    - Mesos Registry views content is:
>>    - Official Apache Content == Link to existing Apache add-ons hosted
>>       on Github/Git repository + metadata (Like those proposed by Dave and
>>       Connor).
>>       - Publicly Maintained Content == Links to existing repositories
>>       (Github / Git / other) + metadata (Like those proposed by Dave and Connor).
>>       - Private Content == Links to existing private (GIT/GITHUB/Other
>>       repositories) + metadata (Like those proposed by Dave and Connor), this
>>       repository is kinda special as it is hosted and created by the cluster
>>       operators and could be a mixed content of locally maintained repository
>>       (GIT repos on a HDFS or TAR on HDFS) and public content repository cloned
>>       from the public content URL/Metadata.
>>
>> I don't know if I'm really clear, so if I'm not, let me know it, I'll do
>> some sketches :D
>>
>>
>> 2014-12-02 1:53 GMT+01:00 Connor Doyle <co...@mesosphere.io>:
>>
>>> Hi Dave,
>>>
>>> This is a timely topic, since we have been prototyping and mocking up
>>> something similar at Mesosphere.  We created a new public GitHub repository
>>> for it about three weeks ago called "universe" (
>>> http://github.com/mesosphere/universe).
>>>
>>> Although we have added some informal specs, it's very malleable at this
>>> point.  We're very much interested in making our "universe" compatible
>>> with, or the same as, the registry you're proposing.  Without delving into
>>> implementation details, some of the goals we have in mind are outlined
>>> below.
>>>
>>> Data Source:
>>>
>>> The package repository should be easily consumable by third-party
>>> command-line and other programs.  There should be a condensed “index”
>>> representation of the package repository available.
>>>
>>> Packages within the repository should be versioned.
>>>
>>> The package repository format itself should be versioned.
>>>
>>> Decentralization and Composability:
>>>
>>> The package metadata should be hosted in a public place (we like GitHub)
>>> so that additional packages can be added by the community by simply
>>> submitting pull requests.  We have added some rudimentary commit hooks and
>>> automated validation to protect the repo against breaking changes.
>>>
>>> It’s important that no single entity “owns the keys” to the universe,
>>> and that the spec and implementation remain public.  It should be easy and
>>> free for organizations to maintain a private package repository.
>>>
>>> A corollary is that it should be easy for consumers to pull from a
>>> hierarchy of upstream repositories.  One setup we have in mind is that an
>>> organization might have staging and production repositories running
>>> internally.  Packages are pushed to staging where integration testing can
>>> run before “deployment” to production.  If a package isn’t in the local
>>> repository it might be looked up and installed from upstream.
>>>
>>> <hierarchy.png>
>>>
>>>
>>> Repositories should be able to be proxied and cached in this way.
>>> Organizations should be able to isolate their datacenter but also
>>> selectively add external packages for experimentation. The system should be
>>> sufficiently portable and extensible to accomodate these and similar use
>>> cases.
>>>
>>> Meta-Framework Descriptors:
>>>
>>> Our conception of the package repository is a bit more expansive than
>>> just Mesos frameworks; it includes descriptions of how to install any piece
>>> of server software on a Mesos cluster.  Frameworks and non-frameworks alike
>>> may be installed using some other meta-framework that’s responsible for
>>> starting all other cluster services.  Likely candidates for this role are
>>> the long-lived frameworks: Aurora, Marathon, Singularity, and eventually
>>> Kubernetes.  In any case, the repository spec should not be prescriptive
>>> with respect to this choice.
>>>
>>> The package repository metadata should make it easy for Mesos framework
>>> authors (and authors of non-Mesos-aware programs) to describe how to
>>> install their software on a Mesos cluster.  To this end, our prototype
>>> package spec allows for Meta-framework descriptor files for each package in
>>> the repository.  For example for a given package we might see a
>>> `marathon.json` file as well as a `my-app.aurora` file.
>>>
>>> An obvious concern is how to specify site-specific arguments upon
>>> installation.  Here packages should describe data that must be marshalled
>>> from the environment (e.g. by prompting a user) and combined with the raw
>>> meta-framework descriptor to launch the app.  These configuration
>>> parameters should be agnostic of the supported meta-frameworks.  More
>>> concretely, in our prototype we describe configuration data in terms of a
>>> JSON-Schema.
>>>
>>> CLI Integration:
>>>
>>> Part of our proposed package format is an optional descriptor for how to
>>> fetch and install the command-line tools for interacting with the
>>> application.  For now, we only have one implementation of this, which is to
>>> fetch a python egg from PyPI.
>>>
>>> Governance:
>>>
>>> All in all, we think that making this effort more community driven is a
>>> healthy way to proceed.  Any input is very welcome.  For example, if others
>>> think that what we have is a good starting point we could transfer
>>> ownership of the repository to the mesos organization on GitHub.
>>>
>>> Cheers,
>>> --
>>> Connor Doyle
>>> http://mesosphere.com
>>>
>>>
>>>
>>>
>>>
>>> On Nov 30, 2014, at 17:32, Dave Lester <da...@gmail.com> wrote:
>>>
>>> As the number of Mesos frameworks grows (and now, a module system), I
>>> think it's time to create a community-maintained registry with the goal of
>>> making frameworks and modules easier to discover, contribute to, and
>>> install.
>>>
>>> There's already a JIRA ticket tracking this (MESOS-1759) and I've
>>> chatted with several folks (thanks in particular Victor Vieux, Tom Arnfeld,
>>> Vinod Kone, Timothy St Clair, and Joe Stein). I'd like to advance the
>>> conversation by offering a proposal on the public mailing list.
>>>
>>> I imagine two initiatives to achieve this:
>>>
>>> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with
>>> how Jenkins maintains their plugins on GitHub [1], but they allow
>>> individual plugins to have their own repo within their GH org. Plugins are
>>> repos with a specific naming convention (in their case, they've appended
>>> "-plugin" to the repo name but this isn't the same approach we'd need to
>>> take). Having a shared GH org creates greater visibility to what people are
>>> doing, and encourages community participation and ownership.
>>>
>>> In the case of Mesos, not all frameworks will jump at the chance to have
>>> community hosting which is fine. But particularly for smaller frameworks, I
>>> think this is a good option given the success Jenkins has had. We
>>> could potentially use the existing /mesos github org, or create a new org
>>> /mesos-extensions or something of the like. It seems like the only
>>> potential snag here is that we'll want to be explicit that these aren't
>>> Apache-blessed plugins and instead maintained by the larger ecosystem, but
>>> I believe we can achieve that by offering a notice in the GH org
>>> description.
>>>
>>> 2) A registry that allows framework writers to publish new versions of
>>> their frameworks to a central repository that could be programmatically
>>> accessed and rendered online. The v1 could be incredibly simple, but I
>>> think this is a foundational piece that can grow with the project in the
>>> future. Since this is a little bit more-involved, I've created a separate
>>> Google Doc [2] to begin drafting the scope for this:
>>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing,
>>> and have intentionally punted on some of the implementation details until
>>> the scope is finalized and I gauge interest from users and potential
>>> collaborators.
>>>
>>> What do you think? If folks are on board I will assign the JIRA issue to
>>> myself and get to work; I'd also welcome collaborators to help get this off
>>> the ground since I think it will be a boost for the community. Feedback
>>> is welcome!
>>>
>>> Thanks,
>>> Dave
>>>
>>> [1] https://github.com/jenkinsci/
>>> [2]
>>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing
>>>
>>>
>>>
>>
>

Re: Proposal: shared Mesos framework hosting and registry

[Dave Lester <da...@gmail.com>]

Hey Connor,

It would be great to hear back about this and get the ball rolling on a
shared registry. Let me know what you think!

Thanks,
Dave

On Wed, Dec 3, 2014 at 1:11 PM, Dave Lester <da...@gmail.com> wrote:
>
> Hi Connor,
>
> Thanks for sharing the work you've been doing with Universe. Sounds
> complimentary and pretty exciting! I thought I'd take a stab at making some
> clarifications.
>
> To rephrase your proposal by using a comparison, you're proposing the
> "homebrew model" of package management here in terms of how the registry is
> maintained, correct? A registry of metadata that's hosted via a single
> GitHub repository, that community members submit pull requests to when they
> want to have a release included in the registry.
>
> In this case of Homebrew, they manage "formulas"; in our case it would be
> package.json files (which is how NodeJS handles package info).
>
> *Architecture*
> The only way Universe diverges from my original thinking is that you're
> advocating decentralization of package metadata, whereas my original
> thought was to have it much more centralized. There's no reason it couldn't
> be brought together through a single interface (command-line, or searchable
> web interface) so I don't think it's incompatible technically, although I
> do think it creates some potential social/governance overhead (more below).
>
> *Metadata*
> The package.json file is minimal and similar to what you see in most
> package management systems. Where you're taking it a step further is a JSON
> manifest for how to actually run the program, which is outside of the scope
> of what I was hoping to accomplish but it looks awesome.
>
> *Managing The Registry*
> One part of your proposal that's unclear to me is when information is
> pushed to the registry repository, and how often (for each release, or just
> when the package is first added). Additionally, what information is pushed
> here -- just a pointer to the git repo? It's my current understanding that
> the package.json and framework.json files are both within the actual
> package's repo, not the registry. I would love to hear how you're thinking
> about this.
>
> If it's just a pointer to another repo, it would make it possible for any
> package already vetted to become part of the registry to do their own
> updates without requiring someone to accept a PR. I think this is ideal. If
> it's the latter, it brings up some interesting questions related to who
> would manage this and under what guidelines they would do this vetting;
> we'd essentially get a PR for every release within the community at that
> point.
>
> *The Registry "Home"*
> I feel strongly that this should be a community-driven effort, and I see
> your proposal along with my own as very compatible. Would love to work
> together, and if this works as I hope it will be a huge driver of activity
> in the ecosystem.
>
> Dave
>
> On Tue, Dec 2, 2014 at 1:43 AM, Jing Dong <ji...@qubitdigital.com> wrote:
>
>> I like the idea of having a single UI for installing frameworks. One Web
>> UI endpoint for managing setting and view state of multiple installed
>> frameworks.
>>
>>
>> On 2 Dec 2014, at 09:26, Billy Bones <ga...@gmail.com> wrote:
>>
>> Wooo mesosphere is making such a good work pushing forward Mesos
>> technologies!
>>
>> I deeply think that Mesos should implement registry as a core feature.
>>
>> Let me explain that, Installing manually frameworks / modules / other
>> third party adds-on is quite cool when you're beginner and want to learn
>> how the things works, but once you're in production, you've to be fast and
>> react quickly.
>>
>> Let said that I'm a system engineer that have customers requests like
>> "Hum... Could you had this " insert Framework/module/other name here" on
>> our cluster region?", I'll have to add it as soon and quick as possible on
>> the targeted environment, I'll be OK to do it twice, but I wont make my job
>> a request nightmare and I'll probably search for a way to provide to my
>> customers a restrictive access (WebUI) to the cluster manager and allow
>> them to deploy those new add-on by themself.
>>
>> With this study case in mind, I'll be happy to have a registry which
>> allow a filtering mechanism to hide or show frameworks/modules/etc by
>> status (Official Apache Content / Public Content / Private Content).
>>
>> Here is my thoughts about the registry articulation:
>>
>>    - Mesos Registry is an integrated part of Mesos master services.
>>    - Mesos Registry is an endpoint available to WebUI and CLI.
>>    - Mesos Registry is nothing but a metadata registry.
>>    - Mesos Registry save its configs and metadata on a key/value store
>>    (zookeeper?).
>>    - Mesos Registry is empty at the first launch.
>>    - Mesos Registry as three views: Official Apache Content | Publicly
>>    Maintained Content | Private Maintained Content
>>    - Mesos Registry views content is:
>>    - Official Apache Content == Link to existing Apache add-ons hosted
>>       on Github/Git repository + metadata (Like those proposed by Dave and
>>       Connor).
>>       - Publicly Maintained Content == Links to existing repositories
>>       (Github / Git / other) + metadata (Like those proposed by Dave and Connor).
>>       - Private Content == Links to existing private (GIT/GITHUB/Other
>>       repositories) + metadata (Like those proposed by Dave and Connor), this
>>       repository is kinda special as it is hosted and created by the cluster
>>       operators and could be a mixed content of locally maintained repository
>>       (GIT repos on a HDFS or TAR on HDFS) and public content repository cloned
>>       from the public content URL/Metadata.
>>
>> I don't know if I'm really clear, so if I'm not, let me know it, I'll do
>> some sketches :D
>>
>>
>> 2014-12-02 1:53 GMT+01:00 Connor Doyle <co...@mesosphere.io>:
>>
>>> Hi Dave,
>>>
>>> This is a timely topic, since we have been prototyping and mocking up
>>> something similar at Mesosphere.  We created a new public GitHub repository
>>> for it about three weeks ago called "universe" (
>>> http://github.com/mesosphere/universe).
>>>
>>> Although we have added some informal specs, it's very malleable at this
>>> point.  We're very much interested in making our "universe" compatible
>>> with, or the same as, the registry you're proposing.  Without delving into
>>> implementation details, some of the goals we have in mind are outlined
>>> below.
>>>
>>> Data Source:
>>>
>>> The package repository should be easily consumable by third-party
>>> command-line and other programs.  There should be a condensed “index”
>>> representation of the package repository available.
>>>
>>> Packages within the repository should be versioned.
>>>
>>> The package repository format itself should be versioned.
>>>
>>> Decentralization and Composability:
>>>
>>> The package metadata should be hosted in a public place (we like GitHub)
>>> so that additional packages can be added by the community by simply
>>> submitting pull requests.  We have added some rudimentary commit hooks and
>>> automated validation to protect the repo against breaking changes.
>>>
>>> It’s important that no single entity “owns the keys” to the universe,
>>> and that the spec and implementation remain public.  It should be easy and
>>> free for organizations to maintain a private package repository.
>>>
>>> A corollary is that it should be easy for consumers to pull from a
>>> hierarchy of upstream repositories.  One setup we have in mind is that an
>>> organization might have staging and production repositories running
>>> internally.  Packages are pushed to staging where integration testing can
>>> run before “deployment” to production.  If a package isn’t in the local
>>> repository it might be looked up and installed from upstream.
>>>
>>> <hierarchy.png>
>>>
>>>
>>> Repositories should be able to be proxied and cached in this way.
>>> Organizations should be able to isolate their datacenter but also
>>> selectively add external packages for experimentation. The system should be
>>> sufficiently portable and extensible to accomodate these and similar use
>>> cases.
>>>
>>> Meta-Framework Descriptors:
>>>
>>> Our conception of the package repository is a bit more expansive than
>>> just Mesos frameworks; it includes descriptions of how to install any piece
>>> of server software on a Mesos cluster.  Frameworks and non-frameworks alike
>>> may be installed using some other meta-framework that’s responsible for
>>> starting all other cluster services.  Likely candidates for this role are
>>> the long-lived frameworks: Aurora, Marathon, Singularity, and eventually
>>> Kubernetes.  In any case, the repository spec should not be prescriptive
>>> with respect to this choice.
>>>
>>> The package repository metadata should make it easy for Mesos framework
>>> authors (and authors of non-Mesos-aware programs) to describe how to
>>> install their software on a Mesos cluster.  To this end, our prototype
>>> package spec allows for Meta-framework descriptor files for each package in
>>> the repository.  For example for a given package we might see a
>>> `marathon.json` file as well as a `my-app.aurora` file.
>>>
>>> An obvious concern is how to specify site-specific arguments upon
>>> installation.  Here packages should describe data that must be marshalled
>>> from the environment (e.g. by prompting a user) and combined with the raw
>>> meta-framework descriptor to launch the app.  These configuration
>>> parameters should be agnostic of the supported meta-frameworks.  More
>>> concretely, in our prototype we describe configuration data in terms of a
>>> JSON-Schema.
>>>
>>> CLI Integration:
>>>
>>> Part of our proposed package format is an optional descriptor for how to
>>> fetch and install the command-line tools for interacting with the
>>> application.  For now, we only have one implementation of this, which is to
>>> fetch a python egg from PyPI.
>>>
>>> Governance:
>>>
>>> All in all, we think that making this effort more community driven is a
>>> healthy way to proceed.  Any input is very welcome.  For example, if others
>>> think that what we have is a good starting point we could transfer
>>> ownership of the repository to the mesos organization on GitHub.
>>>
>>> Cheers,
>>> --
>>> Connor Doyle
>>> http://mesosphere.com
>>>
>>>
>>>
>>>
>>>
>>> On Nov 30, 2014, at 17:32, Dave Lester <da...@gmail.com> wrote:
>>>
>>> As the number of Mesos frameworks grows (and now, a module system), I
>>> think it's time to create a community-maintained registry with the goal of
>>> making frameworks and modules easier to discover, contribute to, and
>>> install.
>>>
>>> There's already a JIRA ticket tracking this (MESOS-1759) and I've
>>> chatted with several folks (thanks in particular Victor Vieux, Tom Arnfeld,
>>> Vinod Kone, Timothy St Clair, and Joe Stein). I'd like to advance the
>>> conversation by offering a proposal on the public mailing list.
>>>
>>> I imagine two initiatives to achieve this:
>>>
>>> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with
>>> how Jenkins maintains their plugins on GitHub [1], but they allow
>>> individual plugins to have their own repo within their GH org. Plugins are
>>> repos with a specific naming convention (in their case, they've appended
>>> "-plugin" to the repo name but this isn't the same approach we'd need to
>>> take). Having a shared GH org creates greater visibility to what people are
>>> doing, and encourages community participation and ownership.
>>>
>>> In the case of Mesos, not all frameworks will jump at the chance to have
>>> community hosting which is fine. But particularly for smaller frameworks, I
>>> think this is a good option given the success Jenkins has had. We
>>> could potentially use the existing /mesos github org, or create a new org
>>> /mesos-extensions or something of the like. It seems like the only
>>> potential snag here is that we'll want to be explicit that these aren't
>>> Apache-blessed plugins and instead maintained by the larger ecosystem, but
>>> I believe we can achieve that by offering a notice in the GH org
>>> description.
>>>
>>> 2) A registry that allows framework writers to publish new versions of
>>> their frameworks to a central repository that could be programmatically
>>> accessed and rendered online. The v1 could be incredibly simple, but I
>>> think this is a foundational piece that can grow with the project in the
>>> future. Since this is a little bit more-involved, I've created a separate
>>> Google Doc [2] to begin drafting the scope for this:
>>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing,
>>> and have intentionally punted on some of the implementation details until
>>> the scope is finalized and I gauge interest from users and potential
>>> collaborators.
>>>
>>> What do you think? If folks are on board I will assign the JIRA issue to
>>> myself and get to work; I'd also welcome collaborators to help get this off
>>> the ground since I think it will be a boost for the community. Feedback
>>> is welcome!
>>>
>>> Thanks,
>>> Dave
>>>
>>> [1] https://github.com/jenkinsci/
>>> [2]
>>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing
>>>
>>>
>>>
>>
>

Re: Proposal: shared Mesos framework hosting and registry

[Dave Lester <da...@gmail.com>]

Hi Connor,

Thanks for sharing the work you've been doing with Universe. Sounds
complimentary and pretty exciting! I thought I'd take a stab at making some
clarifications.

To rephrase your proposal by using a comparison, you're proposing the
"homebrew model" of package management here in terms of how the registry is
maintained, correct? A registry of metadata that's hosted via a single
GitHub repository, that community members submit pull requests to when they
want to have a release included in the registry.

In this case of Homebrew, they manage "formulas"; in our case it would be
package.json files (which is how NodeJS handles package info).

*Architecture*
The only way Universe diverges from my original thinking is that you're
advocating decentralization of package metadata, whereas my original
thought was to have it much more centralized. There's no reason it couldn't
be brought together through a single interface (command-line, or searchable
web interface) so I don't think it's incompatible technically, although I
do think it creates some potential social/governance overhead (more below).

*Metadata*
The package.json file is minimal and similar to what you see in most
package management systems. Where you're taking it a step further is a JSON
manifest for how to actually run the program, which is outside of the scope
of what I was hoping to accomplish but it looks awesome.

*Managing The Registry*
One part of your proposal that's unclear to me is when information is
pushed to the registry repository, and how often (for each release, or just
when the package is first added). Additionally, what information is pushed
here -- just a pointer to the git repo? It's my current understanding that
the package.json and framework.json files are both within the actual
package's repo, not the registry. I would love to hear how you're thinking
about this.

If it's just a pointer to another repo, it would make it possible for any
package already vetted to become part of the registry to do their own
updates without requiring someone to accept a PR. I think this is ideal. If
it's the latter, it brings up some interesting questions related to who
would manage this and under what guidelines they would do this vetting;
we'd essentially get a PR for every release within the community at that
point.

*The Registry "Home"*
I feel strongly that this should be a community-driven effort, and I see
your proposal along with my own as very compatible. Would love to work
together, and if this works as I hope it will be a huge driver of activity
in the ecosystem.

Dave

On Tue, Dec 2, 2014 at 1:43 AM, Jing Dong <ji...@qubitdigital.com> wrote:

> I like the idea of having a single UI for installing frameworks. One Web
> UI endpoint for managing setting and view state of multiple installed
> frameworks.
>
>
> On 2 Dec 2014, at 09:26, Billy Bones <ga...@gmail.com> wrote:
>
> Wooo mesosphere is making such a good work pushing forward Mesos
> technologies!
>
> I deeply think that Mesos should implement registry as a core feature.
>
> Let me explain that, Installing manually frameworks / modules / other
> third party adds-on is quite cool when you're beginner and want to learn
> how the things works, but once you're in production, you've to be fast and
> react quickly.
>
> Let said that I'm a system engineer that have customers requests like
> "Hum... Could you had this " insert Framework/module/other name here" on
> our cluster region?", I'll have to add it as soon and quick as possible on
> the targeted environment, I'll be OK to do it twice, but I wont make my job
> a request nightmare and I'll probably search for a way to provide to my
> customers a restrictive access (WebUI) to the cluster manager and allow
> them to deploy those new add-on by themself.
>
> With this study case in mind, I'll be happy to have a registry which allow
> a filtering mechanism to hide or show frameworks/modules/etc by status
> (Official Apache Content / Public Content / Private Content).
>
> Here is my thoughts about the registry articulation:
>
>    - Mesos Registry is an integrated part of Mesos master services.
>    - Mesos Registry is an endpoint available to WebUI and CLI.
>    - Mesos Registry is nothing but a metadata registry.
>    - Mesos Registry save its configs and metadata on a key/value store
>    (zookeeper?).
>    - Mesos Registry is empty at the first launch.
>    - Mesos Registry as three views: Official Apache Content | Publicly
>    Maintained Content | Private Maintained Content
>    - Mesos Registry views content is:
>    - Official Apache Content == Link to existing Apache add-ons hosted on
>       Github/Git repository + metadata (Like those proposed by Dave and Connor).
>       - Publicly Maintained Content == Links to existing repositories
>       (Github / Git / other) + metadata (Like those proposed by Dave and Connor).
>       - Private Content == Links to existing private (GIT/GITHUB/Other
>       repositories) + metadata (Like those proposed by Dave and Connor), this
>       repository is kinda special as it is hosted and created by the cluster
>       operators and could be a mixed content of locally maintained repository
>       (GIT repos on a HDFS or TAR on HDFS) and public content repository cloned
>       from the public content URL/Metadata.
>
> I don't know if I'm really clear, so if I'm not, let me know it, I'll do
> some sketches :D
>
>
> 2014-12-02 1:53 GMT+01:00 Connor Doyle <co...@mesosphere.io>:
>
>> Hi Dave,
>>
>> This is a timely topic, since we have been prototyping and mocking up
>> something similar at Mesosphere.  We created a new public GitHub repository
>> for it about three weeks ago called "universe" (
>> http://github.com/mesosphere/universe).
>>
>> Although we have added some informal specs, it's very malleable at this
>> point.  We're very much interested in making our "universe" compatible
>> with, or the same as, the registry you're proposing.  Without delving into
>> implementation details, some of the goals we have in mind are outlined
>> below.
>>
>> Data Source:
>>
>> The package repository should be easily consumable by third-party
>> command-line and other programs.  There should be a condensed “index”
>> representation of the package repository available.
>>
>> Packages within the repository should be versioned.
>>
>> The package repository format itself should be versioned.
>>
>> Decentralization and Composability:
>>
>> The package metadata should be hosted in a public place (we like GitHub)
>> so that additional packages can be added by the community by simply
>> submitting pull requests.  We have added some rudimentary commit hooks and
>> automated validation to protect the repo against breaking changes.
>>
>> It’s important that no single entity “owns the keys” to the universe, and
>> that the spec and implementation remain public.  It should be easy and free
>> for organizations to maintain a private package repository.
>>
>> A corollary is that it should be easy for consumers to pull from a
>> hierarchy of upstream repositories.  One setup we have in mind is that an
>> organization might have staging and production repositories running
>> internally.  Packages are pushed to staging where integration testing can
>> run before “deployment” to production.  If a package isn’t in the local
>> repository it might be looked up and installed from upstream.
>>
>> <hierarchy.png>
>>
>>
>> Repositories should be able to be proxied and cached in this way.
>> Organizations should be able to isolate their datacenter but also
>> selectively add external packages for experimentation. The system should be
>> sufficiently portable and extensible to accomodate these and similar use
>> cases.
>>
>> Meta-Framework Descriptors:
>>
>> Our conception of the package repository is a bit more expansive than
>> just Mesos frameworks; it includes descriptions of how to install any piece
>> of server software on a Mesos cluster.  Frameworks and non-frameworks alike
>> may be installed using some other meta-framework that’s responsible for
>> starting all other cluster services.  Likely candidates for this role are
>> the long-lived frameworks: Aurora, Marathon, Singularity, and eventually
>> Kubernetes.  In any case, the repository spec should not be prescriptive
>> with respect to this choice.
>>
>> The package repository metadata should make it easy for Mesos framework
>> authors (and authors of non-Mesos-aware programs) to describe how to
>> install their software on a Mesos cluster.  To this end, our prototype
>> package spec allows for Meta-framework descriptor files for each package in
>> the repository.  For example for a given package we might see a
>> `marathon.json` file as well as a `my-app.aurora` file.
>>
>> An obvious concern is how to specify site-specific arguments upon
>> installation.  Here packages should describe data that must be marshalled
>> from the environment (e.g. by prompting a user) and combined with the raw
>> meta-framework descriptor to launch the app.  These configuration
>> parameters should be agnostic of the supported meta-frameworks.  More
>> concretely, in our prototype we describe configuration data in terms of a
>> JSON-Schema.
>>
>> CLI Integration:
>>
>> Part of our proposed package format is an optional descriptor for how to
>> fetch and install the command-line tools for interacting with the
>> application.  For now, we only have one implementation of this, which is to
>> fetch a python egg from PyPI.
>>
>> Governance:
>>
>> All in all, we think that making this effort more community driven is a
>> healthy way to proceed.  Any input is very welcome.  For example, if others
>> think that what we have is a good starting point we could transfer
>> ownership of the repository to the mesos organization on GitHub.
>>
>> Cheers,
>> --
>> Connor Doyle
>> http://mesosphere.com
>>
>>
>>
>>
>>
>> On Nov 30, 2014, at 17:32, Dave Lester <da...@gmail.com> wrote:
>>
>> As the number of Mesos frameworks grows (and now, a module system), I
>> think it's time to create a community-maintained registry with the goal of
>> making frameworks and modules easier to discover, contribute to, and
>> install.
>>
>> There's already a JIRA ticket tracking this (MESOS-1759) and I've chatted
>> with several folks (thanks in particular Victor Vieux, Tom Arnfeld, Vinod
>> Kone, Timothy St Clair, and Joe Stein). I'd like to advance the
>> conversation by offering a proposal on the public mailing list.
>>
>> I imagine two initiatives to achieve this:
>>
>> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with
>> how Jenkins maintains their plugins on GitHub [1], but they allow
>> individual plugins to have their own repo within their GH org. Plugins are
>> repos with a specific naming convention (in their case, they've appended
>> "-plugin" to the repo name but this isn't the same approach we'd need to
>> take). Having a shared GH org creates greater visibility to what people are
>> doing, and encourages community participation and ownership.
>>
>> In the case of Mesos, not all frameworks will jump at the chance to have
>> community hosting which is fine. But particularly for smaller frameworks, I
>> think this is a good option given the success Jenkins has had. We
>> could potentially use the existing /mesos github org, or create a new org
>> /mesos-extensions or something of the like. It seems like the only
>> potential snag here is that we'll want to be explicit that these aren't
>> Apache-blessed plugins and instead maintained by the larger ecosystem, but
>> I believe we can achieve that by offering a notice in the GH org
>> description.
>>
>> 2) A registry that allows framework writers to publish new versions of
>> their frameworks to a central repository that could be programmatically
>> accessed and rendered online. The v1 could be incredibly simple, but I
>> think this is a foundational piece that can grow with the project in the
>> future. Since this is a little bit more-involved, I've created a separate
>> Google Doc [2] to begin drafting the scope for this:
>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing,
>> and have intentionally punted on some of the implementation details until
>> the scope is finalized and I gauge interest from users and potential
>> collaborators.
>>
>> What do you think? If folks are on board I will assign the JIRA issue to
>> myself and get to work; I'd also welcome collaborators to help get this off
>> the ground since I think it will be a boost for the community. Feedback
>> is welcome!
>>
>> Thanks,
>> Dave
>>
>> [1] https://github.com/jenkinsci/
>> [2]
>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing
>>
>>
>>
>

Re: Proposal: shared Mesos framework hosting and registry

[Dave Lester <da...@gmail.com>]

Hi Connor,

Thanks for sharing the work you've been doing with Universe. Sounds
complimentary and pretty exciting! I thought I'd take a stab at making some
clarifications.

To rephrase your proposal by using a comparison, you're proposing the
"homebrew model" of package management here in terms of how the registry is
maintained, correct? A registry of metadata that's hosted via a single
GitHub repository, that community members submit pull requests to when they
want to have a release included in the registry.

In this case of Homebrew, they manage "formulas"; in our case it would be
package.json files (which is how NodeJS handles package info).

*Architecture*
The only way Universe diverges from my original thinking is that you're
advocating decentralization of package metadata, whereas my original
thought was to have it much more centralized. There's no reason it couldn't
be brought together through a single interface (command-line, or searchable
web interface) so I don't think it's incompatible technically, although I
do think it creates some potential social/governance overhead (more below).

*Metadata*
The package.json file is minimal and similar to what you see in most
package management systems. Where you're taking it a step further is a JSON
manifest for how to actually run the program, which is outside of the scope
of what I was hoping to accomplish but it looks awesome.

*Managing The Registry*
One part of your proposal that's unclear to me is when information is
pushed to the registry repository, and how often (for each release, or just
when the package is first added). Additionally, what information is pushed
here -- just a pointer to the git repo? It's my current understanding that
the package.json and framework.json files are both within the actual
package's repo, not the registry. I would love to hear how you're thinking
about this.

If it's just a pointer to another repo, it would make it possible for any
package already vetted to become part of the registry to do their own
updates without requiring someone to accept a PR. I think this is ideal. If
it's the latter, it brings up some interesting questions related to who
would manage this and under what guidelines they would do this vetting;
we'd essentially get a PR for every release within the community at that
point.

*The Registry "Home"*
I feel strongly that this should be a community-driven effort, and I see
your proposal along with my own as very compatible. Would love to work
together, and if this works as I hope it will be a huge driver of activity
in the ecosystem.

Dave

On Tue, Dec 2, 2014 at 1:43 AM, Jing Dong <ji...@qubitdigital.com> wrote:

> I like the idea of having a single UI for installing frameworks. One Web
> UI endpoint for managing setting and view state of multiple installed
> frameworks.
>
>
> On 2 Dec 2014, at 09:26, Billy Bones <ga...@gmail.com> wrote:
>
> Wooo mesosphere is making such a good work pushing forward Mesos
> technologies!
>
> I deeply think that Mesos should implement registry as a core feature.
>
> Let me explain that, Installing manually frameworks / modules / other
> third party adds-on is quite cool when you're beginner and want to learn
> how the things works, but once you're in production, you've to be fast and
> react quickly.
>
> Let said that I'm a system engineer that have customers requests like
> "Hum... Could you had this " insert Framework/module/other name here" on
> our cluster region?", I'll have to add it as soon and quick as possible on
> the targeted environment, I'll be OK to do it twice, but I wont make my job
> a request nightmare and I'll probably search for a way to provide to my
> customers a restrictive access (WebUI) to the cluster manager and allow
> them to deploy those new add-on by themself.
>
> With this study case in mind, I'll be happy to have a registry which allow
> a filtering mechanism to hide or show frameworks/modules/etc by status
> (Official Apache Content / Public Content / Private Content).
>
> Here is my thoughts about the registry articulation:
>
>    - Mesos Registry is an integrated part of Mesos master services.
>    - Mesos Registry is an endpoint available to WebUI and CLI.
>    - Mesos Registry is nothing but a metadata registry.
>    - Mesos Registry save its configs and metadata on a key/value store
>    (zookeeper?).
>    - Mesos Registry is empty at the first launch.
>    - Mesos Registry as three views: Official Apache Content | Publicly
>    Maintained Content | Private Maintained Content
>    - Mesos Registry views content is:
>    - Official Apache Content == Link to existing Apache add-ons hosted on
>       Github/Git repository + metadata (Like those proposed by Dave and Connor).
>       - Publicly Maintained Content == Links to existing repositories
>       (Github / Git / other) + metadata (Like those proposed by Dave and Connor).
>       - Private Content == Links to existing private (GIT/GITHUB/Other
>       repositories) + metadata (Like those proposed by Dave and Connor), this
>       repository is kinda special as it is hosted and created by the cluster
>       operators and could be a mixed content of locally maintained repository
>       (GIT repos on a HDFS or TAR on HDFS) and public content repository cloned
>       from the public content URL/Metadata.
>
> I don't know if I'm really clear, so if I'm not, let me know it, I'll do
> some sketches :D
>
>
> 2014-12-02 1:53 GMT+01:00 Connor Doyle <co...@mesosphere.io>:
>
>> Hi Dave,
>>
>> This is a timely topic, since we have been prototyping and mocking up
>> something similar at Mesosphere.  We created a new public GitHub repository
>> for it about three weeks ago called "universe" (
>> http://github.com/mesosphere/universe).
>>
>> Although we have added some informal specs, it's very malleable at this
>> point.  We're very much interested in making our "universe" compatible
>> with, or the same as, the registry you're proposing.  Without delving into
>> implementation details, some of the goals we have in mind are outlined
>> below.
>>
>> Data Source:
>>
>> The package repository should be easily consumable by third-party
>> command-line and other programs.  There should be a condensed “index”
>> representation of the package repository available.
>>
>> Packages within the repository should be versioned.
>>
>> The package repository format itself should be versioned.
>>
>> Decentralization and Composability:
>>
>> The package metadata should be hosted in a public place (we like GitHub)
>> so that additional packages can be added by the community by simply
>> submitting pull requests.  We have added some rudimentary commit hooks and
>> automated validation to protect the repo against breaking changes.
>>
>> It’s important that no single entity “owns the keys” to the universe, and
>> that the spec and implementation remain public.  It should be easy and free
>> for organizations to maintain a private package repository.
>>
>> A corollary is that it should be easy for consumers to pull from a
>> hierarchy of upstream repositories.  One setup we have in mind is that an
>> organization might have staging and production repositories running
>> internally.  Packages are pushed to staging where integration testing can
>> run before “deployment” to production.  If a package isn’t in the local
>> repository it might be looked up and installed from upstream.
>>
>> <hierarchy.png>
>>
>>
>> Repositories should be able to be proxied and cached in this way.
>> Organizations should be able to isolate their datacenter but also
>> selectively add external packages for experimentation. The system should be
>> sufficiently portable and extensible to accomodate these and similar use
>> cases.
>>
>> Meta-Framework Descriptors:
>>
>> Our conception of the package repository is a bit more expansive than
>> just Mesos frameworks; it includes descriptions of how to install any piece
>> of server software on a Mesos cluster.  Frameworks and non-frameworks alike
>> may be installed using some other meta-framework that’s responsible for
>> starting all other cluster services.  Likely candidates for this role are
>> the long-lived frameworks: Aurora, Marathon, Singularity, and eventually
>> Kubernetes.  In any case, the repository spec should not be prescriptive
>> with respect to this choice.
>>
>> The package repository metadata should make it easy for Mesos framework
>> authors (and authors of non-Mesos-aware programs) to describe how to
>> install their software on a Mesos cluster.  To this end, our prototype
>> package spec allows for Meta-framework descriptor files for each package in
>> the repository.  For example for a given package we might see a
>> `marathon.json` file as well as a `my-app.aurora` file.
>>
>> An obvious concern is how to specify site-specific arguments upon
>> installation.  Here packages should describe data that must be marshalled
>> from the environment (e.g. by prompting a user) and combined with the raw
>> meta-framework descriptor to launch the app.  These configuration
>> parameters should be agnostic of the supported meta-frameworks.  More
>> concretely, in our prototype we describe configuration data in terms of a
>> JSON-Schema.
>>
>> CLI Integration:
>>
>> Part of our proposed package format is an optional descriptor for how to
>> fetch and install the command-line tools for interacting with the
>> application.  For now, we only have one implementation of this, which is to
>> fetch a python egg from PyPI.
>>
>> Governance:
>>
>> All in all, we think that making this effort more community driven is a
>> healthy way to proceed.  Any input is very welcome.  For example, if others
>> think that what we have is a good starting point we could transfer
>> ownership of the repository to the mesos organization on GitHub.
>>
>> Cheers,
>> --
>> Connor Doyle
>> http://mesosphere.com
>>
>>
>>
>>
>>
>> On Nov 30, 2014, at 17:32, Dave Lester <da...@gmail.com> wrote:
>>
>> As the number of Mesos frameworks grows (and now, a module system), I
>> think it's time to create a community-maintained registry with the goal of
>> making frameworks and modules easier to discover, contribute to, and
>> install.
>>
>> There's already a JIRA ticket tracking this (MESOS-1759) and I've chatted
>> with several folks (thanks in particular Victor Vieux, Tom Arnfeld, Vinod
>> Kone, Timothy St Clair, and Joe Stein). I'd like to advance the
>> conversation by offering a proposal on the public mailing list.
>>
>> I imagine two initiatives to achieve this:
>>
>> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with
>> how Jenkins maintains their plugins on GitHub [1], but they allow
>> individual plugins to have their own repo within their GH org. Plugins are
>> repos with a specific naming convention (in their case, they've appended
>> "-plugin" to the repo name but this isn't the same approach we'd need to
>> take). Having a shared GH org creates greater visibility to what people are
>> doing, and encourages community participation and ownership.
>>
>> In the case of Mesos, not all frameworks will jump at the chance to have
>> community hosting which is fine. But particularly for smaller frameworks, I
>> think this is a good option given the success Jenkins has had. We
>> could potentially use the existing /mesos github org, or create a new org
>> /mesos-extensions or something of the like. It seems like the only
>> potential snag here is that we'll want to be explicit that these aren't
>> Apache-blessed plugins and instead maintained by the larger ecosystem, but
>> I believe we can achieve that by offering a notice in the GH org
>> description.
>>
>> 2) A registry that allows framework writers to publish new versions of
>> their frameworks to a central repository that could be programmatically
>> accessed and rendered online. The v1 could be incredibly simple, but I
>> think this is a foundational piece that can grow with the project in the
>> future. Since this is a little bit more-involved, I've created a separate
>> Google Doc [2] to begin drafting the scope for this:
>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing,
>> and have intentionally punted on some of the implementation details until
>> the scope is finalized and I gauge interest from users and potential
>> collaborators.
>>
>> What do you think? If folks are on board I will assign the JIRA issue to
>> myself and get to work; I'd also welcome collaborators to help get this off
>> the ground since I think it will be a boost for the community. Feedback
>> is welcome!
>>
>> Thanks,
>> Dave
>>
>> [1] https://github.com/jenkinsci/
>> [2]
>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing
>>
>>
>>
>

Re: Proposal: shared Mesos framework hosting and registry

[Jing Dong <ji...@qubitdigital.com>]

I like the idea of having a single UI for installing frameworks. One Web UI
endpoint for managing setting and view state of multiple installed
frameworks.


On 2 Dec 2014, at 09:26, Billy Bones <ga...@gmail.com> wrote:

Wooo mesosphere is making such a good work pushing forward Mesos
technologies!

I deeply think that Mesos should implement registry as a core feature.

Let me explain that, Installing manually frameworks / modules / other third
party adds-on is quite cool when you're beginner and want to learn how the
things works, but once you're in production, you've to be fast and react
quickly.

Let said that I'm a system engineer that have customers requests like
"Hum... Could you had this " insert Framework/module/other name here" on
our cluster region?", I'll have to add it as soon and quick as possible on
the targeted environment, I'll be OK to do it twice, but I wont make my job
a request nightmare and I'll probably search for a way to provide to my
customers a restrictive access (WebUI) to the cluster manager and allow
them to deploy those new add-on by themself.

With this study case in mind, I'll be happy to have a registry which allow
a filtering mechanism to hide or show frameworks/modules/etc by status
(Official Apache Content / Public Content / Private Content).

Here is my thoughts about the registry articulation:

   - Mesos Registry is an integrated part of Mesos master services.
   - Mesos Registry is an endpoint available to WebUI and CLI.
   - Mesos Registry is nothing but a metadata registry.
   - Mesos Registry save its configs and metadata on a key/value store
   (zookeeper?).
   - Mesos Registry is empty at the first launch.
   - Mesos Registry as three views: Official Apache Content | Publicly
   Maintained Content | Private Maintained Content
   - Mesos Registry views content is:
   - Official Apache Content == Link to existing Apache add-ons hosted on
      Github/Git repository + metadata (Like those proposed by Dave and Connor).
      - Publicly Maintained Content == Links to existing repositories
      (Github / Git / other) + metadata (Like those proposed by Dave
and Connor).
      - Private Content == Links to existing private (GIT/GITHUB/Other
      repositories) + metadata (Like those proposed by Dave and Connor), this
      repository is kinda special as it is hosted and created by the cluster
      operators and could be a mixed content of locally maintained repository
      (GIT repos on a HDFS or TAR on HDFS) and public content repository cloned
      from the public content URL/Metadata.

I don't know if I'm really clear, so if I'm not, let me know it, I'll do
some sketches :D


2014-12-02 1:53 GMT+01:00 Connor Doyle <co...@mesosphere.io>:

> Hi Dave,
>
> This is a timely topic, since we have been prototyping and mocking up
> something similar at Mesosphere.  We created a new public GitHub repository
> for it about three weeks ago called "universe" (
> http://github.com/mesosphere/universe).
>
> Although we have added some informal specs, it's very malleable at this
> point.  We're very much interested in making our "universe" compatible
> with, or the same as, the registry you're proposing.  Without delving into
> implementation details, some of the goals we have in mind are outlined
> below.
>
> Data Source:
>
> The package repository should be easily consumable by third-party
> command-line and other programs.  There should be a condensed “index”
> representation of the package repository available.
>
> Packages within the repository should be versioned.
>
> The package repository format itself should be versioned.
>
> Decentralization and Composability:
>
> The package metadata should be hosted in a public place (we like GitHub)
> so that additional packages can be added by the community by simply
> submitting pull requests.  We have added some rudimentary commit hooks and
> automated validation to protect the repo against breaking changes.
>
> It’s important that no single entity “owns the keys” to the universe, and
> that the spec and implementation remain public.  It should be easy and free
> for organizations to maintain a private package repository.
>
> A corollary is that it should be easy for consumers to pull from a
> hierarchy of upstream repositories.  One setup we have in mind is that an
> organization might have staging and production repositories running
> internally.  Packages are pushed to staging where integration testing can
> run before “deployment” to production.  If a package isn’t in the local
> repository it might be looked up and installed from upstream.
>
> <hierarchy.png>
>
> Repositories should be able to be proxied and cached in this way.
> Organizations should be able to isolate their datacenter but also
> selectively add external packages for experimentation. The system should be
> sufficiently portable and extensible to accomodate these and similar use
> cases.
>
> Meta-Framework Descriptors:
>
> Our conception of the package repository is a bit more expansive than just
> Mesos frameworks; it includes descriptions of how to install any piece of
> server software on a Mesos cluster.  Frameworks and non-frameworks alike
> may be installed using some other meta-framework that’s responsible for
> starting all other cluster services.  Likely candidates for this role are
> the long-lived frameworks: Aurora, Marathon, Singularity, and eventually
> Kubernetes.  In any case, the repository spec should not be prescriptive
> with respect to this choice.
>
> The package repository metadata should make it easy for Mesos framework
> authors (and authors of non-Mesos-aware programs) to describe how to
> install their software on a Mesos cluster.  To this end, our prototype
> package spec allows for Meta-framework descriptor files for each package in
> the repository.  For example for a given package we might see a
> `marathon.json` file as well as a `my-app.aurora` file.
>
> An obvious concern is how to specify site-specific arguments upon
> installation.  Here packages should describe data that must be marshalled
> from the environment (e.g. by prompting a user) and combined with the raw
> meta-framework descriptor to launch the app.  These configuration
> parameters should be agnostic of the supported meta-frameworks.  More
> concretely, in our prototype we describe configuration data in terms of a
> JSON-Schema.
>
> CLI Integration:
>
> Part of our proposed package format is an optional descriptor for how to
> fetch and install the command-line tools for interacting with the
> application.  For now, we only have one implementation of this, which is to
> fetch a python egg from PyPI.
>
> Governance:
>
> All in all, we think that making this effort more community driven is a
> healthy way to proceed.  Any input is very welcome.  For example, if others
> think that what we have is a good starting point we could transfer
> ownership of the repository to the mesos organization on GitHub.
>
> Cheers,
> --
> Connor Doyle
> http://mesosphere.com
>
>
>
>
>
> On Nov 30, 2014, at 17:32, Dave Lester <da...@gmail.com> wrote:
>
> As the number of Mesos frameworks grows (and now, a module system), I
> think it's time to create a community-maintained registry with the goal of
> making frameworks and modules easier to discover, contribute to, and
> install.
>
> There's already a JIRA ticket tracking this (MESOS-1759) and I've chatted
> with several folks (thanks in particular Victor Vieux, Tom Arnfeld, Vinod
> Kone, Timothy St Clair, and Joe Stein). I'd like to advance the
> conversation by offering a proposal on the public mailing list.
>
> I imagine two initiatives to achieve this:
>
> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with
> how Jenkins maintains their plugins on GitHub [1], but they allow
> individual plugins to have their own repo within their GH org. Plugins are
> repos with a specific naming convention (in their case, they've appended
> "-plugin" to the repo name but this isn't the same approach we'd need to
> take). Having a shared GH org creates greater visibility to what people are
> doing, and encourages community participation and ownership.
>
> In the case of Mesos, not all frameworks will jump at the chance to have
> community hosting which is fine. But particularly for smaller frameworks, I
> think this is a good option given the success Jenkins has had. We
> could potentially use the existing /mesos github org, or create a new org
> /mesos-extensions or something of the like. It seems like the only
> potential snag here is that we'll want to be explicit that these aren't
> Apache-blessed plugins and instead maintained by the larger ecosystem, but
> I believe we can achieve that by offering a notice in the GH org
> description.
>
> 2) A registry that allows framework writers to publish new versions of
> their frameworks to a central repository that could be programmatically
> accessed and rendered online. The v1 could be incredibly simple, but I
> think this is a foundational piece that can grow with the project in the
> future. Since this is a little bit more-involved, I've created a separate
> Google Doc [2] to begin drafting the scope for this:
> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing,
> and have intentionally punted on some of the implementation details until
> the scope is finalized and I gauge interest from users and potential
> collaborators.
>
> What do you think? If folks are on board I will assign the JIRA issue to
> myself and get to work; I'd also welcome collaborators to help get this off
> the ground since I think it will be a boost for the community. Feedback
> is welcome!
>
> Thanks,
> Dave
>
> [1] https://github.com/jenkinsci/
> [2]
> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing
>
>
>

Re: Proposal: shared Mesos framework hosting and registry

[Billy Bones <ga...@gmail.com>]

Wooo mesosphere is making such a good work pushing forward Mesos
technologies!

I deeply think that Mesos should implement registry as a core feature.

Let me explain that, Installing manually frameworks / modules / other third
party adds-on is quite cool when you're beginner and want to learn how the
things works, but once you're in production, you've to be fast and react
quickly.

Let said that I'm a system engineer that have customers requests like
"Hum... Could you had this " insert Framework/module/other name here" on
our cluster region?", I'll have to add it as soon and quick as possible on
the targeted environment, I'll be OK to do it twice, but I wont make my job
a request nightmare and I'll probably search for a way to provide to my
customers a restrictive access (WebUI) to the cluster manager and allow
them to deploy those new add-on by themself.

With this study case in mind, I'll be happy to have a registry which allow
a filtering mechanism to hide or show frameworks/modules/etc by status
(Official Apache Content / Public Content / Private Content).

Here is my thoughts about the registry articulation:

   - Mesos Registry is an integrated part of Mesos master services.
   - Mesos Registry is an endpoint available to WebUI and CLI.
   - Mesos Registry is nothing but a metadata registry.
   - Mesos Registry save its configs and metadata on a key/value store
   (zookeeper?).
   - Mesos Registry is empty at the first launch.
   - Mesos Registry as three views: Official Apache Content | Publicly
   Maintained Content | Private Maintained Content
   - Mesos Registry views content is:
   - Official Apache Content == Link to existing Apache add-ons hosted on
      Github/Git repository + metadata (Like those proposed by Dave and Connor).
      - Publicly Maintained Content == Links to existing repositories
      (Github / Git / other) + metadata (Like those proposed by Dave
and Connor).
      - Private Content == Links to existing private (GIT/GITHUB/Other
      repositories) + metadata (Like those proposed by Dave and Connor), this
      repository is kinda special as it is hosted and created by the cluster
      operators and could be a mixed content of locally maintained repository
      (GIT repos on a HDFS or TAR on HDFS) and public content repository cloned
      from the public content URL/Metadata.

I don't know if I'm really clear, so if I'm not, let me know it, I'll do
some sketches :D


2014-12-02 1:53 GMT+01:00 Connor Doyle <co...@mesosphere.io>:

> Hi Dave,
>
> This is a timely topic, since we have been prototyping and mocking up
> something similar at Mesosphere.  We created a new public GitHub repository
> for it about three weeks ago called "universe" (
> http://github.com/mesosphere/universe).
>
> Although we have added some informal specs, it's very malleable at this
> point.  We're very much interested in making our "universe" compatible
> with, or the same as, the registry you're proposing.  Without delving into
> implementation details, some of the goals we have in mind are outlined
> below.
>
> Data Source:
>
> The package repository should be easily consumable by third-party
> command-line and other programs.  There should be a condensed “index”
> representation of the package repository available.
>
> Packages within the repository should be versioned.
>
> The package repository format itself should be versioned.
>
> Decentralization and Composability:
>
> The package metadata should be hosted in a public place (we like GitHub)
> so that additional packages can be added by the community by simply
> submitting pull requests.  We have added some rudimentary commit hooks and
> automated validation to protect the repo against breaking changes.
>
> It’s important that no single entity “owns the keys” to the universe, and
> that the spec and implementation remain public.  It should be easy and free
> for organizations to maintain a private package repository.
>
> A corollary is that it should be easy for consumers to pull from a
> hierarchy of upstream repositories.  One setup we have in mind is that an
> organization might have staging and production repositories running
> internally.  Packages are pushed to staging where integration testing can
> run before “deployment” to production.  If a package isn’t in the local
> repository it might be looked up and installed from upstream.
>
>
>
> Repositories should be able to be proxied and cached in this way.
> Organizations should be able to isolate their datacenter but also
> selectively add external packages for experimentation. The system should be
> sufficiently portable and extensible to accomodate these and similar use
> cases.
>
> Meta-Framework Descriptors:
>
> Our conception of the package repository is a bit more expansive than just
> Mesos frameworks; it includes descriptions of how to install any piece of
> server software on a Mesos cluster.  Frameworks and non-frameworks alike
> may be installed using some other meta-framework that’s responsible for
> starting all other cluster services.  Likely candidates for this role are
> the long-lived frameworks: Aurora, Marathon, Singularity, and eventually
> Kubernetes.  In any case, the repository spec should not be prescriptive
> with respect to this choice.
>
> The package repository metadata should make it easy for Mesos framework
> authors (and authors of non-Mesos-aware programs) to describe how to
> install their software on a Mesos cluster.  To this end, our prototype
> package spec allows for Meta-framework descriptor files for each package in
> the repository.  For example for a given package we might see a
> `marathon.json` file as well as a `my-app.aurora` file.
>
> An obvious concern is how to specify site-specific arguments upon
> installation.  Here packages should describe data that must be marshalled
> from the environment (e.g. by prompting a user) and combined with the raw
> meta-framework descriptor to launch the app.  These configuration
> parameters should be agnostic of the supported meta-frameworks.  More
> concretely, in our prototype we describe configuration data in terms of a
> JSON-Schema.
>
> CLI Integration:
>
> Part of our proposed package format is an optional descriptor for how to
> fetch and install the command-line tools for interacting with the
> application.  For now, we only have one implementation of this, which is to
> fetch a python egg from PyPI.
>
> Governance:
>
> All in all, we think that making this effort more community driven is a
> healthy way to proceed.  Any input is very welcome.  For example, if others
> think that what we have is a good starting point we could transfer
> ownership of the repository to the mesos organization on GitHub.
>
> Cheers,
> --
> Connor Doyle
> http://mesosphere.com
>
>
>
>
>
> On Nov 30, 2014, at 17:32, Dave Lester <da...@gmail.com> wrote:
>
> As the number of Mesos frameworks grows (and now, a module system), I
> think it's time to create a community-maintained registry with the goal of
> making frameworks and modules easier to discover, contribute to, and
> install.
>
> There's already a JIRA ticket tracking this (MESOS-1759) and I've chatted
> with several folks (thanks in particular Victor Vieux, Tom Arnfeld, Vinod
> Kone, Timothy St Clair, and Joe Stein). I'd like to advance the
> conversation by offering a proposal on the public mailing list.
>
> I imagine two initiatives to achieve this:
>
> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with
> how Jenkins maintains their plugins on GitHub [1], but they allow
> individual plugins to have their own repo within their GH org. Plugins are
> repos with a specific naming convention (in their case, they've appended
> "-plugin" to the repo name but this isn't the same approach we'd need to
> take). Having a shared GH org creates greater visibility to what people are
> doing, and encourages community participation and ownership.
>
> In the case of Mesos, not all frameworks will jump at the chance to have
> community hosting which is fine. But particularly for smaller frameworks, I
> think this is a good option given the success Jenkins has had. We
> could potentially use the existing /mesos github org, or create a new org
> /mesos-extensions or something of the like. It seems like the only
> potential snag here is that we'll want to be explicit that these aren't
> Apache-blessed plugins and instead maintained by the larger ecosystem, but
> I believe we can achieve that by offering a notice in the GH org
> description.
>
> 2) A registry that allows framework writers to publish new versions of
> their frameworks to a central repository that could be programmatically
> accessed and rendered online. The v1 could be incredibly simple, but I
> think this is a foundational piece that can grow with the project in the
> future. Since this is a little bit more-involved, I've created a separate
> Google Doc [2] to begin drafting the scope for this:
> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing,
> and have intentionally punted on some of the implementation details until
> the scope is finalized and I gauge interest from users and potential
> collaborators.
>
> What do you think? If folks are on board I will assign the JIRA issue to
> myself and get to work; I'd also welcome collaborators to help get this off
> the ground since I think it will be a boost for the community. Feedback
> is welcome!
>
> Thanks,
> Dave
>
> [1] https://github.com/jenkinsci/
> [2]
> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing
>
>
>

Re: Proposal: shared Mesos framework hosting and registry

[Connor Doyle <co...@mesosphere.io>]

Hi Dave,

This is a timely topic, since we have been prototyping and mocking up something similar at Mesosphere.  We created a new public GitHub repository for it about three weeks ago called "universe" (http://github.com/mesosphere/universe).

Although we have added some informal specs, it's very malleable at this point.  We're very much interested in making our "universe" compatible with, or the same as, the registry you're proposing.  Without delving into implementation details, some of the goals we have in mind are outlined below.

Data Source:

The package repository should be easily consumable by third-party command-line and other programs.  There should be a condensed “index” representation of the package repository available.

Packages within the repository should be versioned.

The package repository format itself should be versioned.

Decentralization and Composability:

The package metadata should be hosted in a public place (we like GitHub) so that additional packages can be added by the community by simply submitting pull requests.  We have added some rudimentary commit hooks and automated validation to protect the repo against breaking changes.

It’s important that no single entity “owns the keys” to the universe, and that the spec and implementation remain public.  It should be easy and free for organizations to maintain a private package repository.

A corollary is that it should be easy for consumers to pull from a hierarchy of upstream repositories.  One setup we have in mind is that an organization might have staging and production repositories running internally.  Packages are pushed to staging where integration testing can run before “deployment” to production.  If a package isn’t in the local repository it might be looked up and installed from upstream.



Repositories should be able to be proxied and cached in this way.  Organizations should be able to isolate their datacenter but also selectively add external packages for experimentation. The system should be sufficiently portable and extensible to accomodate these and similar use cases.

Meta-Framework Descriptors:

Our conception of the package repository is a bit more expansive than just Mesos frameworks; it includes descriptions of how to install any piece of server software on a Mesos cluster.  Frameworks and non-frameworks alike may be installed using some other meta-framework that’s responsible for starting all other cluster services.  Likely candidates for this role are the long-lived frameworks: Aurora, Marathon, Singularity, and eventually Kubernetes.  In any case, the repository spec should not be prescriptive with respect to this choice.

The package repository metadata should make it easy for Mesos framework authors (and authors of non-Mesos-aware programs) to describe how to install their software on a Mesos cluster.  To this end, our prototype package spec allows for Meta-framework descriptor files for each package in the repository.  For example for a given package we might see a `marathon.json` file as well as a `my-app.aurora` file.

An obvious concern is how to specify site-specific arguments upon installation.  Here packages should describe data that must be marshalled from the environment (e.g. by prompting a user) and combined with the raw meta-framework descriptor to launch the app.  These configuration parameters should be agnostic of the supported meta-frameworks.  More concretely, in our prototype we describe configuration data in terms of a JSON-Schema.

CLI Integration:

Part of our proposed package format is an optional descriptor for how to fetch and install the command-line tools for interacting with the application.  For now, we only have one implementation of this, which is to fetch a python egg from PyPI.

Governance:

All in all, we think that making this effort more community driven is a healthy way to proceed.  Any input is very welcome.  For example, if others think that what we have is a good starting point we could transfer ownership of the repository to the mesos organization on GitHub.

Cheers,
--
Connor Doyle
http://mesosphere.com




> On Nov 30, 2014, at 17:32, Dave Lester <da...@gmail.com> wrote:
> 
> As the number of Mesos frameworks grows (and now, a module system), I think it's time to create a community-maintained registry with the goal of making frameworks and modules easier to discover, contribute to, and install.
> 
> There's already a JIRA ticket tracking this (MESOS-1759) and I've chatted with several folks (thanks in particular Victor Vieux, Tom Arnfeld, Vinod Kone, Timothy St Clair, and Joe Stein). I'd like to advance the conversation by offering a proposal on the public mailing list.
> 
> I imagine two initiatives to achieve this:
> 
> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with how Jenkins maintains their plugins on GitHub [1], but they allow individual plugins to have their own repo within their GH org. Plugins are repos with a specific naming convention (in their case, they've appended "-plugin" to the repo name but this isn't the same approach we'd need to take). Having a shared GH org creates greater visibility to what people are doing, and encourages community participation and ownership.
> 
> In the case of Mesos, not all frameworks will jump at the chance to have community hosting which is fine. But particularly for smaller frameworks, I think this is a good option given the success Jenkins has had. We could potentially use the existing /mesos github org, or create a new org /mesos-extensions or something of the like. It seems like the only potential snag here is that we'll want to be explicit that these aren't Apache-blessed plugins and instead maintained by the larger ecosystem, but I believe we can achieve that by offering a notice in the GH org description.
> 
> 2) A registry that allows framework writers to publish new versions of their frameworks to a central repository that could be programmatically accessed and rendered online. The v1 could be incredibly simple, but I think this is a foundational piece that can grow with the project in the future. Since this is a little bit more-involved, I've created a separate Google Doc [2] to begin drafting the scope for this: https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing, and have intentionally punted on some of the implementation details until the scope is finalized and I gauge interest from users and potential collaborators.
> 
> What do you think? If folks are on board I will assign the JIRA issue to myself and get to work; I'd also welcome collaborators to help get this off the ground since I think it will be a boost for the community. Feedback is welcome!
> 
> Thanks,
> Dave
> 
> [1] https://github.com/jenkinsci/
> [2] https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing