You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by Adam Bordelon <ad...@mesosphere.io> on 2015/02/14 02:06:22 UTC
[DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Hello friends,
The Myriad team and I would like to propose the Myriad project for
inclusion in the Apache Incubator.
Full text of the proposal is below. I can add it to the incubator wiki as
well, if desired.
Please review and discuss. If there are no major concerns, I will call for
a Vote after a week.
Cheers,
-Adam-
me@apache
==========================================================
Apache Myriad Proposal
* Abstract
Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos together
on the same cluster and allows dynamic resource allocations across both
Hadoop and other applications running on the same physical data center
infrastructure.
* Proposal
The vision of Myriad is to provide a comprehensive framework to ensure
Apache Hadoop YARN and Apache Mesos can interoperate with minimal changes
on either side and prevent the static fragmentation of data center
resources.
* Background
Project Myriad is the first resource management framework that allows big
data developers to run YARN-based Hadoop jobs alongside other applications
and services in production. ebay Inc., MapR, and Mesosphere jointly built
Myriad (available on Github at https://github.com/mesos/myriad) with the
vision of freeing big data jobs from siloed clusters and consolidating
infrastructure into a single pool of resources for greater utilization and
operational efficiency. Several companies including Twitter have expressed
interest in Myriad and have begun testing it.
* Rationale
Many Hadoop users are building larger clusters (data lake/data hub
architectures) that support multiple workloads - made possible by the
advent of Apache Hadoop YARN. As the clusters grow in size and importance,
they become an important application within the broader datacenter. At the
same time, Apache Mesos enables efficient resource isolation and sharing
across distributed applications for the broader data center, for instance
MPI, Spark, long running web services, build/test infrastructure,
traditional linux applications/scripts, and others (including arbitrary
docker images).
Myriad aims to enable co-existence of Apache Hadoop YARN and Apache Mesos
on the same physical data center resources, reducing fragmentation of data
center resources.
* Project Goals
** Initial Goals
- Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow policy
based allocation of data center resources across Apache Hadoop and other
distributed applications
- Ensure YARN based execution frameworks work without any changes when
running alongside Myriad. YARN Applications will continue to interact and
run on top of YARN and can choose to be unaware of Myriad.
- Ensure Mesos based execution frameworks work without any changes when
running alongside Myriad. Mesos applications will continue to interact and
run on Mesos and can choose to be unaware of Myriad.
- Provide isolation for multi-tenancy.
- Use linux cgroups (and optionally Docker-like technologies to ease
packaging, deployment and broader isolation) so that multiple YARN clusters
can run in their own space and are isolated from each other. YARN’s RM and
NMs are dockerized.
- Myriad should be able to manage full YARN lifecycle:
- Bring up YARN (RM, NM)
- Scale Up/Down YARN
- Release resources and shut down YARN
** Longer Term Goals
- Allow fine-grained dynamic allocation of resources to Hadoop including
the ability to scale up and scale down the cluster.
- Provide different policies to allow downsizing running applications on
Hadoop when resources are taken away from it.
- Provide a framework so the downsizing policy is pluggable and users can
write their own implementations.
- Allow multiple versions of Apache Hadoop to run on the same physical
infrastructure
- Allow workload portability - ability to migrate YARN workloads across
various cloud infrastructures seamlessly (e.g. GCE, AWS, etc)
- Security:
- Authentication Requirements:
- Support basic CRAM-MD5 password authentication between Myriad and
Mesos. Additional authentication mechanisms may be supported in the future.
- Traditional user authentication with Hadoop’s HTTP web-consoles
should work as usual.
- Authorization:
- Only authorized users are allowed to launch YARN clusters. Mesos
allows to specify which framework principal is allowed to register as a
particular role.
- Encryption on wire:
- All control traffic to/from Myriad/Mesos
- Logs
- Audits (where to store them)
- Log all major activities/events with audit trail - who, what, when,
result
- Launching YARN/RM
- Launching NM’s
- Downsizing NM’s
- Terminating YARN/RM
- What to do with old logs?
- Debuggability/Visibility
- Hooks to identify different YARN cluster lifecycles (yarn-id?)
- GUI: Capability to scale-up and scale-down by selecting nodes and
providing a scale-up/scale-down factor.
* Architectural Overview
The following diagram illustrates the high level architecture. YARN (with
Myriad) is registered as a framework with Mesos master along with possibly
other Mesos frameworks. This enables YARN to share cluster resources with
other Mesos frameworks providing elasticity of resources between Hadoop
workloads and Mesos frameworks.
See
https://github.com/mesos/myriad/blob/phase1/docs/images/high-level-architecture.png
* Current Status
Myriad is under active development. Key components of Myriad are:
** Myriad Resource Manager (RM) Plugin
- Plugs into Resource Manager Java process via yarn-site.xml configuration.
- Registers Myriad as a framework with Mesos. Receives resource offers from
Mesos.
- Monitors YARN’s application pipeline and scheduling events to drive
scale-up or scale-down decisions for Hadoop.
- Exposes REST APIs to help admins control Hadoop/YARN’s resource
consumption. Currently the following APIs are supported:
- Scale Up (e.g. “launch 4 Node Manager instances with 10G/6CPU capacity”)
- Scale Down (e.g. “kill 2 Node Manager instances with 10G/6CPU
capacity”)
** Myriad Mesos Executor
- Launched on a Mesos slave node by Myriad RM plugin via Mesos.
- Responsible for launching Node Manager process with appropriate
capacities configured in yarn-site.xml.
- Mounts YARN’s cgroup hierarchy under Mesos’ cgroup hierarchy in case
YARN’s cgroups are enabled.
Currently, a working prototype/demo had been built for the goals listed
under the “Initial Goals” section. Open issues and enhancements are tracked
at https://github.com/mesos/myriad/issues. Myriad is not yet tested for
production use.
** Meritocracy
We plan to invest in supporting a meritocracy. We will discuss the
requirements in a public forum. Several companies have already expressed
interest in this project, and we intend to invite developers to contribute
and gain karma. We will encourage and monitor community participation so
that privileges can be extended to those that contribute.
** Community
We are happy to report that there are existing Apache committers and
corporate users who are closely involved in the project already. We hope to
extend the user and developer base further in the future and build a solid
open source community around Myriad, growing the community and adding
committers following the Apache Way.
** Core Developers
The initial technology was built independently by ebay and MapR. ebay built
the technology in consultation with Ben Hindman. MapR built a working
prototype in tight consultation and mentorship with Mesosphere.
** Alignment
The initial committers strongly believe that Apache Hadoop YARN and Apache
Mesos will gain broad adoption and therefore a framework to allow for a
co-existence of these frameworks that is transparent to applications
written for YARN and Mesos will serve the needs of the broader community.
* Known Risks
** Inexperience with Open Source
Initial Myriad committers have varying levels of experience using and
contributing to Open Source projects, however by working with our mentors
and the Apache community we believe we will be able to conduct ourselves in
accordance with Apache Incubator guidelines. The close relationship between
the Myriad team and Apache Mesos and Apache Hadoop means there is an
awareness of the incubation process and a willingness to embrace The Apache
Way.
** Homogenous Developers
There is already diversity in the core developer community as they are
employed by three different and independent companies viz. ebay inc., MapR,
and Mesosphere. However, there will continue to be an emphasis on
increasing the diversity of the developer community.
** Reliance on Salaried Developers
Currently, the core developers are paid to work on Myriad. However, once
the project has a community built around it, we expect to get committers,
contributors and community from outside the current participating
organizations.
** Relationships with Other Apache Products
Myriad implements interfaces from both Apache YARN and Apache Mesos, and
requires both to be present so that Myriad can coordinate dynamic resource
sharing between the two.
** An Excessive Fascination with the Apache Brand
While we respect the reputation of the Apache brand and have no doubts that
it will attract contributors and users, our interest is primarily to give
Myriad a solid home as an open source project following an established
development model. We have also given reasons in the Rationale and
Alignment sections.
* Documentation
Documentation is included in a docs directory of the repository (See
https://github.com/mesos/myriad/tree/phase1/docs), and currently details
how Myriad works, developing the project, auto-scaling a YARN cluster, the
Myriad REST API, and more. We will improve docs at every revision drop.
* Initial Source
The Myriad codebase has been posted on GitHub for review and licensed under
an Apache v2 license.
https://github.com/mesos/myriad
* Source and IP Submission Plan
During incubation, the codebase will be available at
https://github.com/apache/incubator-myriad/ and contributors will commit
appropriate contribute license agreements.
* External Dependencies
All Myriad dependencies have Apache compatible licenses.
* Cryptography
Myriad doesn’t use cryptography itself. Hadoop and Mesos projects, however,
use standard API’s and tools for SSH And SSL communication where necessary.
* Required Resources
** Mailing Lists
- myriad-private for private PMC conversations
- myriad-dev
- myriad-commits
- myriad-user
** Version Control
We prefer to use Git as our source control system: git://
git.apache.org/myriad
** Issue Tracking
JIRA Myriad (MYRIAD)
* Initial Committers
- Santosh Marella (smarella at mapr dot com)
- Mohit Soni (mohitsoni1989 at gmail dot com)
- Adam Bordelon (me at apache dot org) *
- Meghdoot Bhattacharya ( mbhattacharya at paypal dot com)
- Anoop Dawar (anoopdawar at gmail dot com)
- Jim Scott (jim at 13ways dot com)
- Ken Sipe (kensipe at gmail dot com)
* Affiliations
- Santosh Marella, MapR
- Mohit Soni, ebay Inc.
- Adam Bordelon, Mesosphere
- Meghdoot Bhattacharya, ebay Inc.
- Anoop Dawar, MapR
- Jim Scott, MapR
- Ken Sipe, Mesosphere
* Sponsors
** Champion (Proposal)
- Ben Hindman (benh at apache dot org)
** Nominated Mentors
- Ben Hindman (benh at apache dot org) - Mesosphere
- Danese Cooper (danese at apache dot org) - ebay, Inc.
- Ted Dunning (tdunning at apache dot org) - MapR
** Sponsoring Entity
Apache Incubator
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Henry Saputra <he...@gmail.com>.
Oh it is painless =)
>From what I have seen, having just dev@ list early would help ramping
up dev quickly.
@Adam and @Ted, IMHO once the transition is over and the project has
one release under ASF adding user@ list would be beneficial.
- Henry
On Tue, Feb 17, 2015 at 9:59 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
> Good point. I'm fine with starting with just a dev@ first, and then we can
> add user@ if/when dev becomes too noisy.
> I assume adding a new mailing list is relatively painless.
>
> On Tue, Feb 17, 2015 at 9:52 PM, Ted Dunning <te...@gmail.com> wrote:
>
>> On Tue, Feb 17, 2015 at 9:38 PM, Henry Saputra <he...@gmail.com>
>> wrote:
>>
>> > @Adam and @Ted, like any new incubator projects coming we always check
>> > if you need user@ so early in the process?
>> > Would probably better to have all discussion in dev@ early in
>> incubation.
>> >
>>
>> Henry,
>>
>> This is a good question to ask (and I have asked it in the past).
>>
>> I think that Myriad is in, or nearly in production here and there already.
>> That means that a user@ list might well be useful.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Adam Bordelon <ad...@mesosphere.io>.
Good point. I'm fine with starting with just a dev@ first, and then we can
add user@ if/when dev becomes too noisy.
I assume adding a new mailing list is relatively painless.
On Tue, Feb 17, 2015 at 9:52 PM, Ted Dunning <te...@gmail.com> wrote:
> On Tue, Feb 17, 2015 at 9:38 PM, Henry Saputra <he...@gmail.com>
> wrote:
>
> > @Adam and @Ted, like any new incubator projects coming we always check
> > if you need user@ so early in the process?
> > Would probably better to have all discussion in dev@ early in
> incubation.
> >
>
> Henry,
>
> This is a good question to ask (and I have asked it in the past).
>
> I think that Myriad is in, or nearly in production here and there already.
> That means that a user@ list might well be useful.
>
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Ted Dunning <te...@gmail.com>.
On Tue, Feb 17, 2015 at 9:38 PM, Henry Saputra <he...@gmail.com>
wrote:
> @Adam and @Ted, like any new incubator projects coming we always check
> if you need user@ so early in the process?
> Would probably better to have all discussion in dev@ early in incubation.
>
Henry,
This is a good question to ask (and I have asked it in the past).
I think that Myriad is in, or nearly in production here and there already.
That means that a user@ list might well be useful.
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Ted Dunning <te...@gmail.com>.
Sound right to me.
Ben? Would you like to do the honors?
Sent from my iPhone
> On Feb 21, 2015, at 15:12, Luciano Resende <lu...@gmail.com> wrote:
>
> Discussion has died down, and we had only positive feedback for the
> proposal. Should we start a formal vote ?
>
>> On Wed, Feb 18, 2015 at 9:27 PM, Ted Dunning <te...@gmail.com> wrote:
>>
>> On Wed, Feb 18, 2015 at 12:24 PM, Adam Bordelon <ad...@mesosphere.io>
>> wrote:
>>
>>> I am personally in favor of adding Luciano Resende to the Nominated
>> Mentors
>>> list (the more the merrier, right?), but I want to get approval from the
>>> other mentors/committers before nominating him in the proposal.
>>>
>>
>> +1
>>
>> I don't think that you really need to worry about other mentors approving
>> the addition of a mentor. This is a duty well shared by more hands. I
>> haven't seen a bad mentor except ones that go missing and having an extra
>> helps with that.
>>
>
>
>
> --
> Luciano Resende
> http://people.apache.org/~lresende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Adam Bordelon <ad...@mesosphere.io>.
I have updated the proposal to include Luciano as a Mentor, and marked it
as FINAL.
https://wiki.apache.org/incubator/MyriadProposal?action=recall&rev=7
I will open up a new thread for the VOTE.
On Sat, Feb 21, 2015 at 1:20 PM, jan i <ja...@apache.org> wrote:
> On 21 February 2015 at 21:12, Luciano Resende <lu...@gmail.com>
> wrote:
>
> > Discussion has died down, and we had only positive feedback for the
> > proposal. Should we start a formal vote ?
> >
> please do.
>
> rgds
> jan i
>
> >
> > On Wed, Feb 18, 2015 at 9:27 PM, Ted Dunning <te...@gmail.com>
> > wrote:
> >
> > > On Wed, Feb 18, 2015 at 12:24 PM, Adam Bordelon <ad...@mesosphere.io>
> > > wrote:
> > >
> > > > I am personally in favor of adding Luciano Resende to the Nominated
> > > Mentors
> > > > list (the more the merrier, right?), but I want to get approval from
> > the
> > > > other mentors/committers before nominating him in the proposal.
> > > >
> > >
> > > +1
> > >
> > > I don't think that you really need to worry about other mentors
> approving
> > > the addition of a mentor. This is a duty well shared by more hands. I
> > > haven't seen a bad mentor except ones that go missing and having an
> extra
> > > helps with that.
> > >
> >
> >
> >
> > --
> > Luciano Resende
> > http://people.apache.org/~lresende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by jan i <ja...@apache.org>.
On 21 February 2015 at 21:12, Luciano Resende <lu...@gmail.com> wrote:
> Discussion has died down, and we had only positive feedback for the
> proposal. Should we start a formal vote ?
>
please do.
rgds
jan i
>
> On Wed, Feb 18, 2015 at 9:27 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > On Wed, Feb 18, 2015 at 12:24 PM, Adam Bordelon <ad...@mesosphere.io>
> > wrote:
> >
> > > I am personally in favor of adding Luciano Resende to the Nominated
> > Mentors
> > > list (the more the merrier, right?), but I want to get approval from
> the
> > > other mentors/committers before nominating him in the proposal.
> > >
> >
> > +1
> >
> > I don't think that you really need to worry about other mentors approving
> > the addition of a mentor. This is a duty well shared by more hands. I
> > haven't seen a bad mentor except ones that go missing and having an extra
> > helps with that.
> >
>
>
>
> --
> Luciano Resende
> http://people.apache.org/~lresende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Luciano Resende <lu...@gmail.com>.
Discussion has died down, and we had only positive feedback for the
proposal. Should we start a formal vote ?
On Wed, Feb 18, 2015 at 9:27 PM, Ted Dunning <te...@gmail.com> wrote:
> On Wed, Feb 18, 2015 at 12:24 PM, Adam Bordelon <ad...@mesosphere.io>
> wrote:
>
> > I am personally in favor of adding Luciano Resende to the Nominated
> Mentors
> > list (the more the merrier, right?), but I want to get approval from the
> > other mentors/committers before nominating him in the proposal.
> >
>
> +1
>
> I don't think that you really need to worry about other mentors approving
> the addition of a mentor. This is a duty well shared by more hands. I
> haven't seen a bad mentor except ones that go missing and having an extra
> helps with that.
>
--
Luciano Resende
http://people.apache.org/~lresende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Ted Dunning <te...@gmail.com>.
On Wed, Feb 18, 2015 at 12:24 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
> I am personally in favor of adding Luciano Resende to the Nominated Mentors
> list (the more the merrier, right?), but I want to get approval from the
> other mentors/committers before nominating him in the proposal.
>
+1
I don't think that you really need to worry about other mentors approving
the addition of a mentor. This is a duty well shared by more hands. I
haven't seen a bad mentor except ones that go missing and having an extra
helps with that.
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Adam Bordelon <ad...@mesosphere.io>.
Thanks for your support everyone. I have updated the wiki proposal to
remove the user@ mailing list, and fixed up the formatting.
I am personally in favor of adding Luciano Resende to the Nominated Mentors
list (the more the merrier, right?), but I want to get approval from the
other mentors/committers before nominating him in the proposal. See
http://incubator.apache.org/guides/proposal.html#template-mentors and
http://incubator.apache.org/guides/mentor.html for more details on the
role/responsibilities.
Alternatively, Luciano could act as a (small 'm') mentor, rather than an
official Podling Mentor. Thoughts, opinions?
Any high-level critiques or questions not answered in the proposal? Any
nit-picky grammer/spelling mistakes? [troll]
On Tue, Feb 17, 2015 at 10:19 PM, Naresh Agarwal <na...@inmobi.com>
wrote:
> Looks interesting. Looking forward to this.
>
> Thanks
> Naresh
>
> On Wed, Feb 18, 2015 at 11:08 AM, Henry Saputra <he...@gmail.com>
> wrote:
>
> > I love this project and the idea. Tried to hack it couple years ago
> > could not make it work.
> >
> > Looking forward seeing it in ASF incubator for sure.
> >
> > @Adam and @Ted, like any new incubator projects coming we always check
> > if you need user@ so early in the process?
> > Would probably better to have all discussion in dev@ early in
> incubation.
> >
> > - Henry
> >
> > On Fri, Feb 13, 2015 at 5:06 PM, Adam Bordelon <ad...@mesosphere.io>
> wrote:
> > > Hello friends,
> > >
> > > The Myriad team and I would like to propose the Myriad project for
> > > inclusion in the Apache Incubator.
> > > Full text of the proposal is below. I can add it to the incubator wiki
> as
> > > well, if desired.
> > > Please review and discuss. If there are no major concerns, I will call
> > for
> > > a Vote after a week.
> > >
> > > Cheers,
> > > -Adam-
> > > me@apache
> > >
> > > ==========================================================
> > > Apache Myriad Proposal
> > >
> > > * Abstract
> > > Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos
> > together
> > > on the same cluster and allows dynamic resource allocations across both
> > > Hadoop and other applications running on the same physical data center
> > > infrastructure.
> > >
> > > * Proposal
> > > The vision of Myriad is to provide a comprehensive framework to ensure
> > > Apache Hadoop YARN and Apache Mesos can interoperate with minimal
> changes
> > > on either side and prevent the static fragmentation of data center
> > > resources.
> > >
> > > * Background
> > > Project Myriad is the first resource management framework that allows
> big
> > > data developers to run YARN-based Hadoop jobs alongside other
> > applications
> > > and services in production. ebay Inc., MapR, and Mesosphere jointly
> built
> > > Myriad (available on Github at https://github.com/mesos/myriad) with
> the
> > > vision of freeing big data jobs from siloed clusters and consolidating
> > > infrastructure into a single pool of resources for greater utilization
> > and
> > > operational efficiency. Several companies including Twitter have
> > expressed
> > > interest in Myriad and have begun testing it.
> > >
> > > * Rationale
> > > Many Hadoop users are building larger clusters (data lake/data hub
> > > architectures) that support multiple workloads - made possible by the
> > > advent of Apache Hadoop YARN. As the clusters grow in size and
> > importance,
> > > they become an important application within the broader datacenter. At
> > the
> > > same time, Apache Mesos enables efficient resource isolation and
> sharing
> > > across distributed applications for the broader data center, for
> instance
> > > MPI, Spark, long running web services, build/test infrastructure,
> > > traditional linux applications/scripts, and others (including arbitrary
> > > docker images).
> > >
> > > Myriad aims to enable co-existence of Apache Hadoop YARN and Apache
> Mesos
> > > on the same physical data center resources, reducing fragmentation of
> > data
> > > center resources.
> > >
> > > * Project Goals
> > > ** Initial Goals
> > > - Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow
> > policy
> > > based allocation of data center resources across Apache Hadoop and
> other
> > > distributed applications
> > > - Ensure YARN based execution frameworks work without any changes when
> > > running alongside Myriad. YARN Applications will continue to interact
> and
> > > run on top of YARN and can choose to be unaware of Myriad.
> > > - Ensure Mesos based execution frameworks work without any changes when
> > > running alongside Myriad. Mesos applications will continue to interact
> > and
> > > run on Mesos and can choose to be unaware of Myriad.
> > > - Provide isolation for multi-tenancy.
> > > - Use linux cgroups (and optionally Docker-like technologies to ease
> > > packaging, deployment and broader isolation) so that multiple YARN
> > clusters
> > > can run in their own space and are isolated from each other. YARN’s RM
> > and
> > > NMs are dockerized.
> > > - Myriad should be able to manage full YARN lifecycle:
> > > - Bring up YARN (RM, NM)
> > > - Scale Up/Down YARN
> > > - Release resources and shut down YARN
> > >
> > > ** Longer Term Goals
> > > - Allow fine-grained dynamic allocation of resources to Hadoop
> including
> > > the ability to scale up and scale down the cluster.
> > > - Provide different policies to allow downsizing running applications
> > on
> > > Hadoop when resources are taken away from it.
> > > - Provide a framework so the downsizing policy is pluggable and users
> > can
> > > write their own implementations.
> > > - Allow multiple versions of Apache Hadoop to run on the same physical
> > > infrastructure
> > > - Allow workload portability - ability to migrate YARN workloads across
> > > various cloud infrastructures seamlessly (e.g. GCE, AWS, etc)
> > > - Security:
> > > - Authentication Requirements:
> > > - Support basic CRAM-MD5 password authentication between Myriad and
> > > Mesos. Additional authentication mechanisms may be supported in the
> > future.
> > > - Traditional user authentication with Hadoop’s HTTP web-consoles
> > > should work as usual.
> > > - Authorization:
> > > - Only authorized users are allowed to launch YARN clusters. Mesos
> > > allows to specify which framework principal is allowed to register as a
> > > particular role.
> > > - Encryption on wire:
> > > - All control traffic to/from Myriad/Mesos
> > > - Logs
> > > - Audits (where to store them)
> > > - Log all major activities/events with audit trail - who, what,
> when,
> > > result
> > > - Launching YARN/RM
> > > - Launching NM’s
> > > - Downsizing NM’s
> > > - Terminating YARN/RM
> > > - What to do with old logs?
> > > - Debuggability/Visibility
> > > - Hooks to identify different YARN cluster lifecycles (yarn-id?)
> > > - GUI: Capability to scale-up and scale-down by selecting nodes and
> > > providing a scale-up/scale-down factor.
> > >
> > > * Architectural Overview
> > > The following diagram illustrates the high level architecture. YARN
> (with
> > > Myriad) is registered as a framework with Mesos master along with
> > possibly
> > > other Mesos frameworks. This enables YARN to share cluster resources
> with
> > > other Mesos frameworks providing elasticity of resources between Hadoop
> > > workloads and Mesos frameworks.
> > >
> > > See
> > >
> >
> https://github.com/mesos/myriad/blob/phase1/docs/images/high-level-architecture.png
> > >
> > > * Current Status
> > > Myriad is under active development. Key components of Myriad are:
> > > ** Myriad Resource Manager (RM) Plugin
> > > - Plugs into Resource Manager Java process via yarn-site.xml
> > configuration.
> > > - Registers Myriad as a framework with Mesos. Receives resource offers
> > from
> > > Mesos.
> > > - Monitors YARN’s application pipeline and scheduling events to drive
> > > scale-up or scale-down decisions for Hadoop.
> > > - Exposes REST APIs to help admins control Hadoop/YARN’s resource
> > > consumption. Currently the following APIs are supported:
> > > - Scale Up (e.g. “launch 4 Node Manager instances with 10G/6CPU
> > capacity”)
> > > - Scale Down (e.g. “kill 2 Node Manager instances with 10G/6CPU
> > > capacity”)
> > >
> > > ** Myriad Mesos Executor
> > > - Launched on a Mesos slave node by Myriad RM plugin via Mesos.
> > > - Responsible for launching Node Manager process with appropriate
> > > capacities configured in yarn-site.xml.
> > > - Mounts YARN’s cgroup hierarchy under Mesos’ cgroup hierarchy in case
> > > YARN’s cgroups are enabled.
> > >
> > > Currently, a working prototype/demo had been built for the goals listed
> > > under the “Initial Goals” section. Open issues and enhancements are
> > tracked
> > > at https://github.com/mesos/myriad/issues. Myriad is not yet tested
> for
> > > production use.
> > >
> > > ** Meritocracy
> > > We plan to invest in supporting a meritocracy. We will discuss the
> > > requirements in a public forum. Several companies have already
> expressed
> > > interest in this project, and we intend to invite developers to
> > contribute
> > > and gain karma. We will encourage and monitor community participation
> so
> > > that privileges can be extended to those that contribute.
> > >
> > > ** Community
> > > We are happy to report that there are existing Apache committers and
> > > corporate users who are closely involved in the project already. We
> hope
> > to
> > > extend the user and developer base further in the future and build a
> > solid
> > > open source community around Myriad, growing the community and adding
> > > committers following the Apache Way.
> > >
> > > ** Core Developers
> > > The initial technology was built independently by ebay and MapR. ebay
> > built
> > > the technology in consultation with Ben Hindman. MapR built a working
> > > prototype in tight consultation and mentorship with Mesosphere.
> > >
> > > ** Alignment
> > > The initial committers strongly believe that Apache Hadoop YARN and
> > Apache
> > > Mesos will gain broad adoption and therefore a framework to allow for a
> > > co-existence of these frameworks that is transparent to applications
> > > written for YARN and Mesos will serve the needs of the broader
> community.
> > >
> > > * Known Risks
> > >
> > > ** Inexperience with Open Source
> > > Initial Myriad committers have varying levels of experience using and
> > > contributing to Open Source projects, however by working with our
> mentors
> > > and the Apache community we believe we will be able to conduct
> ourselves
> > in
> > > accordance with Apache Incubator guidelines. The close relationship
> > between
> > > the Myriad team and Apache Mesos and Apache Hadoop means there is an
> > > awareness of the incubation process and a willingness to embrace The
> > Apache
> > > Way.
> > >
> > > ** Homogenous Developers
> > > There is already diversity in the core developer community as they are
> > > employed by three different and independent companies viz. ebay inc.,
> > MapR,
> > > and Mesosphere. However, there will continue to be an emphasis on
> > > increasing the diversity of the developer community.
> > >
> > > ** Reliance on Salaried Developers
> > > Currently, the core developers are paid to work on Myriad. However,
> once
> > > the project has a community built around it, we expect to get
> committers,
> > > contributors and community from outside the current participating
> > > organizations.
> > >
> > > ** Relationships with Other Apache Products
> > > Myriad implements interfaces from both Apache YARN and Apache Mesos,
> and
> > > requires both to be present so that Myriad can coordinate dynamic
> > resource
> > > sharing between the two.
> > >
> > > ** An Excessive Fascination with the Apache Brand
> > > While we respect the reputation of the Apache brand and have no doubts
> > that
> > > it will attract contributors and users, our interest is primarily to
> give
> > > Myriad a solid home as an open source project following an established
> > > development model. We have also given reasons in the Rationale and
> > > Alignment sections.
> > >
> > > * Documentation
> > > Documentation is included in a docs directory of the repository (See
> > > https://github.com/mesos/myriad/tree/phase1/docs), and currently
> details
> > > how Myriad works, developing the project, auto-scaling a YARN cluster,
> > the
> > > Myriad REST API, and more. We will improve docs at every revision drop.
> > >
> > > * Initial Source
> > > The Myriad codebase has been posted on GitHub for review and licensed
> > under
> > > an Apache v2 license.
> > > https://github.com/mesos/myriad
> > >
> > > * Source and IP Submission Plan
> > > During incubation, the codebase will be available at
> > > https://github.com/apache/incubator-myriad/ and contributors will
> commit
> > > appropriate contribute license agreements.
> > >
> > > * External Dependencies
> > > All Myriad dependencies have Apache compatible licenses.
> > >
> > > * Cryptography
> > > Myriad doesn’t use cryptography itself. Hadoop and Mesos projects,
> > however,
> > > use standard API’s and tools for SSH And SSL communication where
> > necessary.
> > >
> > > * Required Resources
> > > ** Mailing Lists
> > > - myriad-private for private PMC conversations
> > > - myriad-dev
> > > - myriad-commits
> > > - myriad-user
> > >
> > > ** Version Control
> > > We prefer to use Git as our source control system: git://
> > > git.apache.org/myriad
> > >
> > > ** Issue Tracking
> > > JIRA Myriad (MYRIAD)
> > >
> > > * Initial Committers
> > > - Santosh Marella (smarella at mapr dot com)
> > > - Mohit Soni (mohitsoni1989 at gmail dot com)
> > > - Adam Bordelon (me at apache dot org) *
> > > - Meghdoot Bhattacharya ( mbhattacharya at paypal dot com)
> > > - Anoop Dawar (anoopdawar at gmail dot com)
> > > - Jim Scott (jim at 13ways dot com)
> > > - Ken Sipe (kensipe at gmail dot com)
> > >
> > > * Affiliations
> > > - Santosh Marella, MapR
> > > - Mohit Soni, ebay Inc.
> > > - Adam Bordelon, Mesosphere
> > > - Meghdoot Bhattacharya, ebay Inc.
> > > - Anoop Dawar, MapR
> > > - Jim Scott, MapR
> > > - Ken Sipe, Mesosphere
> > >
> > > * Sponsors
> > > ** Champion (Proposal)
> > > - Ben Hindman (benh at apache dot org)
> > >
> > > ** Nominated Mentors
> > > - Ben Hindman (benh at apache dot org) - Mesosphere
> > > - Danese Cooper (danese at apache dot org) - ebay, Inc.
> > > - Ted Dunning (tdunning at apache dot org) - MapR
> > >
> > > ** Sponsoring Entity
> > > Apache Incubator
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Naresh Agarwal <na...@inmobi.com>.
Looks interesting. Looking forward to this.
Thanks
Naresh
On Wed, Feb 18, 2015 at 11:08 AM, Henry Saputra <he...@gmail.com>
wrote:
> I love this project and the idea. Tried to hack it couple years ago
> could not make it work.
>
> Looking forward seeing it in ASF incubator for sure.
>
> @Adam and @Ted, like any new incubator projects coming we always check
> if you need user@ so early in the process?
> Would probably better to have all discussion in dev@ early in incubation.
>
> - Henry
>
> On Fri, Feb 13, 2015 at 5:06 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
> > Hello friends,
> >
> > The Myriad team and I would like to propose the Myriad project for
> > inclusion in the Apache Incubator.
> > Full text of the proposal is below. I can add it to the incubator wiki as
> > well, if desired.
> > Please review and discuss. If there are no major concerns, I will call
> for
> > a Vote after a week.
> >
> > Cheers,
> > -Adam-
> > me@apache
> >
> > ==========================================================
> > Apache Myriad Proposal
> >
> > * Abstract
> > Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos
> together
> > on the same cluster and allows dynamic resource allocations across both
> > Hadoop and other applications running on the same physical data center
> > infrastructure.
> >
> > * Proposal
> > The vision of Myriad is to provide a comprehensive framework to ensure
> > Apache Hadoop YARN and Apache Mesos can interoperate with minimal changes
> > on either side and prevent the static fragmentation of data center
> > resources.
> >
> > * Background
> > Project Myriad is the first resource management framework that allows big
> > data developers to run YARN-based Hadoop jobs alongside other
> applications
> > and services in production. ebay Inc., MapR, and Mesosphere jointly built
> > Myriad (available on Github at https://github.com/mesos/myriad) with the
> > vision of freeing big data jobs from siloed clusters and consolidating
> > infrastructure into a single pool of resources for greater utilization
> and
> > operational efficiency. Several companies including Twitter have
> expressed
> > interest in Myriad and have begun testing it.
> >
> > * Rationale
> > Many Hadoop users are building larger clusters (data lake/data hub
> > architectures) that support multiple workloads - made possible by the
> > advent of Apache Hadoop YARN. As the clusters grow in size and
> importance,
> > they become an important application within the broader datacenter. At
> the
> > same time, Apache Mesos enables efficient resource isolation and sharing
> > across distributed applications for the broader data center, for instance
> > MPI, Spark, long running web services, build/test infrastructure,
> > traditional linux applications/scripts, and others (including arbitrary
> > docker images).
> >
> > Myriad aims to enable co-existence of Apache Hadoop YARN and Apache Mesos
> > on the same physical data center resources, reducing fragmentation of
> data
> > center resources.
> >
> > * Project Goals
> > ** Initial Goals
> > - Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow
> policy
> > based allocation of data center resources across Apache Hadoop and other
> > distributed applications
> > - Ensure YARN based execution frameworks work without any changes when
> > running alongside Myriad. YARN Applications will continue to interact and
> > run on top of YARN and can choose to be unaware of Myriad.
> > - Ensure Mesos based execution frameworks work without any changes when
> > running alongside Myriad. Mesos applications will continue to interact
> and
> > run on Mesos and can choose to be unaware of Myriad.
> > - Provide isolation for multi-tenancy.
> > - Use linux cgroups (and optionally Docker-like technologies to ease
> > packaging, deployment and broader isolation) so that multiple YARN
> clusters
> > can run in their own space and are isolated from each other. YARN’s RM
> and
> > NMs are dockerized.
> > - Myriad should be able to manage full YARN lifecycle:
> > - Bring up YARN (RM, NM)
> > - Scale Up/Down YARN
> > - Release resources and shut down YARN
> >
> > ** Longer Term Goals
> > - Allow fine-grained dynamic allocation of resources to Hadoop including
> > the ability to scale up and scale down the cluster.
> > - Provide different policies to allow downsizing running applications
> on
> > Hadoop when resources are taken away from it.
> > - Provide a framework so the downsizing policy is pluggable and users
> can
> > write their own implementations.
> > - Allow multiple versions of Apache Hadoop to run on the same physical
> > infrastructure
> > - Allow workload portability - ability to migrate YARN workloads across
> > various cloud infrastructures seamlessly (e.g. GCE, AWS, etc)
> > - Security:
> > - Authentication Requirements:
> > - Support basic CRAM-MD5 password authentication between Myriad and
> > Mesos. Additional authentication mechanisms may be supported in the
> future.
> > - Traditional user authentication with Hadoop’s HTTP web-consoles
> > should work as usual.
> > - Authorization:
> > - Only authorized users are allowed to launch YARN clusters. Mesos
> > allows to specify which framework principal is allowed to register as a
> > particular role.
> > - Encryption on wire:
> > - All control traffic to/from Myriad/Mesos
> > - Logs
> > - Audits (where to store them)
> > - Log all major activities/events with audit trail - who, what, when,
> > result
> > - Launching YARN/RM
> > - Launching NM’s
> > - Downsizing NM’s
> > - Terminating YARN/RM
> > - What to do with old logs?
> > - Debuggability/Visibility
> > - Hooks to identify different YARN cluster lifecycles (yarn-id?)
> > - GUI: Capability to scale-up and scale-down by selecting nodes and
> > providing a scale-up/scale-down factor.
> >
> > * Architectural Overview
> > The following diagram illustrates the high level architecture. YARN (with
> > Myriad) is registered as a framework with Mesos master along with
> possibly
> > other Mesos frameworks. This enables YARN to share cluster resources with
> > other Mesos frameworks providing elasticity of resources between Hadoop
> > workloads and Mesos frameworks.
> >
> > See
> >
> https://github.com/mesos/myriad/blob/phase1/docs/images/high-level-architecture.png
> >
> > * Current Status
> > Myriad is under active development. Key components of Myriad are:
> > ** Myriad Resource Manager (RM) Plugin
> > - Plugs into Resource Manager Java process via yarn-site.xml
> configuration.
> > - Registers Myriad as a framework with Mesos. Receives resource offers
> from
> > Mesos.
> > - Monitors YARN’s application pipeline and scheduling events to drive
> > scale-up or scale-down decisions for Hadoop.
> > - Exposes REST APIs to help admins control Hadoop/YARN’s resource
> > consumption. Currently the following APIs are supported:
> > - Scale Up (e.g. “launch 4 Node Manager instances with 10G/6CPU
> capacity”)
> > - Scale Down (e.g. “kill 2 Node Manager instances with 10G/6CPU
> > capacity”)
> >
> > ** Myriad Mesos Executor
> > - Launched on a Mesos slave node by Myriad RM plugin via Mesos.
> > - Responsible for launching Node Manager process with appropriate
> > capacities configured in yarn-site.xml.
> > - Mounts YARN’s cgroup hierarchy under Mesos’ cgroup hierarchy in case
> > YARN’s cgroups are enabled.
> >
> > Currently, a working prototype/demo had been built for the goals listed
> > under the “Initial Goals” section. Open issues and enhancements are
> tracked
> > at https://github.com/mesos/myriad/issues. Myriad is not yet tested for
> > production use.
> >
> > ** Meritocracy
> > We plan to invest in supporting a meritocracy. We will discuss the
> > requirements in a public forum. Several companies have already expressed
> > interest in this project, and we intend to invite developers to
> contribute
> > and gain karma. We will encourage and monitor community participation so
> > that privileges can be extended to those that contribute.
> >
> > ** Community
> > We are happy to report that there are existing Apache committers and
> > corporate users who are closely involved in the project already. We hope
> to
> > extend the user and developer base further in the future and build a
> solid
> > open source community around Myriad, growing the community and adding
> > committers following the Apache Way.
> >
> > ** Core Developers
> > The initial technology was built independently by ebay and MapR. ebay
> built
> > the technology in consultation with Ben Hindman. MapR built a working
> > prototype in tight consultation and mentorship with Mesosphere.
> >
> > ** Alignment
> > The initial committers strongly believe that Apache Hadoop YARN and
> Apache
> > Mesos will gain broad adoption and therefore a framework to allow for a
> > co-existence of these frameworks that is transparent to applications
> > written for YARN and Mesos will serve the needs of the broader community.
> >
> > * Known Risks
> >
> > ** Inexperience with Open Source
> > Initial Myriad committers have varying levels of experience using and
> > contributing to Open Source projects, however by working with our mentors
> > and the Apache community we believe we will be able to conduct ourselves
> in
> > accordance with Apache Incubator guidelines. The close relationship
> between
> > the Myriad team and Apache Mesos and Apache Hadoop means there is an
> > awareness of the incubation process and a willingness to embrace The
> Apache
> > Way.
> >
> > ** Homogenous Developers
> > There is already diversity in the core developer community as they are
> > employed by three different and independent companies viz. ebay inc.,
> MapR,
> > and Mesosphere. However, there will continue to be an emphasis on
> > increasing the diversity of the developer community.
> >
> > ** Reliance on Salaried Developers
> > Currently, the core developers are paid to work on Myriad. However, once
> > the project has a community built around it, we expect to get committers,
> > contributors and community from outside the current participating
> > organizations.
> >
> > ** Relationships with Other Apache Products
> > Myriad implements interfaces from both Apache YARN and Apache Mesos, and
> > requires both to be present so that Myriad can coordinate dynamic
> resource
> > sharing between the two.
> >
> > ** An Excessive Fascination with the Apache Brand
> > While we respect the reputation of the Apache brand and have no doubts
> that
> > it will attract contributors and users, our interest is primarily to give
> > Myriad a solid home as an open source project following an established
> > development model. We have also given reasons in the Rationale and
> > Alignment sections.
> >
> > * Documentation
> > Documentation is included in a docs directory of the repository (See
> > https://github.com/mesos/myriad/tree/phase1/docs), and currently details
> > how Myriad works, developing the project, auto-scaling a YARN cluster,
> the
> > Myriad REST API, and more. We will improve docs at every revision drop.
> >
> > * Initial Source
> > The Myriad codebase has been posted on GitHub for review and licensed
> under
> > an Apache v2 license.
> > https://github.com/mesos/myriad
> >
> > * Source and IP Submission Plan
> > During incubation, the codebase will be available at
> > https://github.com/apache/incubator-myriad/ and contributors will commit
> > appropriate contribute license agreements.
> >
> > * External Dependencies
> > All Myriad dependencies have Apache compatible licenses.
> >
> > * Cryptography
> > Myriad doesn’t use cryptography itself. Hadoop and Mesos projects,
> however,
> > use standard API’s and tools for SSH And SSL communication where
> necessary.
> >
> > * Required Resources
> > ** Mailing Lists
> > - myriad-private for private PMC conversations
> > - myriad-dev
> > - myriad-commits
> > - myriad-user
> >
> > ** Version Control
> > We prefer to use Git as our source control system: git://
> > git.apache.org/myriad
> >
> > ** Issue Tracking
> > JIRA Myriad (MYRIAD)
> >
> > * Initial Committers
> > - Santosh Marella (smarella at mapr dot com)
> > - Mohit Soni (mohitsoni1989 at gmail dot com)
> > - Adam Bordelon (me at apache dot org) *
> > - Meghdoot Bhattacharya ( mbhattacharya at paypal dot com)
> > - Anoop Dawar (anoopdawar at gmail dot com)
> > - Jim Scott (jim at 13ways dot com)
> > - Ken Sipe (kensipe at gmail dot com)
> >
> > * Affiliations
> > - Santosh Marella, MapR
> > - Mohit Soni, ebay Inc.
> > - Adam Bordelon, Mesosphere
> > - Meghdoot Bhattacharya, ebay Inc.
> > - Anoop Dawar, MapR
> > - Jim Scott, MapR
> > - Ken Sipe, Mesosphere
> >
> > * Sponsors
> > ** Champion (Proposal)
> > - Ben Hindman (benh at apache dot org)
> >
> > ** Nominated Mentors
> > - Ben Hindman (benh at apache dot org) - Mesosphere
> > - Danese Cooper (danese at apache dot org) - ebay, Inc.
> > - Ted Dunning (tdunning at apache dot org) - MapR
> >
> > ** Sponsoring Entity
> > Apache Incubator
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Henry Saputra <he...@gmail.com>.
I love this project and the idea. Tried to hack it couple years ago
could not make it work.
Looking forward seeing it in ASF incubator for sure.
@Adam and @Ted, like any new incubator projects coming we always check
if you need user@ so early in the process?
Would probably better to have all discussion in dev@ early in incubation.
- Henry
On Fri, Feb 13, 2015 at 5:06 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
> Hello friends,
>
> The Myriad team and I would like to propose the Myriad project for
> inclusion in the Apache Incubator.
> Full text of the proposal is below. I can add it to the incubator wiki as
> well, if desired.
> Please review and discuss. If there are no major concerns, I will call for
> a Vote after a week.
>
> Cheers,
> -Adam-
> me@apache
>
> ==========================================================
> Apache Myriad Proposal
>
> * Abstract
> Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos together
> on the same cluster and allows dynamic resource allocations across both
> Hadoop and other applications running on the same physical data center
> infrastructure.
>
> * Proposal
> The vision of Myriad is to provide a comprehensive framework to ensure
> Apache Hadoop YARN and Apache Mesos can interoperate with minimal changes
> on either side and prevent the static fragmentation of data center
> resources.
>
> * Background
> Project Myriad is the first resource management framework that allows big
> data developers to run YARN-based Hadoop jobs alongside other applications
> and services in production. ebay Inc., MapR, and Mesosphere jointly built
> Myriad (available on Github at https://github.com/mesos/myriad) with the
> vision of freeing big data jobs from siloed clusters and consolidating
> infrastructure into a single pool of resources for greater utilization and
> operational efficiency. Several companies including Twitter have expressed
> interest in Myriad and have begun testing it.
>
> * Rationale
> Many Hadoop users are building larger clusters (data lake/data hub
> architectures) that support multiple workloads - made possible by the
> advent of Apache Hadoop YARN. As the clusters grow in size and importance,
> they become an important application within the broader datacenter. At the
> same time, Apache Mesos enables efficient resource isolation and sharing
> across distributed applications for the broader data center, for instance
> MPI, Spark, long running web services, build/test infrastructure,
> traditional linux applications/scripts, and others (including arbitrary
> docker images).
>
> Myriad aims to enable co-existence of Apache Hadoop YARN and Apache Mesos
> on the same physical data center resources, reducing fragmentation of data
> center resources.
>
> * Project Goals
> ** Initial Goals
> - Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow policy
> based allocation of data center resources across Apache Hadoop and other
> distributed applications
> - Ensure YARN based execution frameworks work without any changes when
> running alongside Myriad. YARN Applications will continue to interact and
> run on top of YARN and can choose to be unaware of Myriad.
> - Ensure Mesos based execution frameworks work without any changes when
> running alongside Myriad. Mesos applications will continue to interact and
> run on Mesos and can choose to be unaware of Myriad.
> - Provide isolation for multi-tenancy.
> - Use linux cgroups (and optionally Docker-like technologies to ease
> packaging, deployment and broader isolation) so that multiple YARN clusters
> can run in their own space and are isolated from each other. YARN’s RM and
> NMs are dockerized.
> - Myriad should be able to manage full YARN lifecycle:
> - Bring up YARN (RM, NM)
> - Scale Up/Down YARN
> - Release resources and shut down YARN
>
> ** Longer Term Goals
> - Allow fine-grained dynamic allocation of resources to Hadoop including
> the ability to scale up and scale down the cluster.
> - Provide different policies to allow downsizing running applications on
> Hadoop when resources are taken away from it.
> - Provide a framework so the downsizing policy is pluggable and users can
> write their own implementations.
> - Allow multiple versions of Apache Hadoop to run on the same physical
> infrastructure
> - Allow workload portability - ability to migrate YARN workloads across
> various cloud infrastructures seamlessly (e.g. GCE, AWS, etc)
> - Security:
> - Authentication Requirements:
> - Support basic CRAM-MD5 password authentication between Myriad and
> Mesos. Additional authentication mechanisms may be supported in the future.
> - Traditional user authentication with Hadoop’s HTTP web-consoles
> should work as usual.
> - Authorization:
> - Only authorized users are allowed to launch YARN clusters. Mesos
> allows to specify which framework principal is allowed to register as a
> particular role.
> - Encryption on wire:
> - All control traffic to/from Myriad/Mesos
> - Logs
> - Audits (where to store them)
> - Log all major activities/events with audit trail - who, what, when,
> result
> - Launching YARN/RM
> - Launching NM’s
> - Downsizing NM’s
> - Terminating YARN/RM
> - What to do with old logs?
> - Debuggability/Visibility
> - Hooks to identify different YARN cluster lifecycles (yarn-id?)
> - GUI: Capability to scale-up and scale-down by selecting nodes and
> providing a scale-up/scale-down factor.
>
> * Architectural Overview
> The following diagram illustrates the high level architecture. YARN (with
> Myriad) is registered as a framework with Mesos master along with possibly
> other Mesos frameworks. This enables YARN to share cluster resources with
> other Mesos frameworks providing elasticity of resources between Hadoop
> workloads and Mesos frameworks.
>
> See
> https://github.com/mesos/myriad/blob/phase1/docs/images/high-level-architecture.png
>
> * Current Status
> Myriad is under active development. Key components of Myriad are:
> ** Myriad Resource Manager (RM) Plugin
> - Plugs into Resource Manager Java process via yarn-site.xml configuration.
> - Registers Myriad as a framework with Mesos. Receives resource offers from
> Mesos.
> - Monitors YARN’s application pipeline and scheduling events to drive
> scale-up or scale-down decisions for Hadoop.
> - Exposes REST APIs to help admins control Hadoop/YARN’s resource
> consumption. Currently the following APIs are supported:
> - Scale Up (e.g. “launch 4 Node Manager instances with 10G/6CPU capacity”)
> - Scale Down (e.g. “kill 2 Node Manager instances with 10G/6CPU
> capacity”)
>
> ** Myriad Mesos Executor
> - Launched on a Mesos slave node by Myriad RM plugin via Mesos.
> - Responsible for launching Node Manager process with appropriate
> capacities configured in yarn-site.xml.
> - Mounts YARN’s cgroup hierarchy under Mesos’ cgroup hierarchy in case
> YARN’s cgroups are enabled.
>
> Currently, a working prototype/demo had been built for the goals listed
> under the “Initial Goals” section. Open issues and enhancements are tracked
> at https://github.com/mesos/myriad/issues. Myriad is not yet tested for
> production use.
>
> ** Meritocracy
> We plan to invest in supporting a meritocracy. We will discuss the
> requirements in a public forum. Several companies have already expressed
> interest in this project, and we intend to invite developers to contribute
> and gain karma. We will encourage and monitor community participation so
> that privileges can be extended to those that contribute.
>
> ** Community
> We are happy to report that there are existing Apache committers and
> corporate users who are closely involved in the project already. We hope to
> extend the user and developer base further in the future and build a solid
> open source community around Myriad, growing the community and adding
> committers following the Apache Way.
>
> ** Core Developers
> The initial technology was built independently by ebay and MapR. ebay built
> the technology in consultation with Ben Hindman. MapR built a working
> prototype in tight consultation and mentorship with Mesosphere.
>
> ** Alignment
> The initial committers strongly believe that Apache Hadoop YARN and Apache
> Mesos will gain broad adoption and therefore a framework to allow for a
> co-existence of these frameworks that is transparent to applications
> written for YARN and Mesos will serve the needs of the broader community.
>
> * Known Risks
>
> ** Inexperience with Open Source
> Initial Myriad committers have varying levels of experience using and
> contributing to Open Source projects, however by working with our mentors
> and the Apache community we believe we will be able to conduct ourselves in
> accordance with Apache Incubator guidelines. The close relationship between
> the Myriad team and Apache Mesos and Apache Hadoop means there is an
> awareness of the incubation process and a willingness to embrace The Apache
> Way.
>
> ** Homogenous Developers
> There is already diversity in the core developer community as they are
> employed by three different and independent companies viz. ebay inc., MapR,
> and Mesosphere. However, there will continue to be an emphasis on
> increasing the diversity of the developer community.
>
> ** Reliance on Salaried Developers
> Currently, the core developers are paid to work on Myriad. However, once
> the project has a community built around it, we expect to get committers,
> contributors and community from outside the current participating
> organizations.
>
> ** Relationships with Other Apache Products
> Myriad implements interfaces from both Apache YARN and Apache Mesos, and
> requires both to be present so that Myriad can coordinate dynamic resource
> sharing between the two.
>
> ** An Excessive Fascination with the Apache Brand
> While we respect the reputation of the Apache brand and have no doubts that
> it will attract contributors and users, our interest is primarily to give
> Myriad a solid home as an open source project following an established
> development model. We have also given reasons in the Rationale and
> Alignment sections.
>
> * Documentation
> Documentation is included in a docs directory of the repository (See
> https://github.com/mesos/myriad/tree/phase1/docs), and currently details
> how Myriad works, developing the project, auto-scaling a YARN cluster, the
> Myriad REST API, and more. We will improve docs at every revision drop.
>
> * Initial Source
> The Myriad codebase has been posted on GitHub for review and licensed under
> an Apache v2 license.
> https://github.com/mesos/myriad
>
> * Source and IP Submission Plan
> During incubation, the codebase will be available at
> https://github.com/apache/incubator-myriad/ and contributors will commit
> appropriate contribute license agreements.
>
> * External Dependencies
> All Myriad dependencies have Apache compatible licenses.
>
> * Cryptography
> Myriad doesn’t use cryptography itself. Hadoop and Mesos projects, however,
> use standard API’s and tools for SSH And SSL communication where necessary.
>
> * Required Resources
> ** Mailing Lists
> - myriad-private for private PMC conversations
> - myriad-dev
> - myriad-commits
> - myriad-user
>
> ** Version Control
> We prefer to use Git as our source control system: git://
> git.apache.org/myriad
>
> ** Issue Tracking
> JIRA Myriad (MYRIAD)
>
> * Initial Committers
> - Santosh Marella (smarella at mapr dot com)
> - Mohit Soni (mohitsoni1989 at gmail dot com)
> - Adam Bordelon (me at apache dot org) *
> - Meghdoot Bhattacharya ( mbhattacharya at paypal dot com)
> - Anoop Dawar (anoopdawar at gmail dot com)
> - Jim Scott (jim at 13ways dot com)
> - Ken Sipe (kensipe at gmail dot com)
>
> * Affiliations
> - Santosh Marella, MapR
> - Mohit Soni, ebay Inc.
> - Adam Bordelon, Mesosphere
> - Meghdoot Bhattacharya, ebay Inc.
> - Anoop Dawar, MapR
> - Jim Scott, MapR
> - Ken Sipe, Mesosphere
>
> * Sponsors
> ** Champion (Proposal)
> - Ben Hindman (benh at apache dot org)
>
> ** Nominated Mentors
> - Ben Hindman (benh at apache dot org) - Mesosphere
> - Danese Cooper (danese at apache dot org) - ebay, Inc.
> - Ted Dunning (tdunning at apache dot org) - MapR
>
> ** Sponsoring Entity
> Apache Incubator
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Ted Dunning <te...@gmail.com>.
On Fri, Feb 13, 2015 at 5:06 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
> I can add it to the incubator wiki as
> well, if desired.
>
I added this to the incubator wiki just now.
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Ted Dunning <te...@gmail.com>.
In case there is any doubt, +1 from me!
On Fri, Feb 13, 2015 at 5:15 PM, Luciano Resende <lu...@gmail.com>
wrote:
> On Fri, Feb 13, 2015 at 5:06 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
>
> > Hello friends,
> >
> > The Myriad team and I would like to propose the Myriad project for
> > inclusion in the Apache Incubator.
> > Full text of the proposal is below. I can add it to the incubator wiki as
> > well, if desired.
> > Please review and discuss. If there are no major concerns, I will call
> for
> > a Vote after a week.
> >
> > Cheers,
> > -Adam-
> > me@apache
> >
> > ==========================================================
> > Apache Myriad Proposal
> >
> > * Abstract
> > Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos
> together
> > on the same cluster and allows dynamic resource allocations across both
> > Hadoop and other applications running on the same physical data center
> > infrastructure.
> >
> > * Proposal
> > The vision of Myriad is to provide a comprehensive framework to ensure
> > Apache Hadoop YARN and Apache Mesos can interoperate with minimal changes
> > on either side and prevent the static fragmentation of data center
> > resources.
> >
> > * Background
> > Project Myriad is the first resource management framework that allows big
> > data developers to run YARN-based Hadoop jobs alongside other
> applications
> > and services in production. ebay Inc., MapR, and Mesosphere jointly built
> > Myriad (available on Github at https://github.com/mesos/myriad) with the
> > vision of freeing big data jobs from siloed clusters and consolidating
> > infrastructure into a single pool of resources for greater utilization
> and
> > operational efficiency. Several companies including Twitter have
> expressed
> > interest in Myriad and have begun testing it.
> >
> > * Rationale
> > Many Hadoop users are building larger clusters (data lake/data hub
> > architectures) that support multiple workloads - made possible by the
> > advent of Apache Hadoop YARN. As the clusters grow in size and
> importance,
> > they become an important application within the broader datacenter. At
> the
> > same time, Apache Mesos enables efficient resource isolation and sharing
> > across distributed applications for the broader data center, for instance
> > MPI, Spark, long running web services, build/test infrastructure,
> > traditional linux applications/scripts, and others (including arbitrary
> > docker images).
> >
> > Myriad aims to enable co-existence of Apache Hadoop YARN and Apache Mesos
> > on the same physical data center resources, reducing fragmentation of
> data
> > center resources.
> >
> > * Project Goals
> > ** Initial Goals
> > - Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow
> policy
> > based allocation of data center resources across Apache Hadoop and other
> > distributed applications
> > - Ensure YARN based execution frameworks work without any changes when
> > running alongside Myriad. YARN Applications will continue to interact and
> > run on top of YARN and can choose to be unaware of Myriad.
> > - Ensure Mesos based execution frameworks work without any changes when
> > running alongside Myriad. Mesos applications will continue to interact
> and
> > run on Mesos and can choose to be unaware of Myriad.
> > - Provide isolation for multi-tenancy.
> > - Use linux cgroups (and optionally Docker-like technologies to ease
> > packaging, deployment and broader isolation) so that multiple YARN
> clusters
> > can run in their own space and are isolated from each other. YARN’s RM
> and
> > NMs are dockerized.
> > - Myriad should be able to manage full YARN lifecycle:
> > - Bring up YARN (RM, NM)
> > - Scale Up/Down YARN
> > - Release resources and shut down YARN
> >
> > ** Longer Term Goals
> > - Allow fine-grained dynamic allocation of resources to Hadoop including
> > the ability to scale up and scale down the cluster.
> > - Provide different policies to allow downsizing running applications
> on
> > Hadoop when resources are taken away from it.
> > - Provide a framework so the downsizing policy is pluggable and users
> can
> > write their own implementations.
> > - Allow multiple versions of Apache Hadoop to run on the same physical
> > infrastructure
> > - Allow workload portability - ability to migrate YARN workloads across
> > various cloud infrastructures seamlessly (e.g. GCE, AWS, etc)
> > - Security:
> > - Authentication Requirements:
> > - Support basic CRAM-MD5 password authentication between Myriad and
> > Mesos. Additional authentication mechanisms may be supported in the
> future.
> > - Traditional user authentication with Hadoop’s HTTP web-consoles
> > should work as usual.
> > - Authorization:
> > - Only authorized users are allowed to launch YARN clusters. Mesos
> > allows to specify which framework principal is allowed to register as a
> > particular role.
> > - Encryption on wire:
> > - All control traffic to/from Myriad/Mesos
> > - Logs
> > - Audits (where to store them)
> > - Log all major activities/events with audit trail - who, what, when,
> > result
> > - Launching YARN/RM
> > - Launching NM’s
> > - Downsizing NM’s
> > - Terminating YARN/RM
> > - What to do with old logs?
> > - Debuggability/Visibility
> > - Hooks to identify different YARN cluster lifecycles (yarn-id?)
> > - GUI: Capability to scale-up and scale-down by selecting nodes and
> > providing a scale-up/scale-down factor.
> >
> > * Architectural Overview
> > The following diagram illustrates the high level architecture. YARN (with
> > Myriad) is registered as a framework with Mesos master along with
> possibly
> > other Mesos frameworks. This enables YARN to share cluster resources with
> > other Mesos frameworks providing elasticity of resources between Hadoop
> > workloads and Mesos frameworks.
> >
> > See
> >
> >
> https://github.com/mesos/myriad/blob/phase1/docs/images/high-level-architecture.png
> >
> > * Current Status
> > Myriad is under active development. Key components of Myriad are:
> > ** Myriad Resource Manager (RM) Plugin
> > - Plugs into Resource Manager Java process via yarn-site.xml
> configuration.
> > - Registers Myriad as a framework with Mesos. Receives resource offers
> from
> > Mesos.
> > - Monitors YARN’s application pipeline and scheduling events to drive
> > scale-up or scale-down decisions for Hadoop.
> > - Exposes REST APIs to help admins control Hadoop/YARN’s resource
> > consumption. Currently the following APIs are supported:
> > - Scale Up (e.g. “launch 4 Node Manager instances with 10G/6CPU
> > capacity”)
> > - Scale Down (e.g. “kill 2 Node Manager instances with 10G/6CPU
> > capacity”)
> >
> > ** Myriad Mesos Executor
> > - Launched on a Mesos slave node by Myriad RM plugin via Mesos.
> > - Responsible for launching Node Manager process with appropriate
> > capacities configured in yarn-site.xml.
> > - Mounts YARN’s cgroup hierarchy under Mesos’ cgroup hierarchy in case
> > YARN’s cgroups are enabled.
> >
> > Currently, a working prototype/demo had been built for the goals listed
> > under the “Initial Goals” section. Open issues and enhancements are
> tracked
> > at https://github.com/mesos/myriad/issues. Myriad is not yet tested for
> > production use.
> >
> > ** Meritocracy
> > We plan to invest in supporting a meritocracy. We will discuss the
> > requirements in a public forum. Several companies have already expressed
> > interest in this project, and we intend to invite developers to
> contribute
> > and gain karma. We will encourage and monitor community participation so
> > that privileges can be extended to those that contribute.
> >
> > ** Community
> > We are happy to report that there are existing Apache committers and
> > corporate users who are closely involved in the project already. We hope
> to
> > extend the user and developer base further in the future and build a
> solid
> > open source community around Myriad, growing the community and adding
> > committers following the Apache Way.
> >
> > ** Core Developers
> > The initial technology was built independently by ebay and MapR. ebay
> built
> > the technology in consultation with Ben Hindman. MapR built a working
> > prototype in tight consultation and mentorship with Mesosphere.
> >
> > ** Alignment
> > The initial committers strongly believe that Apache Hadoop YARN and
> Apache
> > Mesos will gain broad adoption and therefore a framework to allow for a
> > co-existence of these frameworks that is transparent to applications
> > written for YARN and Mesos will serve the needs of the broader community.
> >
> > * Known Risks
> >
> > ** Inexperience with Open Source
> > Initial Myriad committers have varying levels of experience using and
> > contributing to Open Source projects, however by working with our mentors
> > and the Apache community we believe we will be able to conduct ourselves
> in
> > accordance with Apache Incubator guidelines. The close relationship
> between
> > the Myriad team and Apache Mesos and Apache Hadoop means there is an
> > awareness of the incubation process and a willingness to embrace The
> Apache
> > Way.
> >
> > ** Homogenous Developers
> > There is already diversity in the core developer community as they are
> > employed by three different and independent companies viz. ebay inc.,
> MapR,
> > and Mesosphere. However, there will continue to be an emphasis on
> > increasing the diversity of the developer community.
> >
> > ** Reliance on Salaried Developers
> > Currently, the core developers are paid to work on Myriad. However, once
> > the project has a community built around it, we expect to get committers,
> > contributors and community from outside the current participating
> > organizations.
> >
> > ** Relationships with Other Apache Products
> > Myriad implements interfaces from both Apache YARN and Apache Mesos, and
> > requires both to be present so that Myriad can coordinate dynamic
> resource
> > sharing between the two.
> >
> > ** An Excessive Fascination with the Apache Brand
> > While we respect the reputation of the Apache brand and have no doubts
> that
> > it will attract contributors and users, our interest is primarily to give
> > Myriad a solid home as an open source project following an established
> > development model. We have also given reasons in the Rationale and
> > Alignment sections.
> >
> > * Documentation
> > Documentation is included in a docs directory of the repository (See
> > https://github.com/mesos/myriad/tree/phase1/docs), and currently details
> > how Myriad works, developing the project, auto-scaling a YARN cluster,
> the
> > Myriad REST API, and more. We will improve docs at every revision drop.
> >
> > * Initial Source
> > The Myriad codebase has been posted on GitHub for review and licensed
> under
> > an Apache v2 license.
> > https://github.com/mesos/myriad
> >
> > * Source and IP Submission Plan
> > During incubation, the codebase will be available at
> > https://github.com/apache/incubator-myriad/ and contributors will commit
> > appropriate contribute license agreements.
> >
> > * External Dependencies
> > All Myriad dependencies have Apache compatible licenses.
> >
> > * Cryptography
> > Myriad doesn’t use cryptography itself. Hadoop and Mesos projects,
> however,
> > use standard API’s and tools for SSH And SSL communication where
> necessary.
> >
> > * Required Resources
> > ** Mailing Lists
> > - myriad-private for private PMC conversations
> > - myriad-dev
> > - myriad-commits
> > - myriad-user
> >
> > ** Version Control
> > We prefer to use Git as our source control system: git://
> > git.apache.org/myriad
> >
> > ** Issue Tracking
> > JIRA Myriad (MYRIAD)
> >
> > * Initial Committers
> > - Santosh Marella (smarella at mapr dot com)
> > - Mohit Soni (mohitsoni1989 at gmail dot com)
> > - Adam Bordelon (me at apache dot org) *
> > - Meghdoot Bhattacharya ( mbhattacharya at paypal dot com)
> > - Anoop Dawar (anoopdawar at gmail dot com)
> > - Jim Scott (jim at 13ways dot com)
> > - Ken Sipe (kensipe at gmail dot com)
> >
> > * Affiliations
> > - Santosh Marella, MapR
> > - Mohit Soni, ebay Inc.
> > - Adam Bordelon, Mesosphere
> > - Meghdoot Bhattacharya, ebay Inc.
> > - Anoop Dawar, MapR
> > - Jim Scott, MapR
> > - Ken Sipe, Mesosphere
> >
> > * Sponsors
> > ** Champion (Proposal)
> > - Ben Hindman (benh at apache dot org)
> >
> > ** Nominated Mentors
> > - Ben Hindman (benh at apache dot org) - Mesosphere
> > - Danese Cooper (danese at apache dot org) - ebay, Inc.
> > - Ted Dunning (tdunning at apache dot org) - MapR
> >
> > ** Sponsoring Entity
> > Apache Incubator
> >
>
>
> Interesting, +1, If you guys need an extra mentor (or committer) please
> count me in.
>
> --
> Luciano Resende
> http://people.apache.org/~lresende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Ted Dunning <te...@gmail.com>.
Luciano,
I would expect that you would make an excellent additional mentor.
On Fri, Feb 13, 2015 at 5:15 PM, Luciano Resende <lu...@gmail.com>
wrote:
> On Fri, Feb 13, 2015 at 5:06 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
>
> > Hello friends,
> >
> > The Myriad team and I would like to propose the Myriad project for
> > inclusion in the Apache Incubator.
> > Full text of the proposal is below. I can add it to the incubator wiki as
> > well, if desired.
> > Please review and discuss. If there are no major concerns, I will call
> for
> > a Vote after a week.
> >
> > Cheers,
> > -Adam-
> > me@apache
> >
> > ==========================================================
> > Apache Myriad Proposal
> >
> > * Abstract
> > Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos
> together
> > on the same cluster and allows dynamic resource allocations across both
> > Hadoop and other applications running on the same physical data center
> > infrastructure.
> >
> > * Proposal
> > The vision of Myriad is to provide a comprehensive framework to ensure
> > Apache Hadoop YARN and Apache Mesos can interoperate with minimal changes
> > on either side and prevent the static fragmentation of data center
> > resources.
> >
> > * Background
> > Project Myriad is the first resource management framework that allows big
> > data developers to run YARN-based Hadoop jobs alongside other
> applications
> > and services in production. ebay Inc., MapR, and Mesosphere jointly built
> > Myriad (available on Github at https://github.com/mesos/myriad) with the
> > vision of freeing big data jobs from siloed clusters and consolidating
> > infrastructure into a single pool of resources for greater utilization
> and
> > operational efficiency. Several companies including Twitter have
> expressed
> > interest in Myriad and have begun testing it.
> >
> > * Rationale
> > Many Hadoop users are building larger clusters (data lake/data hub
> > architectures) that support multiple workloads - made possible by the
> > advent of Apache Hadoop YARN. As the clusters grow in size and
> importance,
> > they become an important application within the broader datacenter. At
> the
> > same time, Apache Mesos enables efficient resource isolation and sharing
> > across distributed applications for the broader data center, for instance
> > MPI, Spark, long running web services, build/test infrastructure,
> > traditional linux applications/scripts, and others (including arbitrary
> > docker images).
> >
> > Myriad aims to enable co-existence of Apache Hadoop YARN and Apache Mesos
> > on the same physical data center resources, reducing fragmentation of
> data
> > center resources.
> >
> > * Project Goals
> > ** Initial Goals
> > - Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow
> policy
> > based allocation of data center resources across Apache Hadoop and other
> > distributed applications
> > - Ensure YARN based execution frameworks work without any changes when
> > running alongside Myriad. YARN Applications will continue to interact and
> > run on top of YARN and can choose to be unaware of Myriad.
> > - Ensure Mesos based execution frameworks work without any changes when
> > running alongside Myriad. Mesos applications will continue to interact
> and
> > run on Mesos and can choose to be unaware of Myriad.
> > - Provide isolation for multi-tenancy.
> > - Use linux cgroups (and optionally Docker-like technologies to ease
> > packaging, deployment and broader isolation) so that multiple YARN
> clusters
> > can run in their own space and are isolated from each other. YARN’s RM
> and
> > NMs are dockerized.
> > - Myriad should be able to manage full YARN lifecycle:
> > - Bring up YARN (RM, NM)
> > - Scale Up/Down YARN
> > - Release resources and shut down YARN
> >
> > ** Longer Term Goals
> > - Allow fine-grained dynamic allocation of resources to Hadoop including
> > the ability to scale up and scale down the cluster.
> > - Provide different policies to allow downsizing running applications
> on
> > Hadoop when resources are taken away from it.
> > - Provide a framework so the downsizing policy is pluggable and users
> can
> > write their own implementations.
> > - Allow multiple versions of Apache Hadoop to run on the same physical
> > infrastructure
> > - Allow workload portability - ability to migrate YARN workloads across
> > various cloud infrastructures seamlessly (e.g. GCE, AWS, etc)
> > - Security:
> > - Authentication Requirements:
> > - Support basic CRAM-MD5 password authentication between Myriad and
> > Mesos. Additional authentication mechanisms may be supported in the
> future.
> > - Traditional user authentication with Hadoop’s HTTP web-consoles
> > should work as usual.
> > - Authorization:
> > - Only authorized users are allowed to launch YARN clusters. Mesos
> > allows to specify which framework principal is allowed to register as a
> > particular role.
> > - Encryption on wire:
> > - All control traffic to/from Myriad/Mesos
> > - Logs
> > - Audits (where to store them)
> > - Log all major activities/events with audit trail - who, what, when,
> > result
> > - Launching YARN/RM
> > - Launching NM’s
> > - Downsizing NM’s
> > - Terminating YARN/RM
> > - What to do with old logs?
> > - Debuggability/Visibility
> > - Hooks to identify different YARN cluster lifecycles (yarn-id?)
> > - GUI: Capability to scale-up and scale-down by selecting nodes and
> > providing a scale-up/scale-down factor.
> >
> > * Architectural Overview
> > The following diagram illustrates the high level architecture. YARN (with
> > Myriad) is registered as a framework with Mesos master along with
> possibly
> > other Mesos frameworks. This enables YARN to share cluster resources with
> > other Mesos frameworks providing elasticity of resources between Hadoop
> > workloads and Mesos frameworks.
> >
> > See
> >
> >
> https://github.com/mesos/myriad/blob/phase1/docs/images/high-level-architecture.png
> >
> > * Current Status
> > Myriad is under active development. Key components of Myriad are:
> > ** Myriad Resource Manager (RM) Plugin
> > - Plugs into Resource Manager Java process via yarn-site.xml
> configuration.
> > - Registers Myriad as a framework with Mesos. Receives resource offers
> from
> > Mesos.
> > - Monitors YARN’s application pipeline and scheduling events to drive
> > scale-up or scale-down decisions for Hadoop.
> > - Exposes REST APIs to help admins control Hadoop/YARN’s resource
> > consumption. Currently the following APIs are supported:
> > - Scale Up (e.g. “launch 4 Node Manager instances with 10G/6CPU
> > capacity”)
> > - Scale Down (e.g. “kill 2 Node Manager instances with 10G/6CPU
> > capacity”)
> >
> > ** Myriad Mesos Executor
> > - Launched on a Mesos slave node by Myriad RM plugin via Mesos.
> > - Responsible for launching Node Manager process with appropriate
> > capacities configured in yarn-site.xml.
> > - Mounts YARN’s cgroup hierarchy under Mesos’ cgroup hierarchy in case
> > YARN’s cgroups are enabled.
> >
> > Currently, a working prototype/demo had been built for the goals listed
> > under the “Initial Goals” section. Open issues and enhancements are
> tracked
> > at https://github.com/mesos/myriad/issues. Myriad is not yet tested for
> > production use.
> >
> > ** Meritocracy
> > We plan to invest in supporting a meritocracy. We will discuss the
> > requirements in a public forum. Several companies have already expressed
> > interest in this project, and we intend to invite developers to
> contribute
> > and gain karma. We will encourage and monitor community participation so
> > that privileges can be extended to those that contribute.
> >
> > ** Community
> > We are happy to report that there are existing Apache committers and
> > corporate users who are closely involved in the project already. We hope
> to
> > extend the user and developer base further in the future and build a
> solid
> > open source community around Myriad, growing the community and adding
> > committers following the Apache Way.
> >
> > ** Core Developers
> > The initial technology was built independently by ebay and MapR. ebay
> built
> > the technology in consultation with Ben Hindman. MapR built a working
> > prototype in tight consultation and mentorship with Mesosphere.
> >
> > ** Alignment
> > The initial committers strongly believe that Apache Hadoop YARN and
> Apache
> > Mesos will gain broad adoption and therefore a framework to allow for a
> > co-existence of these frameworks that is transparent to applications
> > written for YARN and Mesos will serve the needs of the broader community.
> >
> > * Known Risks
> >
> > ** Inexperience with Open Source
> > Initial Myriad committers have varying levels of experience using and
> > contributing to Open Source projects, however by working with our mentors
> > and the Apache community we believe we will be able to conduct ourselves
> in
> > accordance with Apache Incubator guidelines. The close relationship
> between
> > the Myriad team and Apache Mesos and Apache Hadoop means there is an
> > awareness of the incubation process and a willingness to embrace The
> Apache
> > Way.
> >
> > ** Homogenous Developers
> > There is already diversity in the core developer community as they are
> > employed by three different and independent companies viz. ebay inc.,
> MapR,
> > and Mesosphere. However, there will continue to be an emphasis on
> > increasing the diversity of the developer community.
> >
> > ** Reliance on Salaried Developers
> > Currently, the core developers are paid to work on Myriad. However, once
> > the project has a community built around it, we expect to get committers,
> > contributors and community from outside the current participating
> > organizations.
> >
> > ** Relationships with Other Apache Products
> > Myriad implements interfaces from both Apache YARN and Apache Mesos, and
> > requires both to be present so that Myriad can coordinate dynamic
> resource
> > sharing between the two.
> >
> > ** An Excessive Fascination with the Apache Brand
> > While we respect the reputation of the Apache brand and have no doubts
> that
> > it will attract contributors and users, our interest is primarily to give
> > Myriad a solid home as an open source project following an established
> > development model. We have also given reasons in the Rationale and
> > Alignment sections.
> >
> > * Documentation
> > Documentation is included in a docs directory of the repository (See
> > https://github.com/mesos/myriad/tree/phase1/docs), and currently details
> > how Myriad works, developing the project, auto-scaling a YARN cluster,
> the
> > Myriad REST API, and more. We will improve docs at every revision drop.
> >
> > * Initial Source
> > The Myriad codebase has been posted on GitHub for review and licensed
> under
> > an Apache v2 license.
> > https://github.com/mesos/myriad
> >
> > * Source and IP Submission Plan
> > During incubation, the codebase will be available at
> > https://github.com/apache/incubator-myriad/ and contributors will commit
> > appropriate contribute license agreements.
> >
> > * External Dependencies
> > All Myriad dependencies have Apache compatible licenses.
> >
> > * Cryptography
> > Myriad doesn’t use cryptography itself. Hadoop and Mesos projects,
> however,
> > use standard API’s and tools for SSH And SSL communication where
> necessary.
> >
> > * Required Resources
> > ** Mailing Lists
> > - myriad-private for private PMC conversations
> > - myriad-dev
> > - myriad-commits
> > - myriad-user
> >
> > ** Version Control
> > We prefer to use Git as our source control system: git://
> > git.apache.org/myriad
> >
> > ** Issue Tracking
> > JIRA Myriad (MYRIAD)
> >
> > * Initial Committers
> > - Santosh Marella (smarella at mapr dot com)
> > - Mohit Soni (mohitsoni1989 at gmail dot com)
> > - Adam Bordelon (me at apache dot org) *
> > - Meghdoot Bhattacharya ( mbhattacharya at paypal dot com)
> > - Anoop Dawar (anoopdawar at gmail dot com)
> > - Jim Scott (jim at 13ways dot com)
> > - Ken Sipe (kensipe at gmail dot com)
> >
> > * Affiliations
> > - Santosh Marella, MapR
> > - Mohit Soni, ebay Inc.
> > - Adam Bordelon, Mesosphere
> > - Meghdoot Bhattacharya, ebay Inc.
> > - Anoop Dawar, MapR
> > - Jim Scott, MapR
> > - Ken Sipe, Mesosphere
> >
> > * Sponsors
> > ** Champion (Proposal)
> > - Ben Hindman (benh at apache dot org)
> >
> > ** Nominated Mentors
> > - Ben Hindman (benh at apache dot org) - Mesosphere
> > - Danese Cooper (danese at apache dot org) - ebay, Inc.
> > - Ted Dunning (tdunning at apache dot org) - MapR
> >
> > ** Sponsoring Entity
> > Apache Incubator
> >
>
>
> Interesting, +1, If you guys need an extra mentor (or committer) please
> count me in.
>
> --
> Luciano Resende
> http://people.apache.org/~lresende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Posted by Luciano Resende <lu...@gmail.com>.
On Fri, Feb 13, 2015 at 5:06 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
> Hello friends,
>
> The Myriad team and I would like to propose the Myriad project for
> inclusion in the Apache Incubator.
> Full text of the proposal is below. I can add it to the incubator wiki as
> well, if desired.
> Please review and discuss. If there are no major concerns, I will call for
> a Vote after a week.
>
> Cheers,
> -Adam-
> me@apache
>
> ==========================================================
> Apache Myriad Proposal
>
> * Abstract
> Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos together
> on the same cluster and allows dynamic resource allocations across both
> Hadoop and other applications running on the same physical data center
> infrastructure.
>
> * Proposal
> The vision of Myriad is to provide a comprehensive framework to ensure
> Apache Hadoop YARN and Apache Mesos can interoperate with minimal changes
> on either side and prevent the static fragmentation of data center
> resources.
>
> * Background
> Project Myriad is the first resource management framework that allows big
> data developers to run YARN-based Hadoop jobs alongside other applications
> and services in production. ebay Inc., MapR, and Mesosphere jointly built
> Myriad (available on Github at https://github.com/mesos/myriad) with the
> vision of freeing big data jobs from siloed clusters and consolidating
> infrastructure into a single pool of resources for greater utilization and
> operational efficiency. Several companies including Twitter have expressed
> interest in Myriad and have begun testing it.
>
> * Rationale
> Many Hadoop users are building larger clusters (data lake/data hub
> architectures) that support multiple workloads - made possible by the
> advent of Apache Hadoop YARN. As the clusters grow in size and importance,
> they become an important application within the broader datacenter. At the
> same time, Apache Mesos enables efficient resource isolation and sharing
> across distributed applications for the broader data center, for instance
> MPI, Spark, long running web services, build/test infrastructure,
> traditional linux applications/scripts, and others (including arbitrary
> docker images).
>
> Myriad aims to enable co-existence of Apache Hadoop YARN and Apache Mesos
> on the same physical data center resources, reducing fragmentation of data
> center resources.
>
> * Project Goals
> ** Initial Goals
> - Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow policy
> based allocation of data center resources across Apache Hadoop and other
> distributed applications
> - Ensure YARN based execution frameworks work without any changes when
> running alongside Myriad. YARN Applications will continue to interact and
> run on top of YARN and can choose to be unaware of Myriad.
> - Ensure Mesos based execution frameworks work without any changes when
> running alongside Myriad. Mesos applications will continue to interact and
> run on Mesos and can choose to be unaware of Myriad.
> - Provide isolation for multi-tenancy.
> - Use linux cgroups (and optionally Docker-like technologies to ease
> packaging, deployment and broader isolation) so that multiple YARN clusters
> can run in their own space and are isolated from each other. YARN’s RM and
> NMs are dockerized.
> - Myriad should be able to manage full YARN lifecycle:
> - Bring up YARN (RM, NM)
> - Scale Up/Down YARN
> - Release resources and shut down YARN
>
> ** Longer Term Goals
> - Allow fine-grained dynamic allocation of resources to Hadoop including
> the ability to scale up and scale down the cluster.
> - Provide different policies to allow downsizing running applications on
> Hadoop when resources are taken away from it.
> - Provide a framework so the downsizing policy is pluggable and users can
> write their own implementations.
> - Allow multiple versions of Apache Hadoop to run on the same physical
> infrastructure
> - Allow workload portability - ability to migrate YARN workloads across
> various cloud infrastructures seamlessly (e.g. GCE, AWS, etc)
> - Security:
> - Authentication Requirements:
> - Support basic CRAM-MD5 password authentication between Myriad and
> Mesos. Additional authentication mechanisms may be supported in the future.
> - Traditional user authentication with Hadoop’s HTTP web-consoles
> should work as usual.
> - Authorization:
> - Only authorized users are allowed to launch YARN clusters. Mesos
> allows to specify which framework principal is allowed to register as a
> particular role.
> - Encryption on wire:
> - All control traffic to/from Myriad/Mesos
> - Logs
> - Audits (where to store them)
> - Log all major activities/events with audit trail - who, what, when,
> result
> - Launching YARN/RM
> - Launching NM’s
> - Downsizing NM’s
> - Terminating YARN/RM
> - What to do with old logs?
> - Debuggability/Visibility
> - Hooks to identify different YARN cluster lifecycles (yarn-id?)
> - GUI: Capability to scale-up and scale-down by selecting nodes and
> providing a scale-up/scale-down factor.
>
> * Architectural Overview
> The following diagram illustrates the high level architecture. YARN (with
> Myriad) is registered as a framework with Mesos master along with possibly
> other Mesos frameworks. This enables YARN to share cluster resources with
> other Mesos frameworks providing elasticity of resources between Hadoop
> workloads and Mesos frameworks.
>
> See
>
> https://github.com/mesos/myriad/blob/phase1/docs/images/high-level-architecture.png
>
> * Current Status
> Myriad is under active development. Key components of Myriad are:
> ** Myriad Resource Manager (RM) Plugin
> - Plugs into Resource Manager Java process via yarn-site.xml configuration.
> - Registers Myriad as a framework with Mesos. Receives resource offers from
> Mesos.
> - Monitors YARN’s application pipeline and scheduling events to drive
> scale-up or scale-down decisions for Hadoop.
> - Exposes REST APIs to help admins control Hadoop/YARN’s resource
> consumption. Currently the following APIs are supported:
> - Scale Up (e.g. “launch 4 Node Manager instances with 10G/6CPU
> capacity”)
> - Scale Down (e.g. “kill 2 Node Manager instances with 10G/6CPU
> capacity”)
>
> ** Myriad Mesos Executor
> - Launched on a Mesos slave node by Myriad RM plugin via Mesos.
> - Responsible for launching Node Manager process with appropriate
> capacities configured in yarn-site.xml.
> - Mounts YARN’s cgroup hierarchy under Mesos’ cgroup hierarchy in case
> YARN’s cgroups are enabled.
>
> Currently, a working prototype/demo had been built for the goals listed
> under the “Initial Goals” section. Open issues and enhancements are tracked
> at https://github.com/mesos/myriad/issues. Myriad is not yet tested for
> production use.
>
> ** Meritocracy
> We plan to invest in supporting a meritocracy. We will discuss the
> requirements in a public forum. Several companies have already expressed
> interest in this project, and we intend to invite developers to contribute
> and gain karma. We will encourage and monitor community participation so
> that privileges can be extended to those that contribute.
>
> ** Community
> We are happy to report that there are existing Apache committers and
> corporate users who are closely involved in the project already. We hope to
> extend the user and developer base further in the future and build a solid
> open source community around Myriad, growing the community and adding
> committers following the Apache Way.
>
> ** Core Developers
> The initial technology was built independently by ebay and MapR. ebay built
> the technology in consultation with Ben Hindman. MapR built a working
> prototype in tight consultation and mentorship with Mesosphere.
>
> ** Alignment
> The initial committers strongly believe that Apache Hadoop YARN and Apache
> Mesos will gain broad adoption and therefore a framework to allow for a
> co-existence of these frameworks that is transparent to applications
> written for YARN and Mesos will serve the needs of the broader community.
>
> * Known Risks
>
> ** Inexperience with Open Source
> Initial Myriad committers have varying levels of experience using and
> contributing to Open Source projects, however by working with our mentors
> and the Apache community we believe we will be able to conduct ourselves in
> accordance with Apache Incubator guidelines. The close relationship between
> the Myriad team and Apache Mesos and Apache Hadoop means there is an
> awareness of the incubation process and a willingness to embrace The Apache
> Way.
>
> ** Homogenous Developers
> There is already diversity in the core developer community as they are
> employed by three different and independent companies viz. ebay inc., MapR,
> and Mesosphere. However, there will continue to be an emphasis on
> increasing the diversity of the developer community.
>
> ** Reliance on Salaried Developers
> Currently, the core developers are paid to work on Myriad. However, once
> the project has a community built around it, we expect to get committers,
> contributors and community from outside the current participating
> organizations.
>
> ** Relationships with Other Apache Products
> Myriad implements interfaces from both Apache YARN and Apache Mesos, and
> requires both to be present so that Myriad can coordinate dynamic resource
> sharing between the two.
>
> ** An Excessive Fascination with the Apache Brand
> While we respect the reputation of the Apache brand and have no doubts that
> it will attract contributors and users, our interest is primarily to give
> Myriad a solid home as an open source project following an established
> development model. We have also given reasons in the Rationale and
> Alignment sections.
>
> * Documentation
> Documentation is included in a docs directory of the repository (See
> https://github.com/mesos/myriad/tree/phase1/docs), and currently details
> how Myriad works, developing the project, auto-scaling a YARN cluster, the
> Myriad REST API, and more. We will improve docs at every revision drop.
>
> * Initial Source
> The Myriad codebase has been posted on GitHub for review and licensed under
> an Apache v2 license.
> https://github.com/mesos/myriad
>
> * Source and IP Submission Plan
> During incubation, the codebase will be available at
> https://github.com/apache/incubator-myriad/ and contributors will commit
> appropriate contribute license agreements.
>
> * External Dependencies
> All Myriad dependencies have Apache compatible licenses.
>
> * Cryptography
> Myriad doesn’t use cryptography itself. Hadoop and Mesos projects, however,
> use standard API’s and tools for SSH And SSL communication where necessary.
>
> * Required Resources
> ** Mailing Lists
> - myriad-private for private PMC conversations
> - myriad-dev
> - myriad-commits
> - myriad-user
>
> ** Version Control
> We prefer to use Git as our source control system: git://
> git.apache.org/myriad
>
> ** Issue Tracking
> JIRA Myriad (MYRIAD)
>
> * Initial Committers
> - Santosh Marella (smarella at mapr dot com)
> - Mohit Soni (mohitsoni1989 at gmail dot com)
> - Adam Bordelon (me at apache dot org) *
> - Meghdoot Bhattacharya ( mbhattacharya at paypal dot com)
> - Anoop Dawar (anoopdawar at gmail dot com)
> - Jim Scott (jim at 13ways dot com)
> - Ken Sipe (kensipe at gmail dot com)
>
> * Affiliations
> - Santosh Marella, MapR
> - Mohit Soni, ebay Inc.
> - Adam Bordelon, Mesosphere
> - Meghdoot Bhattacharya, ebay Inc.
> - Anoop Dawar, MapR
> - Jim Scott, MapR
> - Ken Sipe, Mesosphere
>
> * Sponsors
> ** Champion (Proposal)
> - Ben Hindman (benh at apache dot org)
>
> ** Nominated Mentors
> - Ben Hindman (benh at apache dot org) - Mesosphere
> - Danese Cooper (danese at apache dot org) - ebay, Inc.
> - Ted Dunning (tdunning at apache dot org) - MapR
>
> ** Sponsoring Entity
> Apache Incubator
>
Interesting, +1, If you guys need an extra mentor (or committer) please
count me in.
--
Luciano Resende
http://people.apache.org/~lresende
http://twitter.com/lresende1975
http://lresende.blogspot.com/