You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@myriad.apache.org by Haripriya Ayyalasomayajula <ah...@gmail.com> on 2015/09/24 21:21:09 UTC
Reg Basic Myriad functionality, referring to MesosCon 2015 talk slides
Hi,
My understanding was that the Yarn Resource manager will run alongside
Mesos Master (In practise I was starting a resource manager on a node with
Mesos master running and no mesos slave process) but I see that the RM is
running on a node which is a mesos slave - can anyone please clarify?
http://events.linuxfoundation.org/sites/events/files/slides/Apache_Myriad_MesosCon_2015.pdf
(slide 24)
--
Regards,
Haripriya Ayyalasomayajula
Re: Reg Basic Myriad functionality, referring to MesosCon 2015 talk slides
Posted by Adam Bordelon <ad...@mesosphere.io>.
Haripriya, if you're launching the RM manually, you might choose to launch
it on a Mesos Master (or a slave, or a non-Mesos node), but if you're using
a tool like Marathon to relaunch the RM if it (or its node) fails, then
you'll end up using Mesos to launch the RM itself as a Mesos task, so it
will run on a Mesos slave node that has sufficient resources to run the RM.
On Thu, Sep 24, 2015 at 12:21 PM, Haripriya Ayyalasomayajula <
aharipriya92@gmail.com> wrote:
> Hi,
>
> My understanding was that the Yarn Resource manager will run alongside
> Mesos Master (In practise I was starting a resource manager on a node with
> Mesos master running and no mesos slave process) but I see that the RM is
> running on a node which is a mesos slave - can anyone please clarify?
>
>
> http://events.linuxfoundation.org/sites/events/files/slides/Apache_Myriad_MesosCon_2015.pdf
> (slide 24)
>
> --
> Regards,
> Haripriya Ayyalasomayajula
>
Re: Reg Basic Myriad functionality, referring to MesosCon 2015 talk slides
Posted by Haripriya Ayyalasomayajula <ah...@gmail.com>.
Sorry sorry! In my earlier mail I was referring to Resource Manager.
Let me reformulate my question:
Launch RM using Marathon, it starts as a mesos slave task. In this case
what is the work flow? Does it still register as a framework with
Mesos-Master? is there any document that you can point me to - I'm trying
to understand the workflow better in this case?
On Thu, Sep 24, 2015 at 3:59 PM, Aashreya Shankar <as...@maprtech.com>
wrote:
> Hi,
>
> With Myriad we do not launch Node Manager using Marathon.
> We launch ResourceManager from Marathon.
> Now Myriad takes care of launching more NodeManagers on Mesos slaves as it
> gets flex up requests.
>
> Thank you
> Aashreya
>
>
> On Thu, Sep 24, 2015 at 3:54 PM, Haripriya Ayyalasomayajula <
> aharipriya92@gmail.com> wrote:
>
> > Thanks.
> >
> > So when we use marathon to launch node manager - it starts as a mesos
> slave
> > task. In this case how can it register as a framework with Mesos-Master?
> is
> > there any document that you can point me to - I'm trying to understand
> the
> > workflow better in this case?
> >
> > On Thu, Sep 24, 2015 at 12:56 PM, Swapnil Daingade <
> > swapnil.daingade@gmail.com> wrote:
> >
> > > RM is just another Mesos task if launched using something like
> Marathon.
> > RM
> > > in turn will register with Mesos as another framework.
> > > You will need mesos slave to launch a mesos task. The RM mesos task can
> > get
> > > launched on any node having a Mesos slave.
> > >
> > > You can technically launch a RM on any node which has the right
> binaries
> > > and config files from the cli
> > > but then resources consumed by RM will not be accounted for by Mesos.
> > Also
> > > you will not have access
> > > to features like HA.
> > >
> > > Regards
> > > Swapnil
> > >
> > >
> > >
> > > On Thu, Sep 24, 2015 at 12:21 PM, Haripriya Ayyalasomayajula <
> > > aharipriya92@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > My understanding was that the Yarn Resource manager will run
> alongside
> > > > Mesos Master (In practise I was starting a resource manager on a node
> > > with
> > > > Mesos master running and no mesos slave process) but I see that the
> RM
> > is
> > > > running on a node which is a mesos slave - can anyone please clarify?
> > > >
> > > >
> > > >
> > >
> >
> http://events.linuxfoundation.org/sites/events/files/slides/Apache_Myriad_MesosCon_2015.pdf
> > > > (slide 24)
> > > >
> > > > --
> > > > Regards,
> > > > Haripriya Ayyalasomayajula
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Haripriya Ayyalasomayajula
> >
>
--
Regards,
Haripriya Ayyalasomayajula
Re: Cassandra Summit
Posted by Ken Sipe <ke...@mesosphere.io>.
Thank you Ruth!! Love that you are helping with that!
Sent from my iPhone
> On Sep 25, 2015, at 11:23 AM, Ruth Harris <rh...@maprtech.com> wrote:
>
> Okee dokey!
>
> I'll read through and pull John and Ken's descriptions into the wiki.
>
> Thank John and Ken!!
>
> --ruth
>
>> On Fri, Sep 25, 2015 at 8:48 AM, Jim Klucar <kl...@gmail.com> wrote:
>>
>> That is a great reply, thanks. We should cut/paste it into the wiki.
>>
>>> On Fri, Sep 25, 2015 at 10:23 AM, Ken Sipe <ke...@mesosphere.io> wrote:
>>>
>>> that is my list as well… the bullet points
>>>
>>> * multi-versions of hadoop in the same cluster (we aren’t there yet)
>>> * scale down v1 of hadoop as you scale up v2 (completely different way of
>>> “decommissioning” services)
>>> * co-located services and data
>>> * multi tenant (manage hadoop, spark, kubernetes and other mesos services
>>> with 1 view into the resource / capacity utilization)
>>> * scale up yarn dynamically to utilities dc resources during off peak
>>> availability (imagine how awesome this will be after over provisioning is
>>> in place)
>>>
>>> Ken
>>>
>>>> On Sep 25, 2015, at 9:13 AM, John Omernik <jo...@omernik.com> wrote:
>>>>
>>>> "Why would you want to do that?"
>>>>
>>>> As a potential user of Myriad, in the enterprise I see a number of
>>> reasons
>>>> I'd "want to do that" they are:
>>>>
>>>> - The ability to use Mesos' purpose built and well design resource
>>>> management with Map Reduce. Right now Yarn is is the only option to run
>>> Map
>>>> Reduce V2 Applications, and while Yarn is far superior to Resource
>>>> Management in Map Reduce V1, we have still have an important
>> application
>>>> that is intrinsically tied to the resource schedule. Things that run on
>>>> resource schedulers should not be tied to them. Map Reduce V2 should
>> not
>>>> have a specific resource scheduler as a requirement.
>>>>
>>>> - Multi Tenancy: Right now if you have a cluster of computers, you can
>>> run
>>>> one Yarn cluster on them. With Myriad, the option exists to have
>> smaller
>>>> clusters, that are purpose built running on one set of harder, think a
>>> Yarn
>>>> cluster for marketing, or one for HR. This is great option for better
>>>> utilizing your resources, as well as better scaling growth and costs
>>>> associated with growth. Consider setting up separate clusters in Yarn
>>>> without Mesos: Many services duplicated, VMs or Physical node
>> management
>>>> issues, etc.
>>>>
>>>> - To build on Multi Tenancy, consider different version of Yarn and Map
>>>> Reduce. Right now, a new feature or bug fix comes out in a version of
>>> Yarn,
>>>> and there is not a good way to put that into play with your data. You
>>> have
>>>> to go through horrible testing process just to upgrade, and you have to
>>>> make sure ALL other jobs are not affected by the upgrade. With Myriad,
>>> keep
>>>> your production jobs at version X of yarn, and then spin up a new Yarn
>>>> cluster at version x+1. Now you can test your jobs slowly, and
>> migrated
>>>> them one by one without impact to production processes. Upgrading is
>> now
>>>> not all or nothing, but a controlled process where you can "fail fast"
>>> i.e.
>>>> if the job doesn't work, roll it back to the older version of Yarn.
>>>>
>>>> - The ability to have applications (think Docker containers) sitting
>>> right
>>>> next to the data (Hadoop data) they may be interacting with. Monitoring
>>> all
>>>> the jobs in one place rather than distinct clusters for containers and
>>>> others for data frameworks.
>>>>
>>>> - Data frameworks!! Like the multi-tenancy conversation, what happens
>>> when
>>>> you want to have Drill or Impala, plus Map Reduce V2 (multiple of
>> these),
>>>> plus Spark, or Storm, or Kafka all working together. With Yarn now,
>> you
>>>> it's much more locked in to a monolithic cluster, still with static
>>>> partitioning all over the place (think a Cloudera cluster with Yarn,
>>> Impala
>>>> and Hive... want to change something? You have to make sure all the
>>> pieces
>>>> change together) With Mesos/Myriad, you have the flexibility to move
>> and
>>>> try new things, with minimal impact to your production, without
>> standing
>>> up
>>>> addition servers/clusters. Myriad is the missing link here in that
>> YARN
>>>> only applications (Map ReduceV2!!!) are now part of that vision for a
>>>> unified data center, you no longer have to make a choice between Myriad
>>> or
>>>> Yarn, now it's Myriad AND Yarn.
>>>>
>>>> Those are the points that get me excited, ecosystem lock in a huge
>>> concern
>>>> for many enterprises. I don't want to imply I am not excited about
>> the
>>>> dynamic flexup/flexdown or the HA components, obviously those are
>> awesome
>>>> too, but for me those are cherries on top to the other components that
>>> let
>>>> me envision a data environment where options exist everywhere, where
>>>> innovation can happen faster, and I never have a situation where an
>> idea
>>> is
>>>> left on the cutting room floor because We don't support X.
>>>>
>>>> Random thoughts from me...
>>>>
>>>> John
>>>>
>>>>
>>>>
>>>>> On Fri, Sep 25, 2015 at 7:59 AM, Jim Klucar <kl...@gmail.com> wrote:
>>>>>
>>>>> Awesome. I assume it was good talk? I need to get better at answering
>>> the
>>>>> "Why would you want to do that?" question.
>>>>>
>>>>>> On Thu, Sep 24, 2015 at 9:08 PM, Ken Sipe <ke...@mesosphere.io> wrote:
>>>>>>
>>>>>> I just gave a talk at the cassandra summit. It included details
>> around
>>>>>> spark and analytics with cassandra in the cluster. There were lots
>> of
>>>>>> questions, etc. I just wanted to let this group know that the 2nd
>>>>> largest
>>>>>> topic of conversation and questions was around myriad… there was a
>> lot
>>> of
>>>>>> excitement for our project.
>>>>>>
>>>>>> Ken
>
>
>
> --
> Ruth Harris
> Sr. Technical Writer, MapR
Re: Cassandra Summit
Posted by Ruth Harris <rh...@maprtech.com>.
Okee dokey!
I'll read through and pull John and Ken's descriptions into the wiki.
Thank John and Ken!!
--ruth
On Fri, Sep 25, 2015 at 8:48 AM, Jim Klucar <kl...@gmail.com> wrote:
> That is a great reply, thanks. We should cut/paste it into the wiki.
>
> On Fri, Sep 25, 2015 at 10:23 AM, Ken Sipe <ke...@mesosphere.io> wrote:
>
> > that is my list as well… the bullet points
> >
> > * multi-versions of hadoop in the same cluster (we aren’t there yet)
> > * scale down v1 of hadoop as you scale up v2 (completely different way of
> > “decommissioning” services)
> > * co-located services and data
> > * multi tenant (manage hadoop, spark, kubernetes and other mesos services
> > with 1 view into the resource / capacity utilization)
> > * scale up yarn dynamically to utilities dc resources during off peak
> > availability (imagine how awesome this will be after over provisioning is
> > in place)
> >
> > Ken
> >
> > > On Sep 25, 2015, at 9:13 AM, John Omernik <jo...@omernik.com> wrote:
> > >
> > > "Why would you want to do that?"
> > >
> > > As a potential user of Myriad, in the enterprise I see a number of
> > reasons
> > > I'd "want to do that" they are:
> > >
> > > - The ability to use Mesos' purpose built and well design resource
> > > management with Map Reduce. Right now Yarn is is the only option to run
> > Map
> > > Reduce V2 Applications, and while Yarn is far superior to Resource
> > > Management in Map Reduce V1, we have still have an important
> application
> > > that is intrinsically tied to the resource schedule. Things that run on
> > > resource schedulers should not be tied to them. Map Reduce V2 should
> not
> > > have a specific resource scheduler as a requirement.
> > >
> > > - Multi Tenancy: Right now if you have a cluster of computers, you can
> > run
> > > one Yarn cluster on them. With Myriad, the option exists to have
> smaller
> > > clusters, that are purpose built running on one set of harder, think a
> > Yarn
> > > cluster for marketing, or one for HR. This is great option for better
> > > utilizing your resources, as well as better scaling growth and costs
> > > associated with growth. Consider setting up separate clusters in Yarn
> > > without Mesos: Many services duplicated, VMs or Physical node
> management
> > > issues, etc.
> > >
> > > - To build on Multi Tenancy, consider different version of Yarn and Map
> > > Reduce. Right now, a new feature or bug fix comes out in a version of
> > Yarn,
> > > and there is not a good way to put that into play with your data. You
> > have
> > > to go through horrible testing process just to upgrade, and you have to
> > > make sure ALL other jobs are not affected by the upgrade. With Myriad,
> > keep
> > > your production jobs at version X of yarn, and then spin up a new Yarn
> > > cluster at version x+1. Now you can test your jobs slowly, and
> migrated
> > > them one by one without impact to production processes. Upgrading is
> now
> > > not all or nothing, but a controlled process where you can "fail fast"
> > i.e.
> > > if the job doesn't work, roll it back to the older version of Yarn.
> > >
> > > - The ability to have applications (think Docker containers) sitting
> > right
> > > next to the data (Hadoop data) they may be interacting with. Monitoring
> > all
> > > the jobs in one place rather than distinct clusters for containers and
> > > others for data frameworks.
> > >
> > > - Data frameworks!! Like the multi-tenancy conversation, what happens
> > when
> > > you want to have Drill or Impala, plus Map Reduce V2 (multiple of
> these),
> > > plus Spark, or Storm, or Kafka all working together. With Yarn now,
> you
> > > it's much more locked in to a monolithic cluster, still with static
> > > partitioning all over the place (think a Cloudera cluster with Yarn,
> > Impala
> > > and Hive... want to change something? You have to make sure all the
> > pieces
> > > change together) With Mesos/Myriad, you have the flexibility to move
> and
> > > try new things, with minimal impact to your production, without
> standing
> > up
> > > addition servers/clusters. Myriad is the missing link here in that
> YARN
> > > only applications (Map ReduceV2!!!) are now part of that vision for a
> > > unified data center, you no longer have to make a choice between Myriad
> > or
> > > Yarn, now it's Myriad AND Yarn.
> > >
> > > Those are the points that get me excited, ecosystem lock in a huge
> > concern
> > > for many enterprises. I don't want to imply I am not excited about
> the
> > > dynamic flexup/flexdown or the HA components, obviously those are
> awesome
> > > too, but for me those are cherries on top to the other components that
> > let
> > > me envision a data environment where options exist everywhere, where
> > > innovation can happen faster, and I never have a situation where an
> idea
> > is
> > > left on the cutting room floor because We don't support X.
> > >
> > > Random thoughts from me...
> > >
> > > John
> > >
> > >
> > >
> > > On Fri, Sep 25, 2015 at 7:59 AM, Jim Klucar <kl...@gmail.com> wrote:
> > >
> > >> Awesome. I assume it was good talk? I need to get better at answering
> > the
> > >> "Why would you want to do that?" question.
> > >>
> > >> On Thu, Sep 24, 2015 at 9:08 PM, Ken Sipe <ke...@mesosphere.io> wrote:
> > >>
> > >>> I just gave a talk at the cassandra summit. It included details
> around
> > >>> spark and analytics with cassandra in the cluster. There were lots
> of
> > >>> questions, etc. I just wanted to let this group know that the 2nd
> > >> largest
> > >>> topic of conversation and questions was around myriad… there was a
> lot
> > of
> > >>> excitement for our project.
> > >>>
> > >>> Ken
> > >>
> >
> >
>
--
Ruth Harris
Sr. Technical Writer, MapR
Re: Cassandra Summit
Posted by Jim Klucar <kl...@gmail.com>.
That is a great reply, thanks. We should cut/paste it into the wiki.
On Fri, Sep 25, 2015 at 10:23 AM, Ken Sipe <ke...@mesosphere.io> wrote:
> that is my list as well… the bullet points
>
> * multi-versions of hadoop in the same cluster (we aren’t there yet)
> * scale down v1 of hadoop as you scale up v2 (completely different way of
> “decommissioning” services)
> * co-located services and data
> * multi tenant (manage hadoop, spark, kubernetes and other mesos services
> with 1 view into the resource / capacity utilization)
> * scale up yarn dynamically to utilities dc resources during off peak
> availability (imagine how awesome this will be after over provisioning is
> in place)
>
> Ken
>
> > On Sep 25, 2015, at 9:13 AM, John Omernik <jo...@omernik.com> wrote:
> >
> > "Why would you want to do that?"
> >
> > As a potential user of Myriad, in the enterprise I see a number of
> reasons
> > I'd "want to do that" they are:
> >
> > - The ability to use Mesos' purpose built and well design resource
> > management with Map Reduce. Right now Yarn is is the only option to run
> Map
> > Reduce V2 Applications, and while Yarn is far superior to Resource
> > Management in Map Reduce V1, we have still have an important application
> > that is intrinsically tied to the resource schedule. Things that run on
> > resource schedulers should not be tied to them. Map Reduce V2 should not
> > have a specific resource scheduler as a requirement.
> >
> > - Multi Tenancy: Right now if you have a cluster of computers, you can
> run
> > one Yarn cluster on them. With Myriad, the option exists to have smaller
> > clusters, that are purpose built running on one set of harder, think a
> Yarn
> > cluster for marketing, or one for HR. This is great option for better
> > utilizing your resources, as well as better scaling growth and costs
> > associated with growth. Consider setting up separate clusters in Yarn
> > without Mesos: Many services duplicated, VMs or Physical node management
> > issues, etc.
> >
> > - To build on Multi Tenancy, consider different version of Yarn and Map
> > Reduce. Right now, a new feature or bug fix comes out in a version of
> Yarn,
> > and there is not a good way to put that into play with your data. You
> have
> > to go through horrible testing process just to upgrade, and you have to
> > make sure ALL other jobs are not affected by the upgrade. With Myriad,
> keep
> > your production jobs at version X of yarn, and then spin up a new Yarn
> > cluster at version x+1. Now you can test your jobs slowly, and migrated
> > them one by one without impact to production processes. Upgrading is now
> > not all or nothing, but a controlled process where you can "fail fast"
> i.e.
> > if the job doesn't work, roll it back to the older version of Yarn.
> >
> > - The ability to have applications (think Docker containers) sitting
> right
> > next to the data (Hadoop data) they may be interacting with. Monitoring
> all
> > the jobs in one place rather than distinct clusters for containers and
> > others for data frameworks.
> >
> > - Data frameworks!! Like the multi-tenancy conversation, what happens
> when
> > you want to have Drill or Impala, plus Map Reduce V2 (multiple of these),
> > plus Spark, or Storm, or Kafka all working together. With Yarn now, you
> > it's much more locked in to a monolithic cluster, still with static
> > partitioning all over the place (think a Cloudera cluster with Yarn,
> Impala
> > and Hive... want to change something? You have to make sure all the
> pieces
> > change together) With Mesos/Myriad, you have the flexibility to move and
> > try new things, with minimal impact to your production, without standing
> up
> > addition servers/clusters. Myriad is the missing link here in that YARN
> > only applications (Map ReduceV2!!!) are now part of that vision for a
> > unified data center, you no longer have to make a choice between Myriad
> or
> > Yarn, now it's Myriad AND Yarn.
> >
> > Those are the points that get me excited, ecosystem lock in a huge
> concern
> > for many enterprises. I don't want to imply I am not excited about the
> > dynamic flexup/flexdown or the HA components, obviously those are awesome
> > too, but for me those are cherries on top to the other components that
> let
> > me envision a data environment where options exist everywhere, where
> > innovation can happen faster, and I never have a situation where an idea
> is
> > left on the cutting room floor because We don't support X.
> >
> > Random thoughts from me...
> >
> > John
> >
> >
> >
> > On Fri, Sep 25, 2015 at 7:59 AM, Jim Klucar <kl...@gmail.com> wrote:
> >
> >> Awesome. I assume it was good talk? I need to get better at answering
> the
> >> "Why would you want to do that?" question.
> >>
> >> On Thu, Sep 24, 2015 at 9:08 PM, Ken Sipe <ke...@mesosphere.io> wrote:
> >>
> >>> I just gave a talk at the cassandra summit. It included details around
> >>> spark and analytics with cassandra in the cluster. There were lots of
> >>> questions, etc. I just wanted to let this group know that the 2nd
> >> largest
> >>> topic of conversation and questions was around myriad… there was a lot
> of
> >>> excitement for our project.
> >>>
> >>> Ken
> >>
>
>
Re: Cassandra Summit
Posted by Ken Sipe <ke...@mesosphere.io>.
that is my list as well… the bullet points
* multi-versions of hadoop in the same cluster (we aren’t there yet)
* scale down v1 of hadoop as you scale up v2 (completely different way of “decommissioning” services)
* co-located services and data
* multi tenant (manage hadoop, spark, kubernetes and other mesos services with 1 view into the resource / capacity utilization)
* scale up yarn dynamically to utilities dc resources during off peak availability (imagine how awesome this will be after over provisioning is in place)
Ken
> On Sep 25, 2015, at 9:13 AM, John Omernik <jo...@omernik.com> wrote:
>
> "Why would you want to do that?"
>
> As a potential user of Myriad, in the enterprise I see a number of reasons
> I'd "want to do that" they are:
>
> - The ability to use Mesos' purpose built and well design resource
> management with Map Reduce. Right now Yarn is is the only option to run Map
> Reduce V2 Applications, and while Yarn is far superior to Resource
> Management in Map Reduce V1, we have still have an important application
> that is intrinsically tied to the resource schedule. Things that run on
> resource schedulers should not be tied to them. Map Reduce V2 should not
> have a specific resource scheduler as a requirement.
>
> - Multi Tenancy: Right now if you have a cluster of computers, you can run
> one Yarn cluster on them. With Myriad, the option exists to have smaller
> clusters, that are purpose built running on one set of harder, think a Yarn
> cluster for marketing, or one for HR. This is great option for better
> utilizing your resources, as well as better scaling growth and costs
> associated with growth. Consider setting up separate clusters in Yarn
> without Mesos: Many services duplicated, VMs or Physical node management
> issues, etc.
>
> - To build on Multi Tenancy, consider different version of Yarn and Map
> Reduce. Right now, a new feature or bug fix comes out in a version of Yarn,
> and there is not a good way to put that into play with your data. You have
> to go through horrible testing process just to upgrade, and you have to
> make sure ALL other jobs are not affected by the upgrade. With Myriad, keep
> your production jobs at version X of yarn, and then spin up a new Yarn
> cluster at version x+1. Now you can test your jobs slowly, and migrated
> them one by one without impact to production processes. Upgrading is now
> not all or nothing, but a controlled process where you can "fail fast" i.e.
> if the job doesn't work, roll it back to the older version of Yarn.
>
> - The ability to have applications (think Docker containers) sitting right
> next to the data (Hadoop data) they may be interacting with. Monitoring all
> the jobs in one place rather than distinct clusters for containers and
> others for data frameworks.
>
> - Data frameworks!! Like the multi-tenancy conversation, what happens when
> you want to have Drill or Impala, plus Map Reduce V2 (multiple of these),
> plus Spark, or Storm, or Kafka all working together. With Yarn now, you
> it's much more locked in to a monolithic cluster, still with static
> partitioning all over the place (think a Cloudera cluster with Yarn, Impala
> and Hive... want to change something? You have to make sure all the pieces
> change together) With Mesos/Myriad, you have the flexibility to move and
> try new things, with minimal impact to your production, without standing up
> addition servers/clusters. Myriad is the missing link here in that YARN
> only applications (Map ReduceV2!!!) are now part of that vision for a
> unified data center, you no longer have to make a choice between Myriad or
> Yarn, now it's Myriad AND Yarn.
>
> Those are the points that get me excited, ecosystem lock in a huge concern
> for many enterprises. I don't want to imply I am not excited about the
> dynamic flexup/flexdown or the HA components, obviously those are awesome
> too, but for me those are cherries on top to the other components that let
> me envision a data environment where options exist everywhere, where
> innovation can happen faster, and I never have a situation where an idea is
> left on the cutting room floor because We don't support X.
>
> Random thoughts from me...
>
> John
>
>
>
> On Fri, Sep 25, 2015 at 7:59 AM, Jim Klucar <kl...@gmail.com> wrote:
>
>> Awesome. I assume it was good talk? I need to get better at answering the
>> "Why would you want to do that?" question.
>>
>> On Thu, Sep 24, 2015 at 9:08 PM, Ken Sipe <ke...@mesosphere.io> wrote:
>>
>>> I just gave a talk at the cassandra summit. It included details around
>>> spark and analytics with cassandra in the cluster. There were lots of
>>> questions, etc. I just wanted to let this group know that the 2nd
>> largest
>>> topic of conversation and questions was around myriad… there was a lot of
>>> excitement for our project.
>>>
>>> Ken
>>
Re: Cassandra Summit
Posted by John Omernik <jo...@omernik.com>.
"Why would you want to do that?"
As a potential user of Myriad, in the enterprise I see a number of reasons
I'd "want to do that" they are:
- The ability to use Mesos' purpose built and well design resource
management with Map Reduce. Right now Yarn is is the only option to run Map
Reduce V2 Applications, and while Yarn is far superior to Resource
Management in Map Reduce V1, we have still have an important application
that is intrinsically tied to the resource schedule. Things that run on
resource schedulers should not be tied to them. Map Reduce V2 should not
have a specific resource scheduler as a requirement.
- Multi Tenancy: Right now if you have a cluster of computers, you can run
one Yarn cluster on them. With Myriad, the option exists to have smaller
clusters, that are purpose built running on one set of harder, think a Yarn
cluster for marketing, or one for HR. This is great option for better
utilizing your resources, as well as better scaling growth and costs
associated with growth. Consider setting up separate clusters in Yarn
without Mesos: Many services duplicated, VMs or Physical node management
issues, etc.
- To build on Multi Tenancy, consider different version of Yarn and Map
Reduce. Right now, a new feature or bug fix comes out in a version of Yarn,
and there is not a good way to put that into play with your data. You have
to go through horrible testing process just to upgrade, and you have to
make sure ALL other jobs are not affected by the upgrade. With Myriad, keep
your production jobs at version X of yarn, and then spin up a new Yarn
cluster at version x+1. Now you can test your jobs slowly, and migrated
them one by one without impact to production processes. Upgrading is now
not all or nothing, but a controlled process where you can "fail fast" i.e.
if the job doesn't work, roll it back to the older version of Yarn.
- The ability to have applications (think Docker containers) sitting right
next to the data (Hadoop data) they may be interacting with. Monitoring all
the jobs in one place rather than distinct clusters for containers and
others for data frameworks.
- Data frameworks!! Like the multi-tenancy conversation, what happens when
you want to have Drill or Impala, plus Map Reduce V2 (multiple of these),
plus Spark, or Storm, or Kafka all working together. With Yarn now, you
it's much more locked in to a monolithic cluster, still with static
partitioning all over the place (think a Cloudera cluster with Yarn, Impala
and Hive... want to change something? You have to make sure all the pieces
change together) With Mesos/Myriad, you have the flexibility to move and
try new things, with minimal impact to your production, without standing up
addition servers/clusters. Myriad is the missing link here in that YARN
only applications (Map ReduceV2!!!) are now part of that vision for a
unified data center, you no longer have to make a choice between Myriad or
Yarn, now it's Myriad AND Yarn.
Those are the points that get me excited, ecosystem lock in a huge concern
for many enterprises. I don't want to imply I am not excited about the
dynamic flexup/flexdown or the HA components, obviously those are awesome
too, but for me those are cherries on top to the other components that let
me envision a data environment where options exist everywhere, where
innovation can happen faster, and I never have a situation where an idea is
left on the cutting room floor because We don't support X.
Random thoughts from me...
John
On Fri, Sep 25, 2015 at 7:59 AM, Jim Klucar <kl...@gmail.com> wrote:
> Awesome. I assume it was good talk? I need to get better at answering the
> "Why would you want to do that?" question.
>
> On Thu, Sep 24, 2015 at 9:08 PM, Ken Sipe <ke...@mesosphere.io> wrote:
>
> > I just gave a talk at the cassandra summit. It included details around
> > spark and analytics with cassandra in the cluster. There were lots of
> > questions, etc. I just wanted to let this group know that the 2nd
> largest
> > topic of conversation and questions was around myriad… there was a lot of
> > excitement for our project.
> >
> > Ken
>
Re: Cassandra Summit
Posted by Jim Klucar <kl...@gmail.com>.
Awesome. I assume it was good talk? I need to get better at answering the
"Why would you want to do that?" question.
On Thu, Sep 24, 2015 at 9:08 PM, Ken Sipe <ke...@mesosphere.io> wrote:
> I just gave a talk at the cassandra summit. It included details around
> spark and analytics with cassandra in the cluster. There were lots of
> questions, etc. I just wanted to let this group know that the 2nd largest
> topic of conversation and questions was around myriad… there was a lot of
> excitement for our project.
>
> Ken
Cassandra Summit
Posted by Ken Sipe <ke...@mesosphere.io>.
I just gave a talk at the cassandra summit. It included details around spark and analytics with cassandra in the cluster. There were lots of questions, etc. I just wanted to let this group know that the 2nd largest topic of conversation and questions was around myriad… there was a lot of excitement for our project.
Ken
Re: Reg Basic Myriad functionality, referring to MesosCon 2015 talk slides
Posted by Haripriya Ayyalasomayajula <ah...@gmail.com>.
Interesting. Thanks Adam.
On Thu, Sep 24, 2015 at 4:17 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
> Haripriya, the RM (and Myriad scheduler) run as a Mesos task, and as a part
> of that task, the Myriad scheduler sends a RegisterFrameworkMessage to the
> Mesos master, asking Mesos to register a new Myriad framework and
> communicate with this scheduler process for resource offers and status
> updates.
>
> So you ask Marathon to launch an app with 1 instance of the
> RM/MyriadScheduler, and then that runs as a Mesos/Marathon task, and it
> registers with Mesos as a new framework scheduler, after which point Mesos
> starts making resource offers to the MyriadScheduler (as well as Marathon),
> and the MyriadScheduler uses its offers to launch NMs.
>
> The slightly tricky/confusing part is that the Myriad scheduler process'
> resources are accounted for under the Marathon framework that launched it,
> not as a part of the Myriad framework itself. Only tasks launched by the
> Myriad scheduler (i.e. NodeManagers) are accounted for under the Myriad
> framework's share of resources.
>
> On Thu, Sep 24, 2015 at 3:59 PM, Aashreya Shankar <as...@maprtech.com>
> wrote:
>
> > Hi,
> >
> > With Myriad we do not launch Node Manager using Marathon.
> > We launch ResourceManager from Marathon.
> > Now Myriad takes care of launching more NodeManagers on Mesos slaves as
> it
> > gets flex up requests.
> >
> > Thank you
> > Aashreya
> >
> >
> > On Thu, Sep 24, 2015 at 3:54 PM, Haripriya Ayyalasomayajula <
> > aharipriya92@gmail.com> wrote:
> >
> > > Thanks.
> > >
> > > So when we use marathon to launch node manager - it starts as a mesos
> > slave
> > > task. In this case how can it register as a framework with
> Mesos-Master?
> > is
> > > there any document that you can point me to - I'm trying to understand
> > the
> > > workflow better in this case?
> > >
> > > On Thu, Sep 24, 2015 at 12:56 PM, Swapnil Daingade <
> > > swapnil.daingade@gmail.com> wrote:
> > >
> > > > RM is just another Mesos task if launched using something like
> > Marathon.
> > > RM
> > > > in turn will register with Mesos as another framework.
> > > > You will need mesos slave to launch a mesos task. The RM mesos task
> can
> > > get
> > > > launched on any node having a Mesos slave.
> > > >
> > > > You can technically launch a RM on any node which has the right
> > binaries
> > > > and config files from the cli
> > > > but then resources consumed by RM will not be accounted for by Mesos.
> > > Also
> > > > you will not have access
> > > > to features like HA.
> > > >
> > > > Regards
> > > > Swapnil
> > > >
> > > >
> > > >
> > > > On Thu, Sep 24, 2015 at 12:21 PM, Haripriya Ayyalasomayajula <
> > > > aharipriya92@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > My understanding was that the Yarn Resource manager will run
> > alongside
> > > > > Mesos Master (In practise I was starting a resource manager on a
> node
> > > > with
> > > > > Mesos master running and no mesos slave process) but I see that the
> > RM
> > > is
> > > > > running on a node which is a mesos slave - can anyone please
> clarify?
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> http://events.linuxfoundation.org/sites/events/files/slides/Apache_Myriad_MesosCon_2015.pdf
> > > > > (slide 24)
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Haripriya Ayyalasomayajula
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Haripriya Ayyalasomayajula
> > >
> >
>
--
Regards,
Haripriya Ayyalasomayajula
Re: Reg Basic Myriad functionality, referring to MesosCon 2015 talk slides
Posted by Adam Bordelon <ad...@mesosphere.io>.
Haripriya, the RM (and Myriad scheduler) run as a Mesos task, and as a part
of that task, the Myriad scheduler sends a RegisterFrameworkMessage to the
Mesos master, asking Mesos to register a new Myriad framework and
communicate with this scheduler process for resource offers and status
updates.
So you ask Marathon to launch an app with 1 instance of the
RM/MyriadScheduler, and then that runs as a Mesos/Marathon task, and it
registers with Mesos as a new framework scheduler, after which point Mesos
starts making resource offers to the MyriadScheduler (as well as Marathon),
and the MyriadScheduler uses its offers to launch NMs.
The slightly tricky/confusing part is that the Myriad scheduler process'
resources are accounted for under the Marathon framework that launched it,
not as a part of the Myriad framework itself. Only tasks launched by the
Myriad scheduler (i.e. NodeManagers) are accounted for under the Myriad
framework's share of resources.
On Thu, Sep 24, 2015 at 3:59 PM, Aashreya Shankar <as...@maprtech.com>
wrote:
> Hi,
>
> With Myriad we do not launch Node Manager using Marathon.
> We launch ResourceManager from Marathon.
> Now Myriad takes care of launching more NodeManagers on Mesos slaves as it
> gets flex up requests.
>
> Thank you
> Aashreya
>
>
> On Thu, Sep 24, 2015 at 3:54 PM, Haripriya Ayyalasomayajula <
> aharipriya92@gmail.com> wrote:
>
> > Thanks.
> >
> > So when we use marathon to launch node manager - it starts as a mesos
> slave
> > task. In this case how can it register as a framework with Mesos-Master?
> is
> > there any document that you can point me to - I'm trying to understand
> the
> > workflow better in this case?
> >
> > On Thu, Sep 24, 2015 at 12:56 PM, Swapnil Daingade <
> > swapnil.daingade@gmail.com> wrote:
> >
> > > RM is just another Mesos task if launched using something like
> Marathon.
> > RM
> > > in turn will register with Mesos as another framework.
> > > You will need mesos slave to launch a mesos task. The RM mesos task can
> > get
> > > launched on any node having a Mesos slave.
> > >
> > > You can technically launch a RM on any node which has the right
> binaries
> > > and config files from the cli
> > > but then resources consumed by RM will not be accounted for by Mesos.
> > Also
> > > you will not have access
> > > to features like HA.
> > >
> > > Regards
> > > Swapnil
> > >
> > >
> > >
> > > On Thu, Sep 24, 2015 at 12:21 PM, Haripriya Ayyalasomayajula <
> > > aharipriya92@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > My understanding was that the Yarn Resource manager will run
> alongside
> > > > Mesos Master (In practise I was starting a resource manager on a node
> > > with
> > > > Mesos master running and no mesos slave process) but I see that the
> RM
> > is
> > > > running on a node which is a mesos slave - can anyone please clarify?
> > > >
> > > >
> > > >
> > >
> >
> http://events.linuxfoundation.org/sites/events/files/slides/Apache_Myriad_MesosCon_2015.pdf
> > > > (slide 24)
> > > >
> > > > --
> > > > Regards,
> > > > Haripriya Ayyalasomayajula
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Haripriya Ayyalasomayajula
> >
>
Re: Reg Basic Myriad functionality, referring to MesosCon 2015 talk slides
Posted by Aashreya Shankar <as...@maprtech.com>.
Hi,
With Myriad we do not launch Node Manager using Marathon.
We launch ResourceManager from Marathon.
Now Myriad takes care of launching more NodeManagers on Mesos slaves as it
gets flex up requests.
Thank you
Aashreya
On Thu, Sep 24, 2015 at 3:54 PM, Haripriya Ayyalasomayajula <
aharipriya92@gmail.com> wrote:
> Thanks.
>
> So when we use marathon to launch node manager - it starts as a mesos slave
> task. In this case how can it register as a framework with Mesos-Master? is
> there any document that you can point me to - I'm trying to understand the
> workflow better in this case?
>
> On Thu, Sep 24, 2015 at 12:56 PM, Swapnil Daingade <
> swapnil.daingade@gmail.com> wrote:
>
> > RM is just another Mesos task if launched using something like Marathon.
> RM
> > in turn will register with Mesos as another framework.
> > You will need mesos slave to launch a mesos task. The RM mesos task can
> get
> > launched on any node having a Mesos slave.
> >
> > You can technically launch a RM on any node which has the right binaries
> > and config files from the cli
> > but then resources consumed by RM will not be accounted for by Mesos.
> Also
> > you will not have access
> > to features like HA.
> >
> > Regards
> > Swapnil
> >
> >
> >
> > On Thu, Sep 24, 2015 at 12:21 PM, Haripriya Ayyalasomayajula <
> > aharipriya92@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > My understanding was that the Yarn Resource manager will run alongside
> > > Mesos Master (In practise I was starting a resource manager on a node
> > with
> > > Mesos master running and no mesos slave process) but I see that the RM
> is
> > > running on a node which is a mesos slave - can anyone please clarify?
> > >
> > >
> > >
> >
> http://events.linuxfoundation.org/sites/events/files/slides/Apache_Myriad_MesosCon_2015.pdf
> > > (slide 24)
> > >
> > > --
> > > Regards,
> > > Haripriya Ayyalasomayajula
> > >
> >
>
>
>
> --
> Regards,
> Haripriya Ayyalasomayajula
>
Re: Reg Basic Myriad functionality, referring to MesosCon 2015 talk slides
Posted by Haripriya Ayyalasomayajula <ah...@gmail.com>.
Thanks.
So when we use marathon to launch node manager - it starts as a mesos slave
task. In this case how can it register as a framework with Mesos-Master? is
there any document that you can point me to - I'm trying to understand the
workflow better in this case?
On Thu, Sep 24, 2015 at 12:56 PM, Swapnil Daingade <
swapnil.daingade@gmail.com> wrote:
> RM is just another Mesos task if launched using something like Marathon. RM
> in turn will register with Mesos as another framework.
> You will need mesos slave to launch a mesos task. The RM mesos task can get
> launched on any node having a Mesos slave.
>
> You can technically launch a RM on any node which has the right binaries
> and config files from the cli
> but then resources consumed by RM will not be accounted for by Mesos. Also
> you will not have access
> to features like HA.
>
> Regards
> Swapnil
>
>
>
> On Thu, Sep 24, 2015 at 12:21 PM, Haripriya Ayyalasomayajula <
> aharipriya92@gmail.com> wrote:
>
> > Hi,
> >
> > My understanding was that the Yarn Resource manager will run alongside
> > Mesos Master (In practise I was starting a resource manager on a node
> with
> > Mesos master running and no mesos slave process) but I see that the RM is
> > running on a node which is a mesos slave - can anyone please clarify?
> >
> >
> >
> http://events.linuxfoundation.org/sites/events/files/slides/Apache_Myriad_MesosCon_2015.pdf
> > (slide 24)
> >
> > --
> > Regards,
> > Haripriya Ayyalasomayajula
> >
>
--
Regards,
Haripriya Ayyalasomayajula
Re: Reg Basic Myriad functionality, referring to MesosCon 2015 talk slides
Posted by Swapnil Daingade <sw...@gmail.com>.
RM is just another Mesos task if launched using something like Marathon. RM
in turn will register with Mesos as another framework.
You will need mesos slave to launch a mesos task. The RM mesos task can get
launched on any node having a Mesos slave.
You can technically launch a RM on any node which has the right binaries
and config files from the cli
but then resources consumed by RM will not be accounted for by Mesos. Also
you will not have access
to features like HA.
Regards
Swapnil
On Thu, Sep 24, 2015 at 12:21 PM, Haripriya Ayyalasomayajula <
aharipriya92@gmail.com> wrote:
> Hi,
>
> My understanding was that the Yarn Resource manager will run alongside
> Mesos Master (In practise I was starting a resource manager on a node with
> Mesos master running and no mesos slave process) but I see that the RM is
> running on a node which is a mesos slave - can anyone please clarify?
>
>
> http://events.linuxfoundation.org/sites/events/files/slides/Apache_Myriad_MesosCon_2015.pdf
> (slide 24)
>
> --
> Regards,
> Haripriya Ayyalasomayajula
>