You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by "hsy541@gmail.com" <hs...@gmail.com> on 2014/08/04 20:37:32 UTC
Is there a way to run application on certain subset of nodes?
Hi guys,
I'm new to slider and try to convert some application into yarn app. I
would like to ask is there a way to specify only a subset of nodes in the
cluster to run my app and can slider guarantee every container(of that
application) run on different nodes?
Thank you very much!
Best,
Siyuan
Re: Is there a way to run application on certain subset of nodes?
Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Thanks Sumit!
Best,
Siyuan
On Mon, Aug 4, 2014 at 2:16 PM, Sumit Mohanty <sm...@hortonworks.com>
wrote:
> Its not so much as you cannot run but multiple instance on same host do not
> have much advantage unless its a dev/test cluster and being used from a
> functional testing standpoint. This is what we also do for functional
> testing where multiple slave nodes are hosted on a single host.
>
> So if you plan to have one region server instance per host then it makes
> sense to have the region server instance take as much of the available
> memory as it can - hence we can make sure that the container can consume
> most (not just > 50%) of the available memory.
>
> One anti-affinity is available or Slider adds such a feature itself, you
> need not do use the work-around.
>
>
> On Mon, Aug 4, 2014 at 2:04 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
>
> > Thanks Sumit, Good to know this!
> > I'm curious how does slider do to launch hbase on yarn. What I understand
> > is you can not run region server on the same node.
> > So if you want to run hbase on yarn by slider, you have to specify more
> > than half of the available resource per host?
> >
> > Best,
> > Siyuan
> >
> >
> >
> > On Mon, Aug 4, 2014 at 1:56 PM, Sumit Mohanty <sm...@hortonworks.com>
> > wrote:
> >
> > > Thats another feature called anti-affinity, from YARN, to enable
> > container
> > > allocation on different nodes for same container type. This feature is
> > not
> > > yet available and in the mean time you can enforce it by asking more
> than
> > > half of available resource (e.g. memory) per host.
> > >
> > > Depending on the timeline of the availability of anti-affinity feature
> > from
> > > YARN, Slider can also add support for such a feature but the desirable
> > path
> > > is to get this through YARN.
> > >
> > >
> > > On Mon, Aug 4, 2014 at 1:48 PM, hsy541@gmail.com <hs...@gmail.com>
> > wrote:
> > >
> > > > Thanks guys, so my next question is how can you guarantee containers
> > are
> > > > allocated on different nodes?
> > > >
> > > >
> > > > Best,
> > > > Siyuan
> > > >
> > > >
> > > > On Mon, Aug 4, 2014 at 1:02 PM, Sumit Mohanty <
> > smohanty@hortonworks.com>
> > > > wrote:
> > > >
> > > > > +1 to Joe's explanation.
> > > > >
> > > > > Slider plans to use the "label" support enabled through YARN-796
> and
> > > > allow
> > > > > creation of app instances against a label. This way you can ensure
> > that
> > > > the
> > > > > app instance has containers allocated only on a subset of the
> nodes.
> > > > >
> > > > >
> > > > > On Mon, Aug 4, 2014 at 12:15 PM, Joseph Niemiec <
> > > > jniemiec@hortonworks.com>
> > > > > wrote:
> > > > >
> > > > > > I'll let someone else chime in here but my assumption is this
> will
> > > > > require
> > > > > > YARN-796 and then have Slider expose the config somehow which
> from
> > my
> > > > > > exploration has yet to be done.
> > > > > >
> > > > > > Outside of Slider if you wrote your own AM you can control this
> > today
> > > > in
> > > > > > your resourcerequest to the RM.
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/YARN-796
> > > > > >
> > > > > >
> > > > > > On Mon, Aug 4, 2014 at 2:37 PM, hsy541@gmail.com <
> hsy541@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi guys,
> > > > > > >
> > > > > > > I'm new to slider and try to convert some application into yarn
> > > app.
> > > > I
> > > > > > > would like to ask is there a way to specify only a subset of
> > nodes
> > > in
> > > > > the
> > > > > > > cluster to run my app and can slider guarantee every
> container(of
> > > > that
> > > > > > > application) run on different nodes?
> > > > > > >
> > > > > > > Thank you very much!
> > > > > > >
> > > > > > > Best,
> > > > > > > Siyuan
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Hortonworks Certified Hadoop 2 Java Developer
> > > > > >
> > > > > > Solutions Engineer, Hortonworks
> > > > > > Phone: Office (734)-288-3930 Mobile (734)-301-9812
> > > > > > Email: jniemiec@hortonworks.com
> > > > > >
> > > > > > Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing
> > with
> > > > > > Apache Hadoop 2
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.amazon.com/Apache-Hadoop-YARN-Processing-Addison-Wesley/dp/0321934504
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > CONFIDENTIALITY NOTICE
> > > > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > > > to
> > > > > > which it is addressed and may contain information that is
> > > confidential,
> > > > > > privileged and exempt from disclosure under applicable law. If
> the
> > > > reader
> > > > > > of this message is not the intended recipient, you are hereby
> > > notified
> > > > > that
> > > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > > forwarding of this communication is strictly prohibited. If you
> > have
> > > > > > received this communication in error, please contact the sender
> > > > > immediately
> > > > > > and delete it from your system. Thank You.
> > > > > >
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Is there a way to run application on certain subset of nodes?
Posted by Sumit Mohanty <sm...@hortonworks.com>.
Its not so much as you cannot run but multiple instance on same host do not
have much advantage unless its a dev/test cluster and being used from a
functional testing standpoint. This is what we also do for functional
testing where multiple slave nodes are hosted on a single host.
So if you plan to have one region server instance per host then it makes
sense to have the region server instance take as much of the available
memory as it can - hence we can make sure that the container can consume
most (not just > 50%) of the available memory.
One anti-affinity is available or Slider adds such a feature itself, you
need not do use the work-around.
On Mon, Aug 4, 2014 at 2:04 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
> Thanks Sumit, Good to know this!
> I'm curious how does slider do to launch hbase on yarn. What I understand
> is you can not run region server on the same node.
> So if you want to run hbase on yarn by slider, you have to specify more
> than half of the available resource per host?
>
> Best,
> Siyuan
>
>
>
> On Mon, Aug 4, 2014 at 1:56 PM, Sumit Mohanty <sm...@hortonworks.com>
> wrote:
>
> > Thats another feature called anti-affinity, from YARN, to enable
> container
> > allocation on different nodes for same container type. This feature is
> not
> > yet available and in the mean time you can enforce it by asking more than
> > half of available resource (e.g. memory) per host.
> >
> > Depending on the timeline of the availability of anti-affinity feature
> from
> > YARN, Slider can also add support for such a feature but the desirable
> path
> > is to get this through YARN.
> >
> >
> > On Mon, Aug 4, 2014 at 1:48 PM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
> >
> > > Thanks guys, so my next question is how can you guarantee containers
> are
> > > allocated on different nodes?
> > >
> > >
> > > Best,
> > > Siyuan
> > >
> > >
> > > On Mon, Aug 4, 2014 at 1:02 PM, Sumit Mohanty <
> smohanty@hortonworks.com>
> > > wrote:
> > >
> > > > +1 to Joe's explanation.
> > > >
> > > > Slider plans to use the "label" support enabled through YARN-796 and
> > > allow
> > > > creation of app instances against a label. This way you can ensure
> that
> > > the
> > > > app instance has containers allocated only on a subset of the nodes.
> > > >
> > > >
> > > > On Mon, Aug 4, 2014 at 12:15 PM, Joseph Niemiec <
> > > jniemiec@hortonworks.com>
> > > > wrote:
> > > >
> > > > > I'll let someone else chime in here but my assumption is this will
> > > > require
> > > > > YARN-796 and then have Slider expose the config somehow which from
> my
> > > > > exploration has yet to be done.
> > > > >
> > > > > Outside of Slider if you wrote your own AM you can control this
> today
> > > in
> > > > > your resourcerequest to the RM.
> > > > >
> > > > > https://issues.apache.org/jira/browse/YARN-796
> > > > >
> > > > >
> > > > > On Mon, Aug 4, 2014 at 2:37 PM, hsy541@gmail.com <hsy541@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > > I'm new to slider and try to convert some application into yarn
> > app.
> > > I
> > > > > > would like to ask is there a way to specify only a subset of
> nodes
> > in
> > > > the
> > > > > > cluster to run my app and can slider guarantee every container(of
> > > that
> > > > > > application) run on different nodes?
> > > > > >
> > > > > > Thank you very much!
> > > > > >
> > > > > > Best,
> > > > > > Siyuan
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Hortonworks Certified Hadoop 2 Java Developer
> > > > >
> > > > > Solutions Engineer, Hortonworks
> > > > > Phone: Office (734)-288-3930 Mobile (734)-301-9812
> > > > > Email: jniemiec@hortonworks.com
> > > > >
> > > > > Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing
> with
> > > > > Apache Hadoop 2
> > > > > <
> > > > >
> > > >
> > >
> >
> http://www.amazon.com/Apache-Hadoop-YARN-Processing-Addison-Wesley/dp/0321934504
> > > > > >
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Is there a way to run application on certain subset of nodes?
Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Thanks Sumit, Good to know this!
I'm curious how does slider do to launch hbase on yarn. What I understand
is you can not run region server on the same node.
So if you want to run hbase on yarn by slider, you have to specify more
than half of the available resource per host?
Best,
Siyuan
On Mon, Aug 4, 2014 at 1:56 PM, Sumit Mohanty <sm...@hortonworks.com>
wrote:
> Thats another feature called anti-affinity, from YARN, to enable container
> allocation on different nodes for same container type. This feature is not
> yet available and in the mean time you can enforce it by asking more than
> half of available resource (e.g. memory) per host.
>
> Depending on the timeline of the availability of anti-affinity feature from
> YARN, Slider can also add support for such a feature but the desirable path
> is to get this through YARN.
>
>
> On Mon, Aug 4, 2014 at 1:48 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
>
> > Thanks guys, so my next question is how can you guarantee containers are
> > allocated on different nodes?
> >
> >
> > Best,
> > Siyuan
> >
> >
> > On Mon, Aug 4, 2014 at 1:02 PM, Sumit Mohanty <sm...@hortonworks.com>
> > wrote:
> >
> > > +1 to Joe's explanation.
> > >
> > > Slider plans to use the "label" support enabled through YARN-796 and
> > allow
> > > creation of app instances against a label. This way you can ensure that
> > the
> > > app instance has containers allocated only on a subset of the nodes.
> > >
> > >
> > > On Mon, Aug 4, 2014 at 12:15 PM, Joseph Niemiec <
> > jniemiec@hortonworks.com>
> > > wrote:
> > >
> > > > I'll let someone else chime in here but my assumption is this will
> > > require
> > > > YARN-796 and then have Slider expose the config somehow which from my
> > > > exploration has yet to be done.
> > > >
> > > > Outside of Slider if you wrote your own AM you can control this today
> > in
> > > > your resourcerequest to the RM.
> > > >
> > > > https://issues.apache.org/jira/browse/YARN-796
> > > >
> > > >
> > > > On Mon, Aug 4, 2014 at 2:37 PM, hsy541@gmail.com <hs...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi guys,
> > > > >
> > > > > I'm new to slider and try to convert some application into yarn
> app.
> > I
> > > > > would like to ask is there a way to specify only a subset of nodes
> in
> > > the
> > > > > cluster to run my app and can slider guarantee every container(of
> > that
> > > > > application) run on different nodes?
> > > > >
> > > > > Thank you very much!
> > > > >
> > > > > Best,
> > > > > Siyuan
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Hortonworks Certified Hadoop 2 Java Developer
> > > >
> > > > Solutions Engineer, Hortonworks
> > > > Phone: Office (734)-288-3930 Mobile (734)-301-9812
> > > > Email: jniemiec@hortonworks.com
> > > >
> > > > Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with
> > > > Apache Hadoop 2
> > > > <
> > > >
> > >
> >
> http://www.amazon.com/Apache-Hadoop-YARN-Processing-Addison-Wesley/dp/0321934504
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Is there a way to run application on certain subset of nodes?
Posted by Sumit Mohanty <sm...@hortonworks.com>.
Thats another feature called anti-affinity, from YARN, to enable container
allocation on different nodes for same container type. This feature is not
yet available and in the mean time you can enforce it by asking more than
half of available resource (e.g. memory) per host.
Depending on the timeline of the availability of anti-affinity feature from
YARN, Slider can also add support for such a feature but the desirable path
is to get this through YARN.
On Mon, Aug 4, 2014 at 1:48 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
> Thanks guys, so my next question is how can you guarantee containers are
> allocated on different nodes?
>
>
> Best,
> Siyuan
>
>
> On Mon, Aug 4, 2014 at 1:02 PM, Sumit Mohanty <sm...@hortonworks.com>
> wrote:
>
> > +1 to Joe's explanation.
> >
> > Slider plans to use the "label" support enabled through YARN-796 and
> allow
> > creation of app instances against a label. This way you can ensure that
> the
> > app instance has containers allocated only on a subset of the nodes.
> >
> >
> > On Mon, Aug 4, 2014 at 12:15 PM, Joseph Niemiec <
> jniemiec@hortonworks.com>
> > wrote:
> >
> > > I'll let someone else chime in here but my assumption is this will
> > require
> > > YARN-796 and then have Slider expose the config somehow which from my
> > > exploration has yet to be done.
> > >
> > > Outside of Slider if you wrote your own AM you can control this today
> in
> > > your resourcerequest to the RM.
> > >
> > > https://issues.apache.org/jira/browse/YARN-796
> > >
> > >
> > > On Mon, Aug 4, 2014 at 2:37 PM, hsy541@gmail.com <hs...@gmail.com>
> > wrote:
> > >
> > > > Hi guys,
> > > >
> > > > I'm new to slider and try to convert some application into yarn app.
> I
> > > > would like to ask is there a way to specify only a subset of nodes in
> > the
> > > > cluster to run my app and can slider guarantee every container(of
> that
> > > > application) run on different nodes?
> > > >
> > > > Thank you very much!
> > > >
> > > > Best,
> > > > Siyuan
> > > >
> > >
> > >
> > >
> > > --
> > > Hortonworks Certified Hadoop 2 Java Developer
> > >
> > > Solutions Engineer, Hortonworks
> > > Phone: Office (734)-288-3930 Mobile (734)-301-9812
> > > Email: jniemiec@hortonworks.com
> > >
> > > Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with
> > > Apache Hadoop 2
> > > <
> > >
> >
> http://www.amazon.com/Apache-Hadoop-YARN-Processing-Addison-Wesley/dp/0321934504
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Is there a way to run application on certain subset of nodes?
Posted by Joseph Niemiec <jn...@hortonworks.com>.
>From a YARN perspective you don't, its up to the Application Master to do
so and as we have discussed that is going to require A) Native YARN App, or
B) YARN-796 with Slider integration. As your deploying a Slider app its
really just a YARN container and as such can be provisioned anywhere that
YARN has the resources to deploy this application.
What is the application your making the requires its services to only have
a single instance per node?
On Mon, Aug 4, 2014 at 4:48 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
> Thanks guys, so my next question is how can you guarantee containers are
> allocated on different nodes?
>
>
> Best,
> Siyuan
>
>
> On Mon, Aug 4, 2014 at 1:02 PM, Sumit Mohanty <sm...@hortonworks.com>
> wrote:
>
> > +1 to Joe's explanation.
> >
> > Slider plans to use the "label" support enabled through YARN-796 and
> allow
> > creation of app instances against a label. This way you can ensure that
> the
> > app instance has containers allocated only on a subset of the nodes.
> >
> >
> > On Mon, Aug 4, 2014 at 12:15 PM, Joseph Niemiec <
> jniemiec@hortonworks.com>
> > wrote:
> >
> > > I'll let someone else chime in here but my assumption is this will
> > require
> > > YARN-796 and then have Slider expose the config somehow which from my
> > > exploration has yet to be done.
> > >
> > > Outside of Slider if you wrote your own AM you can control this today
> in
> > > your resourcerequest to the RM.
> > >
> > > https://issues.apache.org/jira/browse/YARN-796
> > >
> > >
> > > On Mon, Aug 4, 2014 at 2:37 PM, hsy541@gmail.com <hs...@gmail.com>
> > wrote:
> > >
> > > > Hi guys,
> > > >
> > > > I'm new to slider and try to convert some application into yarn app.
> I
> > > > would like to ask is there a way to specify only a subset of nodes in
> > the
> > > > cluster to run my app and can slider guarantee every container(of
> that
> > > > application) run on different nodes?
> > > >
> > > > Thank you very much!
> > > >
> > > > Best,
> > > > Siyuan
> > > >
> > >
> > >
> > >
> > > --
> > > Hortonworks Certified Hadoop 2 Java Developer
> > >
> > > Solutions Engineer, Hortonworks
> > > Phone: Office (734)-288-3930 Mobile (734)-301-9812
> > > Email: jniemiec@hortonworks.com
> > >
> > > Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with
> > > Apache Hadoop 2
> > > <
> > >
> >
> http://www.amazon.com/Apache-Hadoop-YARN-Processing-Addison-Wesley/dp/0321934504
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
Hortonworks Certified Hadoop 2 Java Developer
Solutions Engineer, Hortonworks
Phone: Office (734)-288-3930 Mobile (734)-301-9812
Email: jniemiec@hortonworks.com
Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with
Apache Hadoop 2
<http://www.amazon.com/Apache-Hadoop-YARN-Processing-Addison-Wesley/dp/0321934504>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Is there a way to run application on certain subset of nodes?
Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Thanks guys, so my next question is how can you guarantee containers are
allocated on different nodes?
Best,
Siyuan
On Mon, Aug 4, 2014 at 1:02 PM, Sumit Mohanty <sm...@hortonworks.com>
wrote:
> +1 to Joe's explanation.
>
> Slider plans to use the "label" support enabled through YARN-796 and allow
> creation of app instances against a label. This way you can ensure that the
> app instance has containers allocated only on a subset of the nodes.
>
>
> On Mon, Aug 4, 2014 at 12:15 PM, Joseph Niemiec <jn...@hortonworks.com>
> wrote:
>
> > I'll let someone else chime in here but my assumption is this will
> require
> > YARN-796 and then have Slider expose the config somehow which from my
> > exploration has yet to be done.
> >
> > Outside of Slider if you wrote your own AM you can control this today in
> > your resourcerequest to the RM.
> >
> > https://issues.apache.org/jira/browse/YARN-796
> >
> >
> > On Mon, Aug 4, 2014 at 2:37 PM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
> >
> > > Hi guys,
> > >
> > > I'm new to slider and try to convert some application into yarn app. I
> > > would like to ask is there a way to specify only a subset of nodes in
> the
> > > cluster to run my app and can slider guarantee every container(of that
> > > application) run on different nodes?
> > >
> > > Thank you very much!
> > >
> > > Best,
> > > Siyuan
> > >
> >
> >
> >
> > --
> > Hortonworks Certified Hadoop 2 Java Developer
> >
> > Solutions Engineer, Hortonworks
> > Phone: Office (734)-288-3930 Mobile (734)-301-9812
> > Email: jniemiec@hortonworks.com
> >
> > Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with
> > Apache Hadoop 2
> > <
> >
> http://www.amazon.com/Apache-Hadoop-YARN-Processing-Addison-Wesley/dp/0321934504
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Is there a way to run application on certain subset of nodes?
Posted by Sumit Mohanty <sm...@hortonworks.com>.
+1 to Joe's explanation.
Slider plans to use the "label" support enabled through YARN-796 and allow
creation of app instances against a label. This way you can ensure that the
app instance has containers allocated only on a subset of the nodes.
On Mon, Aug 4, 2014 at 12:15 PM, Joseph Niemiec <jn...@hortonworks.com>
wrote:
> I'll let someone else chime in here but my assumption is this will require
> YARN-796 and then have Slider expose the config somehow which from my
> exploration has yet to be done.
>
> Outside of Slider if you wrote your own AM you can control this today in
> your resourcerequest to the RM.
>
> https://issues.apache.org/jira/browse/YARN-796
>
>
> On Mon, Aug 4, 2014 at 2:37 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
>
> > Hi guys,
> >
> > I'm new to slider and try to convert some application into yarn app. I
> > would like to ask is there a way to specify only a subset of nodes in the
> > cluster to run my app and can slider guarantee every container(of that
> > application) run on different nodes?
> >
> > Thank you very much!
> >
> > Best,
> > Siyuan
> >
>
>
>
> --
> Hortonworks Certified Hadoop 2 Java Developer
>
> Solutions Engineer, Hortonworks
> Phone: Office (734)-288-3930 Mobile (734)-301-9812
> Email: jniemiec@hortonworks.com
>
> Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with
> Apache Hadoop 2
> <
> http://www.amazon.com/Apache-Hadoop-YARN-Processing-Addison-Wesley/dp/0321934504
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Is there a way to run application on certain subset of nodes?
Posted by Joseph Niemiec <jn...@hortonworks.com>.
I'll let someone else chime in here but my assumption is this will require
YARN-796 and then have Slider expose the config somehow which from my
exploration has yet to be done.
Outside of Slider if you wrote your own AM you can control this today in
your resourcerequest to the RM.
https://issues.apache.org/jira/browse/YARN-796
On Mon, Aug 4, 2014 at 2:37 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
> Hi guys,
>
> I'm new to slider and try to convert some application into yarn app. I
> would like to ask is there a way to specify only a subset of nodes in the
> cluster to run my app and can slider guarantee every container(of that
> application) run on different nodes?
>
> Thank you very much!
>
> Best,
> Siyuan
>
--
Hortonworks Certified Hadoop 2 Java Developer
Solutions Engineer, Hortonworks
Phone: Office (734)-288-3930 Mobile (734)-301-9812
Email: jniemiec@hortonworks.com
Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with
Apache Hadoop 2
<http://www.amazon.com/Apache-Hadoop-YARN-Processing-Addison-Wesley/dp/0321934504>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.