You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Juan Rodríguez Hortalá <ju...@gmail.com> on 2017/12/02 03:14:40 UTC
Ensure no more than one container for the same application is running
in the same host
Hi,
I'm new to slider, and I'm trying to run an application for a distributed
database on slider using simple packaging. I have to specify a RPC port
that should be the same in all daemons running in each node. The problem is
that if 2 containers for this application are running on the same node then
they would try to use the same port. It is possible to specify somehow that
we don't want to have more than one container for a specific component of
the application running in the same node?
Thanks in advance.
Juan
Re: [EXTERNAL] Re: Ensure no more than one container for the same
application is running in the same host
Posted by Juan Rodríguez Hortalá <ju...@gmail.com>.
For now we are running Hadoop 2.7, but this reference will be very useful
for the future.
Thanks a lot!
On Mon, Dec 4, 2017 at 3:29 PM, David.Serafini <Da...@target.com>
wrote:
> That's great for people on the bleeding edge, but I don't expect our
> production Hadoop cluster will migrate to 3.x any time soon, so I'll
> continue to run slider 0.9x
>
> -david
>
>
> On 12/4/17, 3:26 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
> You are right. Since YARN did not implement anti-affinity, Slider
> implemented it in its own way. No special YARN feature was used to
> implement Slider¹s anti-affinity. Which hadoop version are you using?
>
> By the way, I hope you are following
> https://issues.apache.org/jira/browse/YARN-4692 - Slider is being
> deprecated and has been rolled into YARN as a first class module
> (already
> in trunk now and most likely the first release will be 3.1.x). It is
> being
> referred to as ³YARN Service² now -
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-
> project/hadoop-yarn
> /hadoop-yarn-site/src/site/markdown/yarn-service/QuickStart.md
>
> -Gour
>
> On 12/4/17, 3:09 PM, "Juan Rodríguez Hortalá"
> <ju...@gmail.com> wrote:
>
> >One more question, in
> >https://slider.incubator.apache.org/docs/configuration/
> resources.html#plac
> >ement
> >it says that "Once YARN adds support for this, Slider will support
> it."
> >and
> >https://issues.apache.org/jira/browse/YARN-1042 for "add ability to
> >specify
> >affinity/anti-affinity in container requests" is still unresolved.
> However
> >in I see ANTI_AFFINITY_REQUIRED in
> >https://github.com/apache/incubator-slider/blob/develop/
> slider-core/src/ma
> >in/java/org/apache/slider/providers/PlacementPolicy.java#L57
> >and other places of the code. Is anti affinity implemented in slider
> >before
> >YARN-1042, by other means independent from that YARN feature? Is
> there a
> >minimal version of YARN required to use anti affinity in Slider?
> >
> >Thanks again,
> >
> >Juan
> >
> >On Mon, Dec 4, 2017 at 2:56 PM, Juan Rodríguez Hortalá <
> >juan.rodriguez.hortala@gmail.com> wrote:
> >
> >> Thanks a lot Gour! I'll do so.
> >>
> >> Greetings,
> >>
> >> Juan
> >>
> >> On Fri, Dec 1, 2017 at 11:38 PM, Gour Saha <gs...@hortonworks.com>
> >>wrote:
> >>
> >>> Set yarn.component.placement.policy to 4 for the component in
> >>> resources.json to get anti- affinity.
> >>>
> >>> -Gour
> >>>
> >>> - Sent from my iPhone
> >>>
> >>> On Dec 1, 2017, at 7:14 PM, Juan Rodr?guez Hortal? <
> >>>
> >>>juan.rodriguez.hortala@gmail.com<mailto:juan.
> rodriguez.hortala@gmail.com
> >>>>>
> >>> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I'm new to slider, and I'm trying to run an application for a
> >>>distributed
> >>> database on slider using simple packaging. I have to specify a RPC
> port
> >>> that should be the same in all daemons running in each node. The
> >>>problem
> >>> is
> >>> that if 2 containers for this application are running on the same
> node
> >>> then
> >>> they would try to use the same port. It is possible to specify
> somehow
> >>> that
> >>> we don't want to have more than one container for a specific
> component
> >>>of
> >>> the application running in the same node?
> >>>
> >>> Thanks in advance.
> >>>
> >>> Juan
> >>>
> >>
> >>
>
>
>
>
>
Re: [EXTERNAL] Re: Ensure no more than one container for the same
application is running in the same host
Posted by "David.Serafini" <Da...@target.com>.
That's great for people on the bleeding edge, but I don't expect our production Hadoop cluster will migrate to 3.x any time soon, so I'll continue to run slider 0.9x
-david
On 12/4/17, 3:26 PM, Gour Saha <gs...@hortonworks.com> wrote:
You are right. Since YARN did not implement anti-affinity, Slider
implemented it in its own way. No special YARN feature was used to
implement Slider¹s anti-affinity. Which hadoop version are you using?
By the way, I hope you are following
https://issues.apache.org/jira/browse/YARN-4692 - Slider is being
deprecated and has been rolled into YARN as a first class module (already
in trunk now and most likely the first release will be 3.1.x). It is being
referred to as ³YARN Service² now -
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn
/hadoop-yarn-site/src/site/markdown/yarn-service/QuickStart.md
-Gour
On 12/4/17, 3:09 PM, "Juan Rodríguez Hortalá"
<ju...@gmail.com> wrote:
>One more question, in
>https://slider.incubator.apache.org/docs/configuration/resources.html#plac
>ement
>it says that "Once YARN adds support for this, Slider will support it."
>and
>https://issues.apache.org/jira/browse/YARN-1042 for "add ability to
>specify
>affinity/anti-affinity in container requests" is still unresolved. However
>in I see ANTI_AFFINITY_REQUIRED in
>https://github.com/apache/incubator-slider/blob/develop/slider-core/src/ma
>in/java/org/apache/slider/providers/PlacementPolicy.java#L57
>and other places of the code. Is anti affinity implemented in slider
>before
>YARN-1042, by other means independent from that YARN feature? Is there a
>minimal version of YARN required to use anti affinity in Slider?
>
>Thanks again,
>
>Juan
>
>On Mon, Dec 4, 2017 at 2:56 PM, Juan Rodríguez Hortalá <
>juan.rodriguez.hortala@gmail.com> wrote:
>
>> Thanks a lot Gour! I'll do so.
>>
>> Greetings,
>>
>> Juan
>>
>> On Fri, Dec 1, 2017 at 11:38 PM, Gour Saha <gs...@hortonworks.com>
>>wrote:
>>
>>> Set yarn.component.placement.policy to 4 for the component in
>>> resources.json to get anti- affinity.
>>>
>>> -Gour
>>>
>>> - Sent from my iPhone
>>>
>>> On Dec 1, 2017, at 7:14 PM, Juan Rodr?guez Hortal? <
>>>
>>>juan.rodriguez.hortala@gmail.com<mailto:juan.rodriguez.hortala@gmail.com
>>>>>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I'm new to slider, and I'm trying to run an application for a
>>>distributed
>>> database on slider using simple packaging. I have to specify a RPC port
>>> that should be the same in all daemons running in each node. The
>>>problem
>>> is
>>> that if 2 containers for this application are running on the same node
>>> then
>>> they would try to use the same port. It is possible to specify somehow
>>> that
>>> we don't want to have more than one container for a specific component
>>>of
>>> the application running in the same node?
>>>
>>> Thanks in advance.
>>>
>>> Juan
>>>
>>
>>
Re: Ensure no more than one container for the same application is
running in the same host
Posted by Gour Saha <gs...@hortonworks.com>.
You are right. Since YARN did not implement anti-affinity, Slider
implemented it in its own way. No special YARN feature was used to
implement Slider¹s anti-affinity. Which hadoop version are you using?
By the way, I hope you are following
https://issues.apache.org/jira/browse/YARN-4692 - Slider is being
deprecated and has been rolled into YARN as a first class module (already
in trunk now and most likely the first release will be 3.1.x). It is being
referred to as ³YARN Service² now -
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn
/hadoop-yarn-site/src/site/markdown/yarn-service/QuickStart.md
-Gour
On 12/4/17, 3:09 PM, "Juan Rodríguez Hortalá"
<ju...@gmail.com> wrote:
>One more question, in
>https://slider.incubator.apache.org/docs/configuration/resources.html#plac
>ement
>it says that "Once YARN adds support for this, Slider will support it."
>and
>https://issues.apache.org/jira/browse/YARN-1042 for "add ability to
>specify
>affinity/anti-affinity in container requests" is still unresolved. However
>in I see ANTI_AFFINITY_REQUIRED in
>https://github.com/apache/incubator-slider/blob/develop/slider-core/src/ma
>in/java/org/apache/slider/providers/PlacementPolicy.java#L57
>and other places of the code. Is anti affinity implemented in slider
>before
>YARN-1042, by other means independent from that YARN feature? Is there a
>minimal version of YARN required to use anti affinity in Slider?
>
>Thanks again,
>
>Juan
>
>On Mon, Dec 4, 2017 at 2:56 PM, Juan Rodríguez Hortalá <
>juan.rodriguez.hortala@gmail.com> wrote:
>
>> Thanks a lot Gour! I'll do so.
>>
>> Greetings,
>>
>> Juan
>>
>> On Fri, Dec 1, 2017 at 11:38 PM, Gour Saha <gs...@hortonworks.com>
>>wrote:
>>
>>> Set yarn.component.placement.policy to 4 for the component in
>>> resources.json to get anti- affinity.
>>>
>>> -Gour
>>>
>>> - Sent from my iPhone
>>>
>>> On Dec 1, 2017, at 7:14 PM, Juan Rodr?guez Hortal? <
>>>
>>>juan.rodriguez.hortala@gmail.com<mailto:juan.rodriguez.hortala@gmail.com
>>>>>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I'm new to slider, and I'm trying to run an application for a
>>>distributed
>>> database on slider using simple packaging. I have to specify a RPC port
>>> that should be the same in all daemons running in each node. The
>>>problem
>>> is
>>> that if 2 containers for this application are running on the same node
>>> then
>>> they would try to use the same port. It is possible to specify somehow
>>> that
>>> we don't want to have more than one container for a specific component
>>>of
>>> the application running in the same node?
>>>
>>> Thanks in advance.
>>>
>>> Juan
>>>
>>
>>
Re: Ensure no more than one container for the same application is
running in the same host
Posted by Juan Rodríguez Hortalá <ju...@gmail.com>.
One more question, in
https://slider.incubator.apache.org/docs/configuration/resources.html#placement
it says that "Once YARN adds support for this, Slider will support it." and
https://issues.apache.org/jira/browse/YARN-1042 for "add ability to specify
affinity/anti-affinity in container requests" is still unresolved. However
in I see ANTI_AFFINITY_REQUIRED in
https://github.com/apache/incubator-slider/blob/develop/slider-core/src/main/java/org/apache/slider/providers/PlacementPolicy.java#L57
and other places of the code. Is anti affinity implemented in slider before
YARN-1042, by other means independent from that YARN feature? Is there a
minimal version of YARN required to use anti affinity in Slider?
Thanks again,
Juan
On Mon, Dec 4, 2017 at 2:56 PM, Juan Rodríguez Hortalá <
juan.rodriguez.hortala@gmail.com> wrote:
> Thanks a lot Gour! I'll do so.
>
> Greetings,
>
> Juan
>
> On Fri, Dec 1, 2017 at 11:38 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
>> Set yarn.component.placement.policy to 4 for the component in
>> resources.json to get anti- affinity.
>>
>> -Gour
>>
>> - Sent from my iPhone
>>
>> On Dec 1, 2017, at 7:14 PM, Juan Rodr?guez Hortal? <
>> juan.rodriguez.hortala@gmail.com<ma...@gmail.com>>
>> wrote:
>>
>> Hi,
>>
>> I'm new to slider, and I'm trying to run an application for a distributed
>> database on slider using simple packaging. I have to specify a RPC port
>> that should be the same in all daemons running in each node. The problem
>> is
>> that if 2 containers for this application are running on the same node
>> then
>> they would try to use the same port. It is possible to specify somehow
>> that
>> we don't want to have more than one container for a specific component of
>> the application running in the same node?
>>
>> Thanks in advance.
>>
>> Juan
>>
>
>
Re: Ensure no more than one container for the same application is
running in the same host
Posted by Juan Rodríguez Hortalá <ju...@gmail.com>.
Thanks a lot Gour! I'll do so.
Greetings,
Juan
On Fri, Dec 1, 2017 at 11:38 PM, Gour Saha <gs...@hortonworks.com> wrote:
> Set yarn.component.placement.policy to 4 for the component in
> resources.json to get anti- affinity.
>
> -Gour
>
> - Sent from my iPhone
>
> On Dec 1, 2017, at 7:14 PM, Juan Rodr?guez Hortal? <
> juan.rodriguez.hortala@gmail.com<ma...@gmail.com>>
> wrote:
>
> Hi,
>
> I'm new to slider, and I'm trying to run an application for a distributed
> database on slider using simple packaging. I have to specify a RPC port
> that should be the same in all daemons running in each node. The problem is
> that if 2 containers for this application are running on the same node then
> they would try to use the same port. It is possible to specify somehow that
> we don't want to have more than one container for a specific component of
> the application running in the same node?
>
> Thanks in advance.
>
> Juan
>
Re: Ensure no more than one container for the same application is
running in the same host
Posted by Gour Saha <gs...@hortonworks.com>.
Set yarn.component.placement.policy to 4 for the component in resources.json to get anti- affinity.
-Gour
- Sent from my iPhone
On Dec 1, 2017, at 7:14 PM, Juan Rodr?guez Hortal? <ju...@gmail.com>> wrote:
Hi,
I'm new to slider, and I'm trying to run an application for a distributed
database on slider using simple packaging. I have to specify a RPC port
that should be the same in all daemons running in each node. The problem is
that if 2 containers for this application are running on the same node then
they would try to use the same port. It is possible to specify somehow that
we don't want to have more than one container for a specific component of
the application running in the same node?
Thanks in advance.
Juan