You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Baskar Sikkayan <ba...@gmail.com> on 2018/03/08 05:48:29 UTC
2 Mesos Slaves in same machine
Hi,
We are running 3 node mesos cluster and on each node running 1 mesos
master and 2 mesos slaves. Is this a good practice?
Will it be a problem running 2 slaves as it might offer too much and will
get overloaded.
Thanks,
Baskar.S
Re: 2 Mesos Slaves in same machine
Posted by Baskar Sikkayan <ba...@gmail.com>.
Hi,
Is it still an issue with the below config as we have cgroup setting over
there?
Do i need to use both "--resources" && "--cgroups_root" setting for
running 2 slaves in the same node. Or, either one of them should do.
Also, is there any setting to be done at the OS( cgroup ) level for running
2 slaves.
Anyway, since it is not a recommended to run 2 slaves on the same node, I
am not going with that option. Just trying to understand here ...
Thanks
On Thu, Mar 8, 2018 at 11:02 AM, Baskar Sikkayan <ba...@gmail.com>
wrote:
> Hi,
>
> This is our docker compose for starting mesos slave.
>
> mesos-slave1:
>
> image: docker.com/mesos/mesos-slave:0.28.2
>
> depends_on: [ mesos-master ]
>
> restart: always
>
> privileged: true
>
> network_mode: host
>
> volumes:
>
> - ~/mesos-data/slave-1:/tmp/mesos
>
> - /cgroup:/sys/fs/cgroup
>
> - /var/run/docker.sock:/var/run/docker.sock
>
> environment:
>
> MESOS_CONTAINERIZERS: docker
>
> MESOS_MASTER: "%mesos.zk%"
>
> MESOS_PORT: 5051
>
> MESOS_LOGGING_LEVEL: WARNING
>
> MESOS_INITIALIZE_DRIVER_LOGGING: "false"
>
> MESOS_GC_DELAY: "%mesos.gc.delay%"
>
> MESOS_AUTHENTICATE_HTTP: "true"
>
> PRINCIPAL: "%mesos.principal%"
>
> SECRET: "%mesos.secret%"
>
>
> We are running mesos version 0.28.2. I know, it is pretty old. But no
> option to upgrade in the near future. We use Mesos containerizers.
>
> Is this still an issue with 2 mesos salves on the same node?
>
>
> Thanks
>
> On Wed, Mar 7, 2018 at 10:11 PM, Jie Yu <yu...@gmail.com> wrote:
>
>> Running 2 agents on the same server will send the same offer twice( same
>>> offer by each agent ) and there are chances that resources will be over
>>> utilized by accepting the same offer twice by the framework.
>>
>>
>> If you didn't specify `--resources` flags on the agent, then yes.
>>
>> May I know the settings( cgroup ) required to run 2 slaves in the same
>>> server and How does it behave?
>>
>>
>> Check out `--cgroups_root` flag in https://github.com/apache/m
>> esos/blob/master/docs/configuration/agent.md
>>
>> This flag only applies to MesosContainerizer (can be used to launch
>> Docker containers).
>>
>> - Jie
>>
>>
>> On Wed, Mar 7, 2018 at 10:06 PM, Baskar Sikkayan <ba...@gmail.com>
>> wrote:
>>
>>> 1. Is my understanding correct?
>>>
>>> Running 2 agents on the same server will send the same offer twice( same
>>> offer by each agent ) and there are chances that resources will be over
>>> utilized by accepting the same offer twice by the framework.
>>>
>>> 2. May I know the settings( cgroup ) required to run 2 slaves in the
>>> same server and How does it behave?
>>>
>>> On Wed, Mar 7, 2018 at 10:03 PM, Jie Yu <yu...@gmail.com> wrote:
>>>
>>>> Running Docker containers won't work properly because restarting one
>>>> agent will cause Docker containers managed by the other agent to be deleted.
>>>>
>>>> On Wed, Mar 7, 2018 at 9:58 PM, Baskar Sikkayan <ba...@gmail.com>
>>>> wrote:
>>>>
>>>>> We dont have any isolation setting. Looks like 2 slaves are sending
>>>>> too many offers( same resource offers by 2 agents ) and hence servers gets
>>>>> overloaded by too many docker jobs and docker becomes unresponsive. Even
>>>>> "docker ps" not working in this case and docker meta files gets corrupted
>>>>> on server reboot. Not sure why docker becomes unresponsive. Is running 2
>>>>> slaves are the main reason for this issue?
>>>>>
>>>>> On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> It depends on your isolation setting (mainly cgroup, or any node
>>>>>> level resources). In general, we don't recommend folks use multiple agents
>>>>>> on a node.
>>>>>>
>>>>>> It's possible to make it work by setting `cgroup_root` separately for
>>>>>> MesosContainerizer. For DockerContainerizer, currently, we hard code
>>>>>> `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node
>>>>>> properly.
>>>>>>
>>>>>> - Jie
>>>>>>
>>>>>> On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan <baskar.sks@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We are running 3 node mesos cluster and on each node running 1
>>>>>>> mesos master and 2 mesos slaves. Is this a good practice?
>>>>>>>
>>>>>>> Will it be a problem running 2 slaves as it might offer too much and
>>>>>>> will get overloaded.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Baskar.S
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: 2 Mesos Slaves in same machine
Posted by Baskar Sikkayan <ba...@gmail.com>.
Hi,
This is our docker compose for starting mesos slave.
mesos-slave1:
image: docker.com/mesos/mesos-slave:0.28.2
depends_on: [ mesos-master ]
restart: always
privileged: true
network_mode: host
volumes:
- ~/mesos-data/slave-1:/tmp/mesos
- /cgroup:/sys/fs/cgroup
- /var/run/docker.sock:/var/run/docker.sock
environment:
MESOS_CONTAINERIZERS: docker
MESOS_MASTER: "%mesos.zk%"
MESOS_PORT: 5051
MESOS_LOGGING_LEVEL: WARNING
MESOS_INITIALIZE_DRIVER_LOGGING: "false"
MESOS_GC_DELAY: "%mesos.gc.delay%"
MESOS_AUTHENTICATE_HTTP: "true"
PRINCIPAL: "%mesos.principal%"
SECRET: "%mesos.secret%"
We are running mesos version 0.28.2. I know, it is pretty old. But no
option to upgrade in the near future. We use Mesos containerizers.
Is this still an issue with 2 mesos salves on the same node?
Thanks
On Wed, Mar 7, 2018 at 10:11 PM, Jie Yu <yu...@gmail.com> wrote:
> Running 2 agents on the same server will send the same offer twice( same
>> offer by each agent ) and there are chances that resources will be over
>> utilized by accepting the same offer twice by the framework.
>
>
> If you didn't specify `--resources` flags on the agent, then yes.
>
> May I know the settings( cgroup ) required to run 2 slaves in the same
>> server and How does it behave?
>
>
> Check out `--cgroups_root` flag in https://github.com/apache/
> mesos/blob/master/docs/configuration/agent.md
>
> This flag only applies to MesosContainerizer (can be used to launch Docker
> containers).
>
> - Jie
>
>
> On Wed, Mar 7, 2018 at 10:06 PM, Baskar Sikkayan <ba...@gmail.com>
> wrote:
>
>> 1. Is my understanding correct?
>>
>> Running 2 agents on the same server will send the same offer twice( same
>> offer by each agent ) and there are chances that resources will be over
>> utilized by accepting the same offer twice by the framework.
>>
>> 2. May I know the settings( cgroup ) required to run 2 slaves in the same
>> server and How does it behave?
>>
>> On Wed, Mar 7, 2018 at 10:03 PM, Jie Yu <yu...@gmail.com> wrote:
>>
>>> Running Docker containers won't work properly because restarting one
>>> agent will cause Docker containers managed by the other agent to be deleted.
>>>
>>> On Wed, Mar 7, 2018 at 9:58 PM, Baskar Sikkayan <ba...@gmail.com>
>>> wrote:
>>>
>>>> We dont have any isolation setting. Looks like 2 slaves are sending too
>>>> many offers( same resource offers by 2 agents ) and hence servers gets
>>>> overloaded by too many docker jobs and docker becomes unresponsive. Even
>>>> "docker ps" not working in this case and docker meta files gets corrupted
>>>> on server reboot. Not sure why docker becomes unresponsive. Is running 2
>>>> slaves are the main reason for this issue?
>>>>
>>>> On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu <yu...@gmail.com> wrote:
>>>>
>>>>> It depends on your isolation setting (mainly cgroup, or any node level
>>>>> resources). In general, we don't recommend folks use multiple agents on a
>>>>> node.
>>>>>
>>>>> It's possible to make it work by setting `cgroup_root` separately for
>>>>> MesosContainerizer. For DockerContainerizer, currently, we hard code
>>>>> `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node
>>>>> properly.
>>>>>
>>>>> - Jie
>>>>>
>>>>> On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan <ba...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We are running 3 node mesos cluster and on each node running 1
>>>>>> mesos master and 2 mesos slaves. Is this a good practice?
>>>>>>
>>>>>> Will it be a problem running 2 slaves as it might offer too much and
>>>>>> will get overloaded.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Baskar.S
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: 2 Mesos Slaves in same machine
Posted by Jie Yu <yu...@gmail.com>.
>
> Running 2 agents on the same server will send the same offer twice( same
> offer by each agent ) and there are chances that resources will be over
> utilized by accepting the same offer twice by the framework.
If you didn't specify `--resources` flags on the agent, then yes.
May I know the settings( cgroup ) required to run 2 slaves in the same
> server and How does it behave?
Check out `--cgroups_root` flag in
https://github.com/apache/mesos/blob/master/docs/configuration/agent.md
This flag only applies to MesosContainerizer (can be used to launch Docker
containers).
- Jie
On Wed, Mar 7, 2018 at 10:06 PM, Baskar Sikkayan <ba...@gmail.com>
wrote:
> 1. Is my understanding correct?
>
> Running 2 agents on the same server will send the same offer twice( same
> offer by each agent ) and there are chances that resources will be over
> utilized by accepting the same offer twice by the framework.
>
> 2. May I know the settings( cgroup ) required to run 2 slaves in the same
> server and How does it behave?
>
> On Wed, Mar 7, 2018 at 10:03 PM, Jie Yu <yu...@gmail.com> wrote:
>
>> Running Docker containers won't work properly because restarting one
>> agent will cause Docker containers managed by the other agent to be deleted.
>>
>> On Wed, Mar 7, 2018 at 9:58 PM, Baskar Sikkayan <ba...@gmail.com>
>> wrote:
>>
>>> We dont have any isolation setting. Looks like 2 slaves are sending too
>>> many offers( same resource offers by 2 agents ) and hence servers gets
>>> overloaded by too many docker jobs and docker becomes unresponsive. Even
>>> "docker ps" not working in this case and docker meta files gets corrupted
>>> on server reboot. Not sure why docker becomes unresponsive. Is running 2
>>> slaves are the main reason for this issue?
>>>
>>> On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu <yu...@gmail.com> wrote:
>>>
>>>> It depends on your isolation setting (mainly cgroup, or any node level
>>>> resources). In general, we don't recommend folks use multiple agents on a
>>>> node.
>>>>
>>>> It's possible to make it work by setting `cgroup_root` separately for
>>>> MesosContainerizer. For DockerContainerizer, currently, we hard code
>>>> `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node
>>>> properly.
>>>>
>>>> - Jie
>>>>
>>>> On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan <ba...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We are running 3 node mesos cluster and on each node running 1 mesos
>>>>> master and 2 mesos slaves. Is this a good practice?
>>>>>
>>>>> Will it be a problem running 2 slaves as it might offer too much and
>>>>> will get overloaded.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Baskar.S
>>>>>
>>>>
>>>>
>>>
>>
>
Re: 2 Mesos Slaves in same machine
Posted by Baskar Sikkayan <ba...@gmail.com>.
1. Is my understanding correct?
Running 2 agents on the same server will send the same offer twice( same
offer by each agent ) and there are chances that resources will be over
utilized by accepting the same offer twice by the framework.
2. May I know the settings( cgroup ) required to run 2 slaves in the same
server and How does it behave?
On Wed, Mar 7, 2018 at 10:03 PM, Jie Yu <yu...@gmail.com> wrote:
> Running Docker containers won't work properly because restarting one agent
> will cause Docker containers managed by the other agent to be deleted.
>
> On Wed, Mar 7, 2018 at 9:58 PM, Baskar Sikkayan <ba...@gmail.com>
> wrote:
>
>> We dont have any isolation setting. Looks like 2 slaves are sending too
>> many offers( same resource offers by 2 agents ) and hence servers gets
>> overloaded by too many docker jobs and docker becomes unresponsive. Even
>> "docker ps" not working in this case and docker meta files gets corrupted
>> on server reboot. Not sure why docker becomes unresponsive. Is running 2
>> slaves are the main reason for this issue?
>>
>> On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu <yu...@gmail.com> wrote:
>>
>>> It depends on your isolation setting (mainly cgroup, or any node level
>>> resources). In general, we don't recommend folks use multiple agents on a
>>> node.
>>>
>>> It's possible to make it work by setting `cgroup_root` separately for
>>> MesosContainerizer. For DockerContainerizer, currently, we hard code
>>> `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node
>>> properly.
>>>
>>> - Jie
>>>
>>> On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan <ba...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are running 3 node mesos cluster and on each node running 1 mesos
>>>> master and 2 mesos slaves. Is this a good practice?
>>>>
>>>> Will it be a problem running 2 slaves as it might offer too much and
>>>> will get overloaded.
>>>>
>>>>
>>>> Thanks,
>>>> Baskar.S
>>>>
>>>
>>>
>>
>
Re: 2 Mesos Slaves in same machine
Posted by Jie Yu <yu...@gmail.com>.
Running Docker containers won't work properly because restarting one agent
will cause Docker containers managed by the other agent to be deleted.
On Wed, Mar 7, 2018 at 9:58 PM, Baskar Sikkayan <ba...@gmail.com>
wrote:
> We dont have any isolation setting. Looks like 2 slaves are sending too
> many offers( same resource offers by 2 agents ) and hence servers gets
> overloaded by too many docker jobs and docker becomes unresponsive. Even
> "docker ps" not working in this case and docker meta files gets corrupted
> on server reboot. Not sure why docker becomes unresponsive. Is running 2
> slaves are the main reason for this issue?
>
> On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu <yu...@gmail.com> wrote:
>
>> It depends on your isolation setting (mainly cgroup, or any node level
>> resources). In general, we don't recommend folks use multiple agents on a
>> node.
>>
>> It's possible to make it work by setting `cgroup_root` separately for
>> MesosContainerizer. For DockerContainerizer, currently, we hard code
>> `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node
>> properly.
>>
>> - Jie
>>
>> On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan <ba...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> We are running 3 node mesos cluster and on each node running 1 mesos
>>> master and 2 mesos slaves. Is this a good practice?
>>>
>>> Will it be a problem running 2 slaves as it might offer too much and
>>> will get overloaded.
>>>
>>>
>>> Thanks,
>>> Baskar.S
>>>
>>
>>
>
Re: 2 Mesos Slaves in same machine
Posted by Baskar Sikkayan <ba...@gmail.com>.
We dont have any isolation setting. Looks like 2 slaves are sending too
many offers( same resource offers by 2 agents ) and hence servers gets
overloaded by too many docker jobs and docker becomes unresponsive. Even
"docker ps" not working in this case and docker meta files gets corrupted
on server reboot. Not sure why docker becomes unresponsive. Is running 2
slaves are the main reason for this issue?
On Wed, Mar 7, 2018 at 9:52 PM, Jie Yu <yu...@gmail.com> wrote:
> It depends on your isolation setting (mainly cgroup, or any node level
> resources). In general, we don't recommend folks use multiple agents on a
> node.
>
> It's possible to make it work by setting `cgroup_root` separately for
> MesosContainerizer. For DockerContainerizer, currently, we hard code
> `DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node
> properly.
>
> - Jie
>
> On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan <ba...@gmail.com>
> wrote:
>
>> Hi,
>>
>> We are running 3 node mesos cluster and on each node running 1 mesos
>> master and 2 mesos slaves. Is this a good practice?
>>
>> Will it be a problem running 2 slaves as it might offer too much and will
>> get overloaded.
>>
>>
>> Thanks,
>> Baskar.S
>>
>
>
Re: 2 Mesos Slaves in same machine
Posted by Jie Yu <yu...@gmail.com>.
It depends on your isolation setting (mainly cgroup, or any node level
resources). In general, we don't recommend folks use multiple agents on a
node.
It's possible to make it work by setting `cgroup_root` separately for
MesosContainerizer. For DockerContainerizer, currently, we hard code
`DOCKER_NAME_PREFIX`, making it not possible to use two agents on a node
properly.
- Jie
On Wed, Mar 7, 2018 at 9:48 PM, Baskar Sikkayan <ba...@gmail.com>
wrote:
> Hi,
>
> We are running 3 node mesos cluster and on each node running 1 mesos
> master and 2 mesos slaves. Is this a good practice?
>
> Will it be a problem running 2 slaves as it might offer too much and will
> get overloaded.
>
>
> Thanks,
> Baskar.S
>