You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Qian Zhang <zh...@gmail.com> on 2019/09/02 11:22:11 UTC
Re: Please some help regression testing a task
Can you check if the task is throttled? You can run the command
`/proc/<task-pid>/cgroup` to get the cgroups of the task, and then check
the `cpu.stat` file under task's CPU cgroups, e.g.:
$ cat
> /sys/fs/cgroup/cpuacct/mesos/bd5bc588-7565-4c7e-a5f0-d33850b2ec0a/cpu.stat
> nr_periods 118
> nr_throttled 37
> throttled_time 633829202
>
If `nr_throttled` is greater than 0, then that means the task was throttled
which may affect its performance.
Regards,
Qian Zhang
On Sat, Aug 31, 2019 at 11:48 PM Marc Roos <M....@f1-outsourcing.eu> wrote:
>
>
> mesos-1.8.1-2.0.1.el7.x86_64
> CentOS Linux release 7.6.1810 (Core)
>
>
>
> -----Original Message-----
> To: user
> Subject: Please some help regression testing a task
>
>
> I have a task that under performs. I am unable to discover what is
> causing it. Could this be something mesos specific?
> Performance difference is 1k q/s vs 20k q/s
>
>
> 1. If manually I run the task on the host the performance is ok
> > I think one could rule out network connectivity on/of the host and
> > host issues
>
>
> 2. If I manually run a task in the same netns as the under performing
> task, the performance is ok.
> ip netns exec bind bash
> chroot 04a81d99-9b99-410d-bf83-d6d70ef2c7bb/
> (changed only the config port to 54)
> named -u named
> > I think we can rule out netns issues
>
>
> 3. If I manually remove or change the cgroups of the mesos/marathon
> task, the performance is still bad
>
> echo 2932859 > /sys/fs/cgroup/memory/user.slice/tasks
> echo 2932859 > /sys/fs/cgroup/devices/user.slice/tasks
> echo 2932859 > /sys/fs/cgroup/cpu/user.slice/tasks
> echo 2932859 > /sys/fs/cgroup/cpuacct/user.slice/tasks
> echo 2932859 > /sys/fs/cgroup/pids/user.slice/tasks
> echo 2932859 > /sys/fs/cgroup/blkio/user.slice/tasks
>
> or
>
> echo 2932859 > /sys/fs/cgroup/memory/user.slice/tasks
> echo 2932859 > /sys/fs/cgroup/devices/user.slice/tasks
> echo 2932859 > /sys/fs/cgroup/cpu/user.slice/tasks
> echo 2932859 > /sys/fs/cgroup/cpuacct/user.slice/tasks
> echo 2932859 > /sys/fs/cgroup/pids/user.slice/tasks
> echo 2932859 > /sys/fs/cgroup/blkio/user.slice/tasks
>
>
> [@]# cat /proc/2936696/cgroup
> 11:hugetlb:/
> 10:memory:/user.slice
> 9:devices:/user.slice
> 8:cpuacct,cpu:/user.slice
> 7:perf_event:/
> 6:cpuset:/
> 5:pids:/user.slice
> 4:freezer:/
> 3:blkio:/user.slice
> 2:net_prio,net_cls:/
> 1:name=systemd:/user.slice/user-0.slice/session-17385.scope
>
> [@]# cat /proc/2932859/cgroup
> 11:hugetlb:/
> 10:memory:/user.slice
> 9:devices:/user.slice
> 8:cpuacct,cpu:/user.slice
> 7:perf_event:/
> 6:cpuset:/
> 5:pids:/user.slice
> 4:freezer:/
> 3:blkio:/user.slice
> 2:net_prio,net_cls:/
> 1:name=systemd:/mesos/812c481b-c0a4-444a-aafa-de98da9698e2
>
>
>
>
RE: Please some help regression testing a task
Posted by Marc Roos <M....@f1-outsourcing.eu>.
No it is not throttled, besides changing the runtime cgroups of the task
to user.slice should have revealed some difference then not?
[@~]# cat
/sys/fs/cgroup/cpuacct/mesos/d0923b5a-5b96-41cc-b291-4effc0bfcbb9/cpu.st
at
nr_periods 0
nr_throttled 0
throttled_time 0
-----Original Message-----
To: user
Subject: Re: Please some help regression testing a task
Can you check if the task is throttled? You can run the command
`/proc/<task-pid>/cgroup` to get the cgroups of the task, and then check
the `cpu.stat` file under task's CPU cgroups, e.g.:
$ cat
/sys/fs/cgroup/cpuacct/mesos/bd5bc588-7565-4c7e-a5f0-d33850b2ec0a/cpu.st
at
nr_periods 118
nr_throttled 37
throttled_time 633829202
If `nr_throttled` is greater than 0, then that means the task was
throttled which may affect its performance.
Regards,
Qian Zhang
On Sat, Aug 31, 2019 at 11:48 PM Marc Roos <M....@f1-outsourcing.eu>
wrote:
mesos-1.8.1-2.0.1.el7.x86_64
CentOS Linux release 7.6.1810 (Core)
-----Original Message-----
To: user
Subject: Please some help regression testing a task
I have a task that under performs. I am unable to discover what is
causing it. Could this be something mesos specific?
Performance difference is 1k q/s vs 20k q/s
1. If manually I run the task on the host the performance is ok
> I think one could rule out network connectivity on/of the host
and
> host issues
2. If I manually run a task in the same netns as the under
performing
task, the performance is ok.
ip netns exec bind bash
chroot 04a81d99-9b99-410d-bf83-d6d70ef2c7bb/
(changed only the config port to 54)
named -u named
> I think we can rule out netns issues
3. If I manually remove or change the cgroups of the mesos/marathon
task, the performance is still bad
echo 2932859 > /sys/fs/cgroup/memory/user.slice/tasks
echo 2932859 > /sys/fs/cgroup/devices/user.slice/tasks
echo 2932859 > /sys/fs/cgroup/cpu/user.slice/tasks
echo 2932859 > /sys/fs/cgroup/cpuacct/user.slice/tasks
echo 2932859 > /sys/fs/cgroup/pids/user.slice/tasks
echo 2932859 > /sys/fs/cgroup/blkio/user.slice/tasks
or
echo 2932859 > /sys/fs/cgroup/memory/user.slice/tasks
echo 2932859 > /sys/fs/cgroup/devices/user.slice/tasks
echo 2932859 > /sys/fs/cgroup/cpu/user.slice/tasks
echo 2932859 > /sys/fs/cgroup/cpuacct/user.slice/tasks
echo 2932859 > /sys/fs/cgroup/pids/user.slice/tasks
echo 2932859 > /sys/fs/cgroup/blkio/user.slice/tasks
[@]# cat /proc/2936696/cgroup
11:hugetlb:/
10:memory:/user.slice
9:devices:/user.slice
8:cpuacct,cpu:/user.slice
7:perf_event:/
6:cpuset:/
5:pids:/user.slice
4:freezer:/
3:blkio:/user.slice
2:net_prio,net_cls:/
1:name=systemd:/user.slice/user-0.slice/session-17385.scope
[@]# cat /proc/2932859/cgroup
11:hugetlb:/
10:memory:/user.slice
9:devices:/user.slice
8:cpuacct,cpu:/user.slice
7:perf_event:/
6:cpuset:/
5:pids:/user.slice
4:freezer:/
3:blkio:/user.slice
2:net_prio,net_cls:/
1:name=systemd:/mesos/812c481b-c0a4-444a-aafa-de98da9698e2