You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@mesos.apache.org by "Plotka, Bartlomiej" <ba...@intel.com> on 2015/11/17 15:55:38 UTC

[BUG] Perf isolator (from 0.25.0-rc1) does not work on CentOS 7 with 3.10 kernel.

Hi All,

It seems that we have bug in perf isolator in Apache Mesos from 0.25.0 including master
Currently, Mesos code parses perf stat output https://github.com/apache/mesos/blob/master/src/linux/perf.cpp#L409 separately to each perf version.
It is written that:
// Expected format for Linux kernel <= 3.12 is:
// value,event,cgroup

Actually, on CentOS 7...
CentOS Linux release 7.1.1503 (Core)
Linux kernel 3.10.0-229.el7.x86_64
...we have newest perf from yum CentOS package system:
# perf --version
perf version 3.10.0-229.7.2.el7.x86_64

However the format is the same as in perf(kernel) version between 3.13 - 4.0 from Mesos perspective (!)
So the command like this (perf isolator does similar command):
perf stat --all-cpus --field-separator ',' --log-fd 1 --event instructions --cgroup mesos/2aac84b2-04da-444d-b629-25057937e1dd --event cycles --cgroup mesos/2aac84b2-04da-444d-b629-25057937e1dd
Has format:
816316287,,instructions,mesos/2aac84b2-04da-444d-b629-25057937e1dd
1772022819,,cycles,mesos/2aac84b2-04da-444d-b629-25057937e1dd
Obviously it leads to error in Perf isolator:
E1117 05:44:29.625007 70482 perf_event.cpp:411] Failed to get perf sample: Failed to parse perf sample: Failed to parse perf sample line '25571484,,cycles,mesos/74a181dc-47c0-46b4-870d-1c68f76997d2': Unexpected number of fields
Which really blocks us, because we base our solution on perf metrics.

This commit has changed that before 0.25.0-rc1: https://reviews.apache.org/r/37462/
In earlier Mesos versions in Perf isolator we had stout function strings::tokenize instead of strings::split for parsing perf output. Tokenize does not include empty columns in parsed line which worked that way.

I am aware that there is work in progress to make perf isolator be independent on perf stat output and use ABI (https://issues.apache.org/jira/browse/MESOS-2834). I hope it will land soon (any help needed?)
However I just want to inform that 0.25.0 does not support monitoring perf events on CentOS 7 (3.10 kernel)

Anyone can confirm that? Are there any plans to fix this on 0.25.0? Simply changing split to tokenize would mitigate this.

Kind Regards,
Bartek Plotka (bplotka)

--------------------------------------------------------------------

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

Re: [BUG] Perf isolator (from 0.25.0-rc1) does not work on CentOS 7 with 3.10 kernel.

Posted by Vinod Kone <vi...@gmail.com>.

+cong,idownes

On Tue, Nov 17, 2015 at 6:55 AM, Plotka, Bartlomiej <
bartlomiej.plotka@intel.com> wrote:

> This commit has changed that before 0.25.0-rc1:
> https://reviews.apache.org/r/37462/
> In earlier Mesos versions in Perf isolator we had stout function
> strings::tokenize instead of strings::split for parsing perf output.
> Tokenize does not include empty columns in parsed line which worked that
> way.
>
> I am aware that there is work in progress to make perf isolator be
> independent on perf stat output and use ABI (
> https://issues.apache.org/jira/browse/MESOS-2834). I hope it will land
> soon (any help needed?)
> However I just want to inform that 0.25.0 does not support monitoring perf
> events on CentOS 7 (3.10 kernel)
>

I'll take another look at patches in the review chain starting from
https://reviews.apache.org/r/37540/. More eyes looking at those patches
would be appreciated.