Trafodion on CentOS 7.x: CPU usage differences--cosmetic change or an OS level change that can be used to improve system performance?

["D. Markt" <dm...@gmail.com>]

Hi,

 

  I know, this is the Trafodion user DL not a CentOS 7.x DL, but given an OS
changes major versions months if not years before the next level software
starts to use it, I assume Trafodion users will come looking here once
CentOS 7.x is a supported version.  I have had access to some prototype code
that is running on CentOS 7.2.  One very obvious visual change is how core
assignment/usage has changed.  Unfortunately it's harder to explain than it
is for me to see.

 

  The internal tool I'm using shows the CPU usage of each core, for each
node in the cluster.  top also has a way that you can break down CPU usage
per core, but even how you do that has changed a bit from 6.5 to 7.2.
Having watched many Trafodion on CentOS 6.x workloads over the years every
pattern is possible, but typically what you see is a downward sloping curve
or straight line.  That is, CPU0's usage is higher than CPU1's, which is
higher than CPU2's, etc.  Of course on a very busy CPU all the cores might
show similar usage and in some cases there would be one or more cores that
might be significantly higher than a lower numbered core.

 

  On CentOS 7.2 the pattern seems significantly different, though the
throughput has been equal or better.  One case I've seen multiple times on
an HT-enabled node with 32 cores: 0-15 show a pattern where the even numbers
cores are higher than the odd numbered cores but each set is similar in
usage.  Then cores 16-31 do the same, but the overall usage of each of those
cores is lower than the 0-15 core.   An example is below but of course it's
a point in time view.

 

  As far as I know the higher level software has not changed so this would
seem to be an OS-level change.  The usage values from top and the internal
tool agree so they seem correct.  So that would imply how tasks are assigned
to the cores has changed.  I'm sure there is a CentOS 7 blog entry that
explains this but I've not gone looking.  But if anyone understands this
apparent change in behavior please share so when Trafodion does support the
newer OS version other users will understand the new behavior.

 

Thanks,

Dennis

 

%Cpu0  : 51.9 us, 19.3 sy,  0.0 ni, 24.1 id,  0.0 wa,  0.0 hi,  4.7 si,  0.0
st

%Cpu1  : 36.1 us,  7.8 sy,  0.0 ni, 54.1 id,  0.3 wa,  0.0 hi,  1.7 si,  0.0
st

%Cpu2  : 53.8 us, 17.8 sy,  0.0 ni, 25.0 id,  0.0 wa,  0.0 hi,  3.4 si,  0.0
st

%Cpu3  : 33.7 us,  9.2 sy,  0.0 ni, 55.8 id,  0.0 wa,  0.0 hi,  1.4 si,  0.0
st

%Cpu4  : 52.8 us, 17.6 sy,  0.0 ni, 25.2 id,  0.0 wa,  0.0 hi,  4.5 si,  0.0
st

%Cpu5  : 32.0 us, 10.7 sy,  0.0 ni, 56.0 id,  0.0 wa,  0.0 hi,  1.4 si,  0.0
st

%Cpu6  : 51.9 us, 18.0 sy,  0.0 ni, 23.2 id,  0.0 wa,  0.0 hi,  6.9 si,  0.0
st

%Cpu7  : 33.4 us,  9.0 sy,  0.0 ni, 56.2 id,  0.0 wa,  0.0 hi,  1.4 si,  0.0
st

%Cpu8  : 51.6 us, 18.0 sy,  0.0 ni, 24.2 id,  0.0 wa,  0.0 hi,  6.2 si,  0.0
st

%Cpu9  : 32.1 us, 11.0 sy,  0.0 ni, 55.9 id,  0.0 wa,  0.0 hi,  1.0 si,  0.0
st

%Cpu10 : 53.2 us, 18.0 sy,  0.0 ni, 23.7 id,  0.0 wa,  0.0 hi,  5.1 si,  0.0
st

%Cpu11 : 33.7 us, 10.4 sy,  0.0 ni, 54.9 id,  0.0 wa,  0.0 hi,  1.0 si,  0.0
st

%Cpu12 : 52.1 us, 17.5 sy,  0.0 ni, 24.1 id,  0.0 wa,  0.0 hi,  6.3 si,  0.0
st

%Cpu13 : 34.1 us,  9.5 sy,  0.0 ni, 55.1 id,  0.0 wa,  0.0 hi,  1.4 si,  0.0
st

%Cpu14 : 50.3 us, 18.6 sy,  0.0 ni, 25.2 id,  0.0 wa,  0.0 hi,  5.9 si,  0.0
st

%Cpu15 : 34.8 us,  9.5 sy,  0.0 ni, 54.7 id,  0.0 wa,  0.0 hi,  1.0 si,  0.0
st

%Cpu16 : 45.0 us, 15.6 sy,  0.0 ni, 35.6 id,  0.0 wa,  0.0 hi,  3.8 si,  0.0
st

%Cpu17 : 15.5 us,  4.7 sy,  0.0 ni, 79.4 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0
st

%Cpu18 : 44.7 us, 16.5 sy,  0.0 ni, 30.6 id,  0.0 wa,  0.0 hi,  8.1 si,  0.0
st

%Cpu19 : 15.7 us,  4.7 sy,  0.0 ni, 78.6 id,  0.3 wa,  0.0 hi,  0.7 si,  0.0
st

%Cpu20 : 41.4 us, 16.5 sy,  0.0 ni, 36.8 id,  0.0 wa,  0.0 hi,  5.3 si,  0.0
st

%Cpu21 : 16.7 us,  5.1 sy,  0.0 ni, 77.5 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0
st

%Cpu22 : 42.9 us, 15.9 sy,  0.0 ni, 35.3 id,  0.0 wa,  0.0 hi,  5.9 si,  0.0
st

%Cpu23 : 14.9 us,  5.8 sy,  0.0 ni, 79.0 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0
st

%Cpu24 : 45.1 us, 17.8 sy,  0.0 ni, 29.0 id,  0.0 wa,  0.0 hi,  8.0 si,  0.0
st

%Cpu25 : 15.3 us,  5.0 sy,  0.0 ni, 79.3 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0
st

%Cpu26 : 39.8 us, 15.1 sy,  0.0 ni, 38.7 id,  0.0 wa,  0.0 hi,  6.3 si,  0.0
st

%Cpu27 : 16.9 us,  5.4 sy,  0.0 ni, 77.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0
st

%Cpu28 : 45.1 us, 15.6 sy,  0.0 ni, 35.1 id,  0.0 wa,  0.0 hi,  4.2 si,  0.0
st

%Cpu29 : 14.8 us,  5.7 sy,  0.0 ni, 78.2 id,  0.7 wa,  0.0 hi,  0.7 si,  0.0
st

%Cpu30 : 40.9 us, 16.4 sy,  0.0 ni, 36.7 id,  0.0 wa,  0.0 hi,  6.0 si,  0.0
st

%Cpu31 : 13.9 us,  5.1 sy,  0.0 ni, 80.7 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0
st