Tuned performance profiles for Accumulo

[Christopher <ct...@apache.org>]

Has anybody used tuned ("tune D") to manage their system performance
profiles on an Accumulo cluster?

I've recently been looking into tuned, and found it a very convenient tool
for switching between performance profiles, and verifying the current
configuration. It beats manually setting sysctl settings (which I usually
forget to do right away).

I haven't actually created my own tuned profiles, though, because I'm not
an expert on Linux system tuning. However, I have found the built-in
latency-network profile to be useful.

Has anybody tried a custom profile (for Accumulo specifically, or Hadoop
clusters in general)?

Has anybody else found using tuned profiles to be a useful way to manage
(CM and verification) system configuration for your clusters?

Re: Tuned performance profiles for Accumulo

[David Boyd <db...@incadencecorp.com>]

On 8/30/16 6:05 PM, Jeff Kubina wrote:
>
> I have created tuned profiles for our datanodes (tservers, 
> proc_datanode, mesos-slaves) and various super nodes (namenodes, 
> accumulo master, resource managers, zookeepers, kafka, etc). I use 
> tuned for static tuning only even though it has dynamic tuning 
> available. Each profile is designed for maximum performance of the 
> processes it runs.
>
Jeff:  It would be great if you could share those or at least document 
recommendations.

-- 
========= mailto:dboyd@incadencecorp.com ============
David W. Boyd
VP,  Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office:   +1-703-552-2862
cell:     +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
Chair ANSI/INCITS TC Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org

The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited.  If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.

  

Re: Tuned performance profiles for Accumulo

[Jeff Kubina <je...@gmail.com>]

On Tue, Aug 30, 2016 at 3:51 PM, Christopher <ct...@apache.org> wrote:

> Has anybody used tuned ("tune D") to manage their system performance
> profiles on an Accumulo cluster?
>

Yes, we use it a lot.


> I've recently been looking into tuned, and found it a very convenient tool
> for switching between performance profiles, and verifying the current
> configuration. It beats manually setting sysctl settings (which I usually
> forget to do right away).
>

It does enable tuning of many aspects of the system in one place which is
nice. The only tuning that I had to move to the grub files was turning off
transparent huge pages. Upon startup some processes were able to allocate
THP before tuned disabled them.

I haven't actually created my own tuned profiles, though, because I'm not
> an expert on Linux system tuning. However, I have found the built-in
> latency-network profile to be useful.
>


> Has anybody tried a custom profile (for Accumulo specifically, or Hadoop
> clusters in general)?
>

I have created tuned profiles for our datanodes (tservers, proc_datanode,
mesos-slaves) and various super nodes (namenodes, accumulo master, resource
managers, zookeepers, kafka, etc). I use tuned for static tuning only even
though it has dynamic tuning available. Each profile is designed for
maximum performance of the processes it runs.


> Has anybody else found using tuned profiles to be a useful way to manage
> (CM and verification) system configuration for your clusters?
>

Yes, but just for tuning and there is some tuning that just can be done
using it. For example, mounting datanode drives with noatime. We use
kickstart, salt, and Cloudera Manager for all of our configuration
management and verification. We push out our tuned profiles via salt.