You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Jeffrey Zeiberg <jz...@gmail.com> on 2019/03/22 13:28:58 UTC
Adding additional default JVM parameters to accumulo-env.sh boosts
performance and prevent crashing of Tservers and masters.
Jeffrey Manno (ASRC Federal) and Jeffrey Zeiberg (ASRC Federal) have
discovered that adding a few new JVM options to the JAVA_OPTS set prevented
crashing and increase system performance.
They were added after line 68 in accumulo-env.sh. They are:
'server'
'-XX:+UseParallelOldGC'
'-XX:NewRatio=3'
'-XX:AggressiveHeap'
The machines we are using are 7 year old machines with 8G of main memory,
between 500G - 1T HDD and Intel I5 or I7 processors.
Maybe these parameters should be made in the Accumulo 2.0 distributions
accumulo-env.sh file?
Comments?
Re: Adding additional default JVM parameters to accumulo-env.sh
boosts performance and prevent crashing of Tservers and masters.
Posted by Christopher <ct...@apache.org>.
Are you using native maps?
Also, I'm in agreement with Josh that specific configuration flags may
not apply to everybody. The conf/accumulo-env.sh script is a
user-controllable file intended to be tweaked by system admins for
their environment. That said, it's possible some of these configs may
make sense for most users, and we'd want to make them default, but I'm
not sure.
On Fri, Mar 22, 2019 at 9:49 AM Jeffrey Manno <je...@gmail.com> wrote:
>
> To add some more detail, we are using Java 1.8 openJDK, and Hadoop 3.2.0.
> These new JVM options were also added to HDFS_DATANODE_OPTS and
> HDFS_NAMENODE_OPTs in hadoop-env,sh. This eliminated a replication warning
> that kept occurring on our cluster.
>
> Here are the rest of the JAVA_OPTS in accumulo-env.sh for reference as what
> is normally there by default:
> '-XX:CMSInitiatingOccupancyFraction=75'
> '-XX:+CMSClassUnloadingEnabled'
> '-XX:OnOutOfMemoryError=kill -9 %p'
> '-XX:-OmitStackTraceInFastThrow'
> '-Djava.net.preferIPv4Stack=true'
> "-Daccumulo.native.lib.path=${lib}/native")
>
>
>
> On Fri, Mar 22, 2019 at 9:29 AM Jeffrey Zeiberg <jz...@gmail.com> wrote:
>
> > Jeffrey Manno (ASRC Federal) and Jeffrey Zeiberg (ASRC Federal) have
> > discovered that adding a few new JVM options to the JAVA_OPTS set prevented
> > crashing and increase system performance.
> >
> > They were added after line 68 in accumulo-env.sh. They are:
> > 'server'
> > '-XX:+UseParallelOldGC'
> > '-XX:NewRatio=3'
> > '-XX:AggressiveHeap'
> >
> > The machines we are using are 7 year old machines with 8G of main memory,
> > between 500G - 1T HDD and Intel I5 or I7 processors.
> >
> > Maybe these parameters should be made in the Accumulo 2.0 distributions
> > accumulo-env.sh file?
> >
> > Comments?
> >
Re: Adding additional default JVM parameters to accumulo-env.sh
boosts performance and prevent crashing of Tservers and masters.
Posted by Jeffrey Manno <je...@gmail.com>.
To add some more detail, we are using Java 1.8 openJDK, and Hadoop 3.2.0.
These new JVM options were also added to HDFS_DATANODE_OPTS and
HDFS_NAMENODE_OPTs in hadoop-env,sh. This eliminated a replication warning
that kept occurring on our cluster.
Here are the rest of the JAVA_OPTS in accumulo-env.sh for reference as what
is normally there by default:
'-XX:CMSInitiatingOccupancyFraction=75'
'-XX:+CMSClassUnloadingEnabled'
'-XX:OnOutOfMemoryError=kill -9 %p'
'-XX:-OmitStackTraceInFastThrow'
'-Djava.net.preferIPv4Stack=true'
"-Daccumulo.native.lib.path=${lib}/native")
On Fri, Mar 22, 2019 at 9:29 AM Jeffrey Zeiberg <jz...@gmail.com> wrote:
> Jeffrey Manno (ASRC Federal) and Jeffrey Zeiberg (ASRC Federal) have
> discovered that adding a few new JVM options to the JAVA_OPTS set prevented
> crashing and increase system performance.
>
> They were added after line 68 in accumulo-env.sh. They are:
> 'server'
> '-XX:+UseParallelOldGC'
> '-XX:NewRatio=3'
> '-XX:AggressiveHeap'
>
> The machines we are using are 7 year old machines with 8G of main memory,
> between 500G - 1T HDD and Intel I5 or I7 processors.
>
> Maybe these parameters should be made in the Accumulo 2.0 distributions
> accumulo-env.sh file?
>
> Comments?
>
Re: Adding additional default JVM parameters to accumulo-env.sh
boosts performance and prevent crashing of Tservers and masters.
Posted by Josh Elser <el...@apache.org>.
Very rarely do JVM GC properties universally apply to all users and
workloads.
I think it would be better to document why these options helped in your
workload. Teach folks how to choose the correct JVM properties for their
workloads is a better way forward, than encouraging folks to treat them
as black-boxes.
While this is extremely in-depth, I like the tone of this blog post:
https://blogs.apache.org/hbase/entry/tuning_g1gc_for_your_hbase. The
authors explain what they observed from a system, what they changed,
what effect that change should have, and, finally, the change that they
observe.
On 3/22/19 9:28 AM, Jeffrey Zeiberg wrote:
> Jeffrey Manno (ASRC Federal) and Jeffrey Zeiberg (ASRC Federal) have
> discovered that adding a few new JVM options to the JAVA_OPTS set prevented
> crashing and increase system performance.
>
> They were added after line 68 in accumulo-env.sh. They are:
> 'server'
> '-XX:+UseParallelOldGC'
> '-XX:NewRatio=3'
> '-XX:AggressiveHeap'
>
> The machines we are using are 7 year old machines with 8G of main memory,
> between 500G - 1T HDD and Intel I5 or I7 processors.
>
> Maybe these parameters should be made in the Accumulo 2.0 distributions
> accumulo-env.sh file?
>
> Comments?
>