You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Jeff Kubina <je...@gmail.com> on 2015/08/17 23:21:40 UTC
Question about configuring the linux niceness of tablet servers?
On each of the processing nodes in our cluster we have running 1) HDFS
(datanode), 2) Accumulo (tablet server), and 3) Hadoop. Since Accumulo
depends on the HDFS, and Hadoop depends on the HDFS and sometimes on
Accumulo, we are considering setting the niceness of HDFS to 0 (the current
value), Accumulo to 1, and Hadoop to 2 on each of the nodes. The objective
is to improve the real time performance of Accumulo.
Does anyone have experience configuring their cluster in a similar manner
that they can share? Are there any serious cons to doing this?
Re: Question about configuring the linux niceness of tablet servers?
Posted by Jeff Kubina <je...@gmail.com>.
More like a mapreduce task process.
--
Jeff Kubina
410-988-4436
On Mon, Aug 17, 2015 at 5:33 PM, William Slacum <ws...@gmail.com> wrote:
> By "Hadoop" do you mean a Yarn NodeManager process?
>
> On Mon, Aug 17, 2015 at 4:21 PM, Jeff Kubina <je...@gmail.com>
> wrote:
>
>> On each of the processing nodes in our cluster we have running 1) HDFS
>> (datanode), 2) Accumulo (tablet server), and 3) Hadoop. Since Accumulo
>> depends on the HDFS, and Hadoop depends on the HDFS and sometimes on
>> Accumulo, we are considering setting the niceness of HDFS to 0 (the current
>> value), Accumulo to 1, and Hadoop to 2 on each of the nodes. The objective
>> is to improve the real time performance of Accumulo.
>>
>> Does anyone have experience configuring their cluster in a similar manner
>> that they can share? Are there any serious cons to doing this?
>>
>>
>
Re: Question about configuring the linux niceness of tablet servers?
Posted by William Slacum <ws...@gmail.com>.
By "Hadoop" do you mean a Yarn NodeManager process?
On Mon, Aug 17, 2015 at 4:21 PM, Jeff Kubina <je...@gmail.com> wrote:
> On each of the processing nodes in our cluster we have running 1) HDFS
> (datanode), 2) Accumulo (tablet server), and 3) Hadoop. Since Accumulo
> depends on the HDFS, and Hadoop depends on the HDFS and sometimes on
> Accumulo, we are considering setting the niceness of HDFS to 0 (the current
> value), Accumulo to 1, and Hadoop to 2 on each of the nodes. The objective
> is to improve the real time performance of Accumulo.
>
> Does anyone have experience configuring their cluster in a similar manner
> that they can share? Are there any serious cons to doing this?
>
>
Re: Question about configuring the linux niceness of tablet servers?
Posted by Russ Weeks <rw...@newbrightidea.com>.
Hi, Jeff,
Yes, we have had mapreduce jobs "lock out" the tserver so long that the
> master removes them from its pool for not responding. But also when doing a
> query from another service while the mapreduce jobs are maxed out I want to
> make sure the tserver takes priority.
>
I've observed this too! Zookeeper connection loss on the tablet server,
right? The problem seems to correlate with times of high load avg (avg > #
cores) on the tablet server. Our fix was to reduce the CPU and RAM
available to YARN. Fewer active containers leads to less CPU contention.
Less CPU contention can also improve throughput on the MR job, so if you
hit the sweet spot it might even improve performance. Definitely interested
to hear alternative solutions though.
Regards,
-Russ
>
>> > However, if you do this, I think you actually want HDFS to -2, Accumulo
>> to -1,
>> > and Mapreduce tasks 0 (you'll have to edit the system security limits
>> to allow
>> > niceness below 0 for non-root users).
>> > Otherwise, you're lowering the priority, not increasing it (and
>> lowering it
>> > below other, normal user processes, since the default niceness is 0).
>>
>
> Agreed. That is what I wanted to do originally but it requires much more
> reconfiguration. Ideally all OS root processes currently at 0 get reniced
> to -2, HDFS to -2, Accumulo to -2, and mapreduce to -1 and all other user
> processes stay at 0. Given the amount of threading a datanode does by
> default it could lockout a tserver if its priority was lower, so I think
> the HDFS and Accumulo should have equal priority.
>
>
Re: Question about configuring the linux niceness of tablet servers?
Posted by Christopher <ct...@apache.org>.
So it sounds like what's going to have the biggest effect is making sure
the problematic mapreduce tasks run at a lower priority. I'd definitely be
interested in your results of you experiment with this.
On Tue, Aug 18, 2015, 17:33 Jeff Kubina <je...@gmail.com> wrote:
> > I haven't heard of anybody setting the niceness of the Accumulo processes
>> > before. Are you experiencing a lot of CPU contention on your nodes, such
>> > that you need to prioritize processes?
>>
>
> Yes, we have had mapreduce jobs "lock out" the tserver so long that the
> master removes them from its pool for not responding. But also when doing a
> query from another service while the mapreduce jobs are maxed out I want to
> make sure the tserver takes priority.
>
>
>> > However, if you do this, I think you actually want HDFS to -2, Accumulo
>> to -1,
>> > and Mapreduce tasks 0 (you'll have to edit the system security limits
>> to allow
>> > niceness below 0 for non-root users).
>> > Otherwise, you're lowering the priority, not increasing it (and
>> lowering it
>> > below other, normal user processes, since the default niceness is 0).
>>
>
> Agreed. That is what I wanted to do originally but it requires much more
> reconfiguration. Ideally all OS root processes currently at 0 get reniced
> to -2, HDFS to -2, Accumulo to -2, and mapreduce to -1 and all other user
> processes stay at 0. Given the amount of threading a datanode does by
> default it could lockout a tserver if its priority was lower, so I think
> the HDFS and Accumulo should have equal priority.
>
>
Re: Question about configuring the linux niceness of tablet servers?
Posted by Jeff Kubina <je...@gmail.com>.
> > I haven't heard of anybody setting the niceness of the Accumulo processes
> > before. Are you experiencing a lot of CPU contention on your nodes, such
> > that you need to prioritize processes?
>
Yes, we have had mapreduce jobs "lock out" the tserver so long that the
master removes them from its pool for not responding. But also when doing a
query from another service while the mapreduce jobs are maxed out I want to
make sure the tserver takes priority.
> > However, if you do this, I think you actually want HDFS to -2, Accumulo
> to -1,
> > and Mapreduce tasks 0 (you'll have to edit the system security limits to
> allow
> > niceness below 0 for non-root users).
> > Otherwise, you're lowering the priority, not increasing it (and lowering
> it
> > below other, normal user processes, since the default niceness is 0).
>
Agreed. That is what I wanted to do originally but it requires much more
reconfiguration. Ideally all OS root processes currently at 0 get reniced
to -2, HDFS to -2, Accumulo to -2, and mapreduce to -1 and all other user
processes stay at 0. Given the amount of threading a datanode does by
default it could lockout a tserver if its priority was lower, so I think
the HDFS and Accumulo should have equal priority.
RE: Question about configuring the linux niceness of tablet servers?
Posted by dl...@comcast.net.
Check out the implementation of https://issues.apache.org/jira/browse/ACCUMULO-3793. You could do something similar and start your process with 'nice' instead of 'numactl'.
> -----Original Message-----
> From: Christopher [mailto:ctubbsii@apache.org]
> Sent: Monday, August 17, 2015 7:15 PM
> To: Accumulo User List
> Subject: Re: Question about configuring the linux niceness of tablet servers?
>
> I haven't heard of anybody setting the niceness of the Accumulo processes
> before. Are you experiencing a lot of CPU contention on your nodes, such
> that you need to prioritize processes?
>
> However, if you do this, I think you actually want HDFS to -2, Accumulo to -1,
> and Mapreduce tasks 0 (you'll have to edit the system security limits to allow
> niceness below 0 for non-root users).
> Otherwise, you're lowering the priority, not increasing it (and lowering it
> below other, normal user processes, since the default niceness is 0).
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Mon, Aug 17, 2015 at 5:21 PM, Jeff Kubina <je...@gmail.com>
> wrote:
> > On each of the processing nodes in our cluster we have running 1) HDFS
> > (datanode), 2) Accumulo (tablet server), and 3) Hadoop. Since Accumulo
> > depends on the HDFS, and Hadoop depends on the HDFS and sometimes
> on
> > Accumulo, we are considering setting the niceness of HDFS to 0 (the
> > current value), Accumulo to 1, and Hadoop to 2 on each of the nodes.
> > The objective is to improve the real time performance of Accumulo.
> >
> > Does anyone have experience configuring their cluster in a similar
> > manner that they can share? Are there any serious cons to doing this?
> >
Re: Question about configuring the linux niceness of tablet servers?
Posted by Christopher <ct...@apache.org>.
I haven't heard of anybody setting the niceness of the Accumulo
processes before. Are you experiencing a lot of CPU contention on your
nodes, such that you need to prioritize processes?
However, if you do this, I think you actually want HDFS to -2,
Accumulo to -1, and Mapreduce tasks 0 (you'll have to edit the system
security limits to allow niceness below 0 for non-root users).
Otherwise, you're lowering the priority, not increasing it (and
lowering it below other, normal user processes, since the default
niceness is 0).
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Mon, Aug 17, 2015 at 5:21 PM, Jeff Kubina <je...@gmail.com> wrote:
> On each of the processing nodes in our cluster we have running 1) HDFS
> (datanode), 2) Accumulo (tablet server), and 3) Hadoop. Since Accumulo
> depends on the HDFS, and Hadoop depends on the HDFS and sometimes on
> Accumulo, we are considering setting the niceness of HDFS to 0 (the current
> value), Accumulo to 1, and Hadoop to 2 on each of the nodes. The objective
> is to improve the real time performance of Accumulo.
>
> Does anyone have experience configuring their cluster in a similar manner
> that they can share? Are there any serious cons to doing this?
>