You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Dickson, Matt MR" <ma...@defence.gov.au> on 2013/08/29 07:39:45 UTC
High Ingest on a single server [SEC=UNOFFICIAL]
UNOFFICIAL
We are seeing a single server that has less entries than all the other nodes in the cluster. Accumulo now appears to be directing higher ingest tablets to this node and its now getting 7 times the ingest entries than all other nodes and is slowing or load. Does Accumulo attempt to balance disk usage across the nodes for a table by moving tablets and that is why we are seeing this node ingesting more?
If not, is it possible to make accumulo rebalance the ingest across all servers during a load?
Matt
RE: High Ingest on a single server [SEC=UNOFFICIAL]
Posted by Josh Elser <jo...@gmail.com>.
Do you have unhosted tablets (check the monitor)?
If so and you're doing live ingest (sorry, I forget if you've already
said), loggers might have died whose WAL files you need. Try to quiesce
reads/writes, do a start-all.sh and try to get all the tablets assigned.
The monitor should be helpful if its something more than missing WAL files
due to dead loggers.
Then, see if the master starts balancing things again.
On Sep 1, 2013 10:59 PM, "Dickson, Matt MR" <ma...@defence.gov.au>
wrote:
> UNOFFICIAL
>
> Just checked and there are a lot of 'not balancing because there are
> unhosted tablets' debug messages. Is this the same issue?
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Monday, 2 September 2013 12:52
> To: user@accumulo.apache.org
> Subject: Re: High Ingest on a single server [SEC=UNOFFICIAL]
>
> To verify that this what you're running into, you should see a message in
> the master*.debug.log that matches "not balancing because.*"
>
> On 09/01/2013 09:54 PM, John Vines wrote:
> > Try restarting the master. A few releases had a big where it would get
> > stuck.
> >
> > Sent from my phone, please pardon the typos and brevity.
> >
> > On Sep 1, 2013 6:12 PM, "Dickson, Matt MR"
> > <matt.dickson@defence.gov.au <ma...@defence.gov.au>>
> wrote:
> >
> > __
> >
> > *UNOFFICIAL*
> >
> > Thanks Eric.
> > The tablet of concern has 2000 tablets while all others have 1000 so
> > it looks like the balancers aren't evening out the tablets per node
> > as expected.
> > Is there a way to force the balancer to run or rectify this
> > situation by moving tablets to alternate nodes?
> >
> >
> ------------------------------------------------------------------------
> > *From:* Eric Newton [mailto:eric.newton@gmail.com
> > <ma...@gmail.com>]
> > *Sent:* Thursday, 29 August 2013 23:23
> > *To:* user@accumulo.apache.org <ma...@accumulo.apache.org>
> > *Subject:* Re: High Ingest on a single server [SEC=UNOFFICIAL]
> >
> > The balancers that ship with accumulo attempt to keep an equal
> > number of tablets on each server. An empty tablet, will be balanced
> > with the same weight as a 50G tablet.
> >
> > You can write a new balancer to take advantage of the properties of
> > the tablets, and any expected hotspots you have.
> >
> >
> >
> > On Thu, Aug 29, 2013 at 1:39 AM, Dickson, Matt MR
> > <matt.dickson@defence.gov.au <ma...@defence.gov.au>>
> > wrote:
> >
> > __
> >
> > *UNOFFICIAL*
> >
> > We are seeing a single server that has less entries than all the
> > other nodes in the cluster. Accumulo now appears to be
> > directing higher ingest tablets to this node and its now getting
> > 7 times the ingest entries than all other nodes and is slowing
> > or load. Does Accumulo attempt to balance disk usage across the
> > nodes for a table by moving tablets and that is why we are
> > seeing this node ingesting more?
> > If not, is it possible to make accumulo rebalance the ingest
> > across all servers during a load?
> > Matt
> >
> >
>
RE: High Ingest on a single server [SEC=UNOFFICIAL]
Posted by "Dickson, Matt MR" <ma...@defence.gov.au>.
UNOFFICIAL
Just checked and there are a lot of 'not balancing because there are unhosted tablets' debug messages. Is this the same issue?
-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com]
Sent: Monday, 2 September 2013 12:52
To: user@accumulo.apache.org
Subject: Re: High Ingest on a single server [SEC=UNOFFICIAL]
To verify that this what you're running into, you should see a message in the master*.debug.log that matches "not balancing because.*"
On 09/01/2013 09:54 PM, John Vines wrote:
> Try restarting the master. A few releases had a big where it would get
> stuck.
>
> Sent from my phone, please pardon the typos and brevity.
>
> On Sep 1, 2013 6:12 PM, "Dickson, Matt MR"
> <matt.dickson@defence.gov.au <ma...@defence.gov.au>> wrote:
>
> __
>
> *UNOFFICIAL*
>
> Thanks Eric.
> The tablet of concern has 2000 tablets while all others have 1000 so
> it looks like the balancers aren't evening out the tablets per node
> as expected.
> Is there a way to force the balancer to run or rectify this
> situation by moving tablets to alternate nodes?
>
> ------------------------------------------------------------------------
> *From:* Eric Newton [mailto:eric.newton@gmail.com
> <ma...@gmail.com>]
> *Sent:* Thursday, 29 August 2013 23:23
> *To:* user@accumulo.apache.org <ma...@accumulo.apache.org>
> *Subject:* Re: High Ingest on a single server [SEC=UNOFFICIAL]
>
> The balancers that ship with accumulo attempt to keep an equal
> number of tablets on each server. An empty tablet, will be balanced
> with the same weight as a 50G tablet.
>
> You can write a new balancer to take advantage of the properties of
> the tablets, and any expected hotspots you have.
>
>
>
> On Thu, Aug 29, 2013 at 1:39 AM, Dickson, Matt MR
> <matt.dickson@defence.gov.au <ma...@defence.gov.au>>
> wrote:
>
> __
>
> *UNOFFICIAL*
>
> We are seeing a single server that has less entries than all the
> other nodes in the cluster. Accumulo now appears to be
> directing higher ingest tablets to this node and its now getting
> 7 times the ingest entries than all other nodes and is slowing
> or load. Does Accumulo attempt to balance disk usage across the
> nodes for a table by moving tablets and that is why we are
> seeing this node ingesting more?
> If not, is it possible to make accumulo rebalance the ingest
> across all servers during a load?
> Matt
>
>
Re: High Ingest on a single server [SEC=UNOFFICIAL]
Posted by Josh Elser <jo...@gmail.com>.
To verify that this what you're running into, you should see a message
in the master*.debug.log that matches "not balancing because.*"
On 09/01/2013 09:54 PM, John Vines wrote:
> Try restarting the master. A few releases had a big where it would get
> stuck.
>
> Sent from my phone, please pardon the typos and brevity.
>
> On Sep 1, 2013 6:12 PM, "Dickson, Matt MR" <matt.dickson@defence.gov.au
> <ma...@defence.gov.au>> wrote:
>
> __
>
> *UNOFFICIAL*
>
> Thanks Eric.
> The tablet of concern has 2000 tablets while all others have 1000 so
> it looks like the balancers aren't evening out the tablets per node
> as expected.
> Is there a way to force the balancer to run or rectify this
> situation by moving tablets to alternate nodes?
>
> ------------------------------------------------------------------------
> *From:* Eric Newton [mailto:eric.newton@gmail.com
> <ma...@gmail.com>]
> *Sent:* Thursday, 29 August 2013 23:23
> *To:* user@accumulo.apache.org <ma...@accumulo.apache.org>
> *Subject:* Re: High Ingest on a single server [SEC=UNOFFICIAL]
>
> The balancers that ship with accumulo attempt to keep an equal
> number of tablets on each server. An empty tablet, will be balanced
> with the same weight as a 50G tablet.
>
> You can write a new balancer to take advantage of the properties of
> the tablets, and any expected hotspots you have.
>
>
>
> On Thu, Aug 29, 2013 at 1:39 AM, Dickson, Matt MR
> <matt.dickson@defence.gov.au <ma...@defence.gov.au>>
> wrote:
>
> __
>
> *UNOFFICIAL*
>
> We are seeing a single server that has less entries than all the
> other nodes in the cluster. Accumulo now appears to be
> directing higher ingest tablets to this node and its now getting
> 7 times the ingest entries than all other nodes and is slowing
> or load. Does Accumulo attempt to balance disk usage across the
> nodes for a table by moving tablets and that is why we are
> seeing this node ingesting more?
> If not, is it possible to make accumulo rebalance the ingest
> across all servers during a load?
> Matt
>
>
RE: High Ingest on a single server [SEC=UNOFFICIAL]
Posted by John Vines <vi...@apache.org>.
Try restarting the master. A few releases had a big where it would get
stuck.
Sent from my phone, please pardon the typos and brevity.
On Sep 1, 2013 6:12 PM, "Dickson, Matt MR" <ma...@defence.gov.au>
wrote:
> **
>
> *UNOFFICIAL*
> Thanks Eric.
>
> The tablet of concern has 2000 tablets while all others have 1000 so it
> looks like the balancers aren't evening out the tablets per node as
> expected.
>
> Is there a way to force the balancer to run or rectify this situation by
> moving tablets to alternate nodes?
>
>
>
>
> ------------------------------
> *From:* Eric Newton [mailto:eric.newton@gmail.com]
> *Sent:* Thursday, 29 August 2013 23:23
> *To:* user@accumulo.apache.org
> *Subject:* Re: High Ingest on a single server [SEC=UNOFFICIAL]
>
> The balancers that ship with accumulo attempt to keep an equal number of
> tablets on each server. An empty tablet, will be balanced with the same
> weight as a 50G tablet.
>
> You can write a new balancer to take advantage of the properties of the
> tablets, and any expected hotspots you have.
>
>
>
> On Thu, Aug 29, 2013 at 1:39 AM, Dickson, Matt MR <
> matt.dickson@defence.gov.au> wrote:
>
>> **
>>
>> *UNOFFICIAL*
>> We are seeing a single server that has less entries than all the other
>> nodes in the cluster. Accumulo now appears to be directing higher ingest
>> tablets to this node and its now getting 7 times the ingest entries than
>> all other nodes and is slowing or load. Does Accumulo attempt to balance
>> disk usage across the nodes for a table by moving tablets and that is why
>> we are seeing this node ingesting more?
>>
>> If not, is it possible to make accumulo rebalance the ingest across all
>> servers during a load?
>>
>> Matt
>>
>
>
RE: High Ingest on a single server [SEC=UNOFFICIAL]
Posted by "Dickson, Matt MR" <ma...@defence.gov.au>.
UNOFFICIAL
Thanks Eric.
The tablet of concern has 2000 tablets while all others have 1000 so it looks like the balancers aren't evening out the tablets per node as expected.
Is there a way to force the balancer to run or rectify this situation by moving tablets to alternate nodes?
________________________________
From: Eric Newton [mailto:eric.newton@gmail.com]
Sent: Thursday, 29 August 2013 23:23
To: user@accumulo.apache.org
Subject: Re: High Ingest on a single server [SEC=UNOFFICIAL]
The balancers that ship with accumulo attempt to keep an equal number of tablets on each server. An empty tablet, will be balanced with the same weight as a 50G tablet.
You can write a new balancer to take advantage of the properties of the tablets, and any expected hotspots you have.
On Thu, Aug 29, 2013 at 1:39 AM, Dickson, Matt MR <ma...@defence.gov.au>> wrote:
UNOFFICIAL
We are seeing a single server that has less entries than all the other nodes in the cluster. Accumulo now appears to be directing higher ingest tablets to this node and its now getting 7 times the ingest entries than all other nodes and is slowing or load. Does Accumulo attempt to balance disk usage across the nodes for a table by moving tablets and that is why we are seeing this node ingesting more?
If not, is it possible to make accumulo rebalance the ingest across all servers during a load?
Matt
Re: High Ingest on a single server [SEC=UNOFFICIAL]
Posted by Eric Newton <er...@gmail.com>.
The balancers that ship with accumulo attempt to keep an equal number of
tablets on each server. An empty tablet, will be balanced with the same
weight as a 50G tablet.
You can write a new balancer to take advantage of the properties of the
tablets, and any expected hotspots you have.
On Thu, Aug 29, 2013 at 1:39 AM, Dickson, Matt MR <
matt.dickson@defence.gov.au> wrote:
> **
>
> *UNOFFICIAL*
> We are seeing a single server that has less entries than all the other
> nodes in the cluster. Accumulo now appears to be directing higher ingest
> tablets to this node and its now getting 7 times the ingest entries than
> all other nodes and is slowing or load. Does Accumulo attempt to balance
> disk usage across the nodes for a table by moving tablets and that is why
> we are seeing this node ingesting more?
>
> If not, is it possible to make accumulo rebalance the ingest across all
> servers during a load?
>
> Matt
>
Re: High Ingest on a single server [SEC=UNOFFICIAL]
Posted by David Medinets <da...@gmail.com>.
Accumulo can't use its quantum probability features until you stop
monitoring it. :)
On Thu, Aug 29, 2013 at 1:39 AM, Dickson, Matt MR <
matt.dickson@defence.gov.au> wrote:
> **
>
> *UNOFFICIAL*
> We are seeing a single server that has less entries than all the other
> nodes in the cluster. Accumulo now appears to be directing higher ingest
> tablets to this node and its now getting 7 times the ingest entries than
> all other nodes and is slowing or load. Does Accumulo attempt to balance
> disk usage across the nodes for a table by moving tablets and that is why
> we are seeing this node ingesting more?
>
> If not, is it possible to make accumulo rebalance the ingest across all
> servers during a load?
>
> Matt
>