You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Dickson, Matt MR" <ma...@defence.gov.au> on 2017/01/16 22:31:05 UTC
Merging smaller/empty tablets [SEC=UNOFFICIAL]
UNOFFICIAL
I have a table that has evolved to have 1.07T tablets and I fairly confident a large portion of these are now empty or very small. I'd like to merge smaller tablets and delete empty tablets, is there a smart way to do this?
My thought was to query the metadata table for all tablets under a certain size for the table and then merge these tablets.
Is the first number in thevalue the size of the tablet, ie
> scan -b 1xk -e 1xk\xff -c file
1xk;34234 file:hdfs//name/accumulo/tables/1xk/t-er23423/M423432.rf [] 213134,234234
Also, are there any side effects of this that I need to be aware of when doing this on a massive table?
Thanks in advance,
Matt
Re: Merging smaller/empty tablets [SEC=UNOFFICIAL]
Posted by Josh Elser <jo...@gmail.com>.
Just to clarify: compactions don't implicitly change the table
"distribution". The number and/or boundaries of tablets don't change as
a part of a compaction.
Yamini Joshi wrote:
> Just a thought, will forcing a major compaction take care of this?
> Merging smaller tablets and deleting empty ones?
>
> Best regards,
> Yamini Joshi
>
> On Mon, Jan 16, 2017 at 4:31 PM, Dickson, Matt MR
> <matt.dickson@defence.gov.au <ma...@defence.gov.au>> wrote:
>
> __
>
> *UNOFFICIAL*
>
> I have a table that has evolved to have 1.07T tablets and I fairly
> confident a large portion of these are now empty or very small. I'd
> like to merge smaller tablets and delete empty tablets, is there a
> smart way to do this?
> My thought was to query the metadata table for all tablets under a
> certain size for the table and thenmerge these tablets.
> Is the first number in thevalue the size of the tablet, ie
> > scan -b 1xk -e 1xk\xff -c file
> 1xk;34234 file:hdfs//name/accumulo/tables/1xk/t-er23423/M423432.rf
> [] *213134*,234234
> Also, are there any side effects of this that I need to be aware of
> when doing this on a massive table?
> Thanks in advance,
> Matt
>
>
Re: Merging smaller/empty tablets [SEC=UNOFFICIAL]
Posted by Yamini Joshi <ya...@gmail.com>.
Just a thought, will forcing a major compaction take care of this? Merging
smaller tablets and deleting empty ones?
Best regards,
Yamini Joshi
On Mon, Jan 16, 2017 at 4:31 PM, Dickson, Matt MR <
matt.dickson@defence.gov.au> wrote:
> *UNOFFICIAL*
> I have a table that has evolved to have 1.07T tablets and I fairly
> confident a large portion of these are now empty or very small. I'd like
> to merge smaller tablets and delete empty tablets, is there a smart way to
> do this?
>
> My thought was to query the metadata table for all tablets under a
> certain size for the table and then merge these tablets.
>
> Is the first number in thevalue the size of the tablet, ie
>
> > scan -b 1xk -e 1xk\xff -c file
> 1xk;34234 file:hdfs//name/accumulo/tables/1xk/t-er23423/M423432.rf []
> *213134*,234234
>
> Also, are there any side effects of this that I need to be aware of when
> doing this on a massive table?
>
> Thanks in advance,
> Matt
>
Re: Merging smaller/empty tablets [SEC=UNOFFICIAL]
Posted by Michael Wall <mj...@gmail.com>.
Matt,
Using the merge command can be a little tricky. Imagine your splits look
like this for whatever table 1 is using 'scan -t accumulo.metadata -b 1; -e
2; -c ~tab'
1;a ~tab:~pr [] \x00
1;b ~tab:~pr [] \x01a
1;c ~tab:~pr [] \x01b
1;d ~tab:~pr [] \x01c
1;e ~tab:~pr [] \x01d
1;f ~tab:~pr [] \x01e
1;g ~tab:~pr [] \x01f
1;h ~tab:~pr [] \x01g
1;i ~tab:~pr [] \x01h
1;j ~tab:~pr [] \x01i
1;k ~tab:~pr [] \x01j
1< ~tab:~pr [] \x01k
Merging will take the entire range you specify, and put all the splits into
the last split. Say you want to merge split c and d into e. You might try
this command, but it would be wrong
'merge -t test1 -b c -e e'
because the -b is exclusive. That command would merge d into e but skip
c. A correct command would be something like
'merge -t test1 -b b\x00 -e e'
After this would end up with the following splits
1;a ~tab:~pr [] \x00
1;b ~tab:~pr [] \x01a
1;e ~tab:~pr [] \x01b
1;f ~tab:~pr [] \x01e
1;g ~tab:~pr [] \x01f
1;h ~tab:~pr [] \x01g
1;i ~tab:~pr [] \x01h
1;j ~tab:~pr [] \x01i
1;k ~tab:~pr [] \x01j
1< ~tab:~pr [] \x01k
where the begin range is a value after the ~tab:~pr value for the range you
want to start with.
Also, merges can be really slow. Much of this comes from having to look at
every rfile and doing a special "chop" compaction to pull out what doesn't
belong in the merged range. This can be avoided if you do a full major
compaction on the ranges you are going to merge. This get each split down
to 1 rfile and no "chop" compaction is done.
Suggest you start slow merging one split into another to get a feel for how
it goes and how long it takes.
Good luck.
Mike
On Mon, Jan 16, 2017 at 10:57 PM, Mike Drob <md...@mdrob.com> wrote:
> I can't find this in the docs, but IIRC the merge command can take a
> start/end range for what to merge. So the best option might be to try it on
> a smaller slice and see what happens. At a guess, queries won't block but
> indexing will.
>
> Mike
>
> On Mon, Jan 16, 2017 at 5:23 PM, Dickson, Matt MR <
> matt.dickson@defence.gov.au> wrote:
>
>> *UNOFFICIAL*
>> That looks like a great option. Before using it, whats the cost/impact
>> of running this on a massive table in a system with other large bulk
>> ingests/queries running? In the past when I have used that (which was in
>> 2013 so things may have changed) all ingests were blocked and it took days
>> to complete.
>>
>> With 1.07T tablets to work on this may take some time?
>>
>>
>> ------------------------------
>> *From:* Mike Drob [mailto:mdrob@mdrob.com]
>> *Sent:* Tuesday, 17 January 2017 09:37
>> *To:* user@accumulo.apache.org
>> *Subject:* Re: Merging smaller/empty tablets [SEC=UNOFFICIAL]
>>
>> http://accumulo.apache.org/1.8/accumulo_user_manual.html#_merging_tablets
>>
>> In order to merge small tablets, you can ask Accumulo to merge sections
>> of a table smaller than a given size.
>>
>> root@myinstance> merge -t myTable -s 100M
>>
>>
>>
>> On Mon, Jan 16, 2017 at 4:31 PM, Dickson, Matt MR <
>> matt.dickson@defence.gov.au> wrote:
>>
>>> *UNOFFICIAL*
>>> I have a table that has evolved to have 1.07T tablets and I fairly
>>> confident a large portion of these are now empty or very small. I'd like
>>> to merge smaller tablets and delete empty tablets, is there a smart way to
>>> do this?
>>>
>>> My thought was to query the metadata table for all tablets under a
>>> certain size for the table and then merge these tablets.
>>>
>>> Is the first number in thevalue the size of the tablet, ie
>>>
>>> > scan -b 1xk -e 1xk\xff -c file
>>> 1xk;34234 file:hdfs//name/accumulo/tables/1xk/t-er23423/M423432.rf
>>> [] *213134*,234234
>>>
>>> Also, are there any side effects of this that I need to be aware of when
>>> doing this on a massive table?
>>>
>>> Thanks in advance,
>>> Matt
>>>
>>
>>
>
Re: Merging smaller/empty tablets [SEC=UNOFFICIAL]
Posted by Mike Drob <md...@mdrob.com>.
I can't find this in the docs, but IIRC the merge command can take a
start/end range for what to merge. So the best option might be to try it on
a smaller slice and see what happens. At a guess, queries won't block but
indexing will.
Mike
On Mon, Jan 16, 2017 at 5:23 PM, Dickson, Matt MR <
matt.dickson@defence.gov.au> wrote:
> *UNOFFICIAL*
> That looks like a great option. Before using it, whats the cost/impact
> of running this on a massive table in a system with other large bulk
> ingests/queries running? In the past when I have used that (which was in
> 2013 so things may have changed) all ingests were blocked and it took days
> to complete.
>
> With 1.07T tablets to work on this may take some time?
>
>
> ------------------------------
> *From:* Mike Drob [mailto:mdrob@mdrob.com]
> *Sent:* Tuesday, 17 January 2017 09:37
> *To:* user@accumulo.apache.org
> *Subject:* Re: Merging smaller/empty tablets [SEC=UNOFFICIAL]
>
> http://accumulo.apache.org/1.8/accumulo_user_manual.html#_merging_tablets
>
> In order to merge small tablets, you can ask Accumulo to merge sections of
> a table smaller than a given size.
>
> root@myinstance> merge -t myTable -s 100M
>
>
>
> On Mon, Jan 16, 2017 at 4:31 PM, Dickson, Matt MR <
> matt.dickson@defence.gov.au> wrote:
>
>> *UNOFFICIAL*
>> I have a table that has evolved to have 1.07T tablets and I fairly
>> confident a large portion of these are now empty or very small. I'd like
>> to merge smaller tablets and delete empty tablets, is there a smart way to
>> do this?
>>
>> My thought was to query the metadata table for all tablets under a
>> certain size for the table and then merge these tablets.
>>
>> Is the first number in thevalue the size of the tablet, ie
>>
>> > scan -b 1xk -e 1xk\xff -c file
>> 1xk;34234 file:hdfs//name/accumulo/tables/1xk/t-er23423/M423432.rf []
>> *213134*,234234
>>
>> Also, are there any side effects of this that I need to be aware of when
>> doing this on a massive table?
>>
>> Thanks in advance,
>> Matt
>>
>
>
RE: Merging smaller/empty tablets [SEC=UNOFFICIAL]
Posted by "Dickson, Matt MR" <ma...@defence.gov.au>.
UNOFFICIAL
That looks like a great option. Before using it, whats the cost/impact of running this on a massive table in a system with other large bulk ingests/queries running? In the past when I have used that (which was in 2013 so things may have changed) all ingests were blocked and it took days to complete.
With 1.07T tablets to work on this may take some time?
________________________________
From: Mike Drob [mailto:mdrob@mdrob.com]
Sent: Tuesday, 17 January 2017 09:37
To: user@accumulo.apache.org
Subject: Re: Merging smaller/empty tablets [SEC=UNOFFICIAL]
http://accumulo.apache.org/1.8/accumulo_user_manual.html#_merging_tablets
In order to merge small tablets, you can ask Accumulo to merge sections of a table smaller than a given size.
root@myinstance> merge -t myTable -s 100M
On Mon, Jan 16, 2017 at 4:31 PM, Dickson, Matt MR <ma...@defence.gov.au>> wrote:
UNOFFICIAL
I have a table that has evolved to have 1.07T tablets and I fairly confident a large portion of these are now empty or very small. I'd like to merge smaller tablets and delete empty tablets, is there a smart way to do this?
My thought was to query the metadata table for all tablets under a certain size for the table and then merge these tablets.
Is the first number in thevalue the size of the tablet, ie
> scan -b 1xk -e 1xk\xff -c file
1xk;34234 file:hdfs//name/accumulo/tables/1xk/t-er23423/M423432.rf [] 213134,234234
Also, are there any side effects of this that I need to be aware of when doing this on a massive table?
Thanks in advance,
Matt
Re: Merging smaller/empty tablets [SEC=UNOFFICIAL]
Posted by Mike Drob <md...@mdrob.com>.
http://accumulo.apache.org/1.8/accumulo_user_manual.html#_merging_tablets
In order to merge small tablets, you can ask Accumulo to merge sections of
a table smaller than a given size.
root@myinstance> merge -t myTable -s 100M
On Mon, Jan 16, 2017 at 4:31 PM, Dickson, Matt MR <
matt.dickson@defence.gov.au> wrote:
> *UNOFFICIAL*
> I have a table that has evolved to have 1.07T tablets and I fairly
> confident a large portion of these are now empty or very small. I'd like
> to merge smaller tablets and delete empty tablets, is there a smart way to
> do this?
>
> My thought was to query the metadata table for all tablets under a
> certain size for the table and then merge these tablets.
>
> Is the first number in thevalue the size of the tablet, ie
>
> > scan -b 1xk -e 1xk\xff -c file
> 1xk;34234 file:hdfs//name/accumulo/tables/1xk/t-er23423/M423432.rf []
> *213134*,234234
>
> Also, are there any side effects of this that I need to be aware of when
> doing this on a massive table?
>
> Thanks in advance,
> Matt
>