You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (Created) (JIRA)" <ji...@apache.org> on 2012/01/10 01:24:39 UTC
[jira] [Created] (HBASE-5161) Compaction algorithm should
prioritize reference files
Compaction algorithm should prioritize reference files
------------------------------------------------------
Key: HBASE-5161
URL: https://issues.apache.org/jira/browse/HBASE-5161
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Priority: Minor
Fix For: 0.94.0, 0.92.1
I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5161) Compaction algorithm should
prioritize reference files
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262407#comment-13262407 ]
ramkrishna.s.vasudevan commented on HBASE-5161:
-----------------------------------------------
@Stack and @J-D
We seem to end up in the same problem. We had some 32 reference files created out of which one was never selected in further compaction cycles.
We even tried to stop the writes for 2 to 3 hours but still the compaction did not pick up the reference file. Will dig in more into this.
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1, 0.94.0
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-5161) Compaction algorithm should
prioritize reference files
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan reopened HBASE-5161:
-------------------------------------------
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1, 0.94.0
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-5161) Compaction algorithm
should prioritize reference files
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262407#comment-13262407 ]
ramkrishna.s.vasudevan edited comment on HBASE-5161 at 4/26/12 6:36 AM:
------------------------------------------------------------------------
@Stack and @J-D
We seem to end up in the same problem. We had some 32 reference files created out of which one was never selected in further compaction cycles.
We even tried to stop the writes for 2 to 3 hours but still the compaction did not pick up the reference file.
The region grew upto 400G.
{code}
./hbase-root-regionserver-HOST-192-168-47-205.log:2012-04-26 09:44:15,835 DEBUG org.apache.hadoop.hbase.regionserver.Store: hdfs://10.18.40.217:9000/hbase/ufdr/ce5c144a1714df08db1132238a749116/value/cde90029ecb74ef791500ccd3a1e8908.755d1cf6b960c02cc72c1dd83551df82-hdfs://10.18.40.217:9000/hbase/ufdr/755d1cf6b960c02cc72c1dd83551df82/value/cde90029ecb74ef791500ccd3a1e8908-top is not splittable
{code}
We get the above logs for almost 2 to 3 hours. The pair of this reference file (its bottom) is also not compacted.
Will dig in more to find any other reason for not getting picked up.
was (Author: ram_krish):
@Stack and @J-D
We seem to end up in the same problem. We had some 32 reference files created out of which one was never selected in further compaction cycles.
We even tried to stop the writes for 2 to 3 hours but still the compaction did not pick up the reference file. Will dig in more into this.
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1, 0.94.0
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5161) Compaction algorithm should
prioritize reference files
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5161:
---------------------------------
This is an old existing issue with no patch available. Unassigning from 0.94 for now.
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5161) Compaction algorithm should
prioritize reference files
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262638#comment-13262638 ]
Jieshan Bean commented on HBASE-5161:
-------------------------------------
It's a really special scenario. High input pressure and splitting run in parallel caused this vicious circle.
We should check whether the compaction is still running. Compact a 400G region will take long time.
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1, 0.94.0
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5161) Compaction algorithm should
prioritize reference files
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5161:
-------------------------
Priority: Critical (was: Minor)
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.0, 0.92.1
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5161) Compaction algorithm should
prioritize reference files
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262537#comment-13262537 ]
ramkrishna.s.vasudevan commented on HBASE-5161:
-----------------------------------------------
I did not want to open a new issue. Thought i can reopen this issue. Correct me if am wrong.
I have a high load write operation going on. Region split keeps happening.
When to one of my region the load becomes too heavy the split starts and creates lot of reference files which is greater than my maxfilestocompact.
So suppose my 'hbase.hstore.compaction.max' is 20 and the reference files that are created is 32*2 (64 files).
In compaction selection
{code}
if (compactSelection.getFilesToCompact().size() > this.maxFilesToCompact) {
int pastMax =
compactSelection.getFilesToCompact().size() - this.maxFilesToCompact;
compactSelection.clearSubList(0, pastMax);
}
{code}
The filesToCompact is ordered based on seq id. In this case the set of files from 0 to pastMax (i.e) the reference files which has lesser seq id are not considered for compaction. By the time more store files are created and once again the earlier created ones are avoided. Those being reference files, the split never happens. The region grew upto 400G.
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1, 0.94.0
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5161) Compaction algorithm should
prioritize reference files
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206330#comment-13206330 ]
Jean-Daniel Cryans commented on HBASE-5161:
-------------------------------------------
Just got this again running a 5TB upload, started seeing regions of >50GB that couldn't split.
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Minor
> Fix For: 0.94.0, 0.92.1
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5161) Compaction algorithm should
prioritize reference files
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5161:
---------------------------------
Fix Version/s: (was: 0.94.1)
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-5161) Compaction algorithm
should prioritize reference files
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262537#comment-13262537 ]
ramkrishna.s.vasudevan edited comment on HBASE-5161 at 4/26/12 11:53 AM:
-------------------------------------------------------------------------
I did not want to open a new issue. Thought i can reopen this issue. Correct me if am wrong.
I have a high load write operation going on. Region split keeps happening.
When to one of my region the load becomes too heavy the split starts and creates lot of reference files which is greater than my maxfilestocompact.
So suppose my 'hbase.hstore.compaction.max' is 20 and the reference files that are created is 32*2 (64 files).
In compaction selection
{code}
if (compactSelection.getFilesToCompact().size() > this.maxFilesToCompact) {
int pastMax =
compactSelection.getFilesToCompact().size() - this.maxFilesToCompact;
compactSelection.clearSubList(0, pastMax);
}
{code}
The filesToCompact is ordered based on seq id. In this case the set of files from 0 to pastMax (i.e) the reference files which has lesser seq id are not considered for compaction. By the time more store files are created and once again the earlier created ones are avoided. Those being reference files, the split never happens. The region grew upto 400G.
Note that - We even tried to stop the writes for 2 to 3 hours but still the compaction did not pick up the reference file.
was (Author: ram_krish):
I did not want to open a new issue. Thought i can reopen this issue. Correct me if am wrong.
I have a high load write operation going on. Region split keeps happening.
When to one of my region the load becomes too heavy the split starts and creates lot of reference files which is greater than my maxfilestocompact.
So suppose my 'hbase.hstore.compaction.max' is 20 and the reference files that are created is 32*2 (64 files).
In compaction selection
{code}
if (compactSelection.getFilesToCompact().size() > this.maxFilesToCompact) {
int pastMax =
compactSelection.getFilesToCompact().size() - this.maxFilesToCompact;
compactSelection.clearSubList(0, pastMax);
}
{code}
The filesToCompact is ordered based on seq id. In this case the set of files from 0 to pastMax (i.e) the reference files which has lesser seq id are not considered for compaction. By the time more store files are created and once again the earlier created ones are avoided. Those being reference files, the split never happens. The region grew upto 400G.
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1, 0.94.0
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5161) Compaction algorithm should
prioritize reference files
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217933#comment-13217933 ]
stack commented on HBASE-5161:
------------------------------
This is not actually a problem, right J-D? The actual problem is that it takes a long time to clear the reference files -- even though they are the first things scheduled on region open -- because sometimes we have such a backlog of compaction to catch up on (lots of big files).
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1, 0.94.0
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5161) Compaction algorithm should
prioritize reference files
Posted by "Jean-Daniel Cryans (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-5161.
---------------------------------------
Resolution: Invalid
Right, the real solution is that we need to slow down writes if we get in that sort of situation. We can only compact so fast.
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1, 0.94.0
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5161) Compaction algorithm should
prioritize reference files
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5161:
---------------------------------
Fix Version/s: (was: 0.94.0)
0.94.1
> Compaction algorithm should prioritize reference files
> ------------------------------------------------------
>
> Key: HBASE-5161
> URL: https://issues.apache.org/jira/browse/HBASE-5161
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.92.1, 0.94.1
>
>
> I got myself into a state where my table was un-splittable as long as the insert load was coming in. Emergency flushes because of the low memory barrier don't check the number of store files so it never blocks, to a point where I had in one case 45 store files and the compactions were almost never done on the reference files (had 15 of them, went down by one in 20 minutes). Since you can't split regions with reference files, that region couldn't split and was doomed to just get more store files until the load stopped.
> Marking this as a minor issue, what we really need is a better pushback mechanism but not prioritizing reference files seems wrong.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira