You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org> on 2010/10/27 20:47:19 UTC
[jira] Created: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Compactions: Use more intelligent priorities for PriorityCompactionQueue
------------------------------------------------------------------------
Key: HBASE-3160
URL: https://issues.apache.org/jira/browse/HBASE-3160
Project: HBase
Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg updated HBASE-3160:
---------------------------------------
Fix Version/s: 0.90.0
Affects Version/s: 0.90.0
0.89.20100924
Status: Patch Available (was: Open)
added diff to Review Board
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "Dave Latham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927887#action_12927887 ]
Dave Latham commented on HBASE-3160:
------------------------------------
Does that mean https://issues.apache.org/jira/browse/HBASE-2770 is fixed then?
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
> Attachments: HBASE-3160.patch
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925502#action_12925502 ]
Nicolas Spiegelberg commented on HBASE-3160:
--------------------------------------------
My initial suggestion:
Priority = max(len(s.storefiles) for s in region.stores)
This will preserve the priority of Stores exceeding the 15 file limit while prioritizing Stores that are getting close to exceeding that limit. As a further topic of discussion, it would be good to interrupt a low-pri compaction (< 15 pri) when a 15+ pri request comes in. Of course, a lot of this would be nicer if we had a compaction thread pool. A man can dream...
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926183#action_12926183 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Nicolas" <ns...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/
-----------------------------------------------------------
(Updated 2010-10-28 23:00:28.324768)
Review request for hbase.
Changes
-------
addressed Stack & Ted's comments
Summary
-------
Switch to more intelligent priority metric: blockingSize - max(len(s.storefiles) for s in region.stores) . This will allow us to better prioritize, give us faster responsiveness to users, and feel more cavalier about issuing new compaction requests. Note that we also found/fixed a major compaction downgrade bug while writing this code.
This addresses bug HBASE-3160.
http://issues.apache.org/jira/browse/HBASE-3160
Diffs
-----
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionRequestor.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1027787
trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityCompactionQueue.java 1027787
Diff: http://review.cloudera.org/r/1103/diff
Testing
-------
mvn clean install -Dtest=TestPriorityCompaction
dev cluster tests (on 0.89)
Thanks,
Nicolas
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926344#action_12926344 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Kannan Muthukkaruppan" <ka...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1704
-----------------------------------------------------------
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java
<http://review.cloudera.org/r/1103/#comment5620>
the comment around "if it is already present in the queue" needs to be updated.
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java
<http://review.cloudera.org/r/1103/#comment5621>
I think here and in this diff overall (as well as the log outputs) readability will be much better if we flipped the sign of the priority... i.e. higher values imply higher priority.
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
<http://review.cloudera.org/r/1103/#comment5619>
-1 and not Integer.MAX_VALUE?
If blocking store files is not specified (i.e. user can tolerate any number of files), then as per your intent, don't you want PRIORITY_USER to be higher-pri always?
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
<http://review.cloudera.org/r/1103/#comment5618>
I think we should state it here (or somewhere) explicitly that smaller values implies higher priority.
- Kannan
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
> Attachments: HBASE-3160.patch
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926173#action_12926173 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Nicolas" <ns...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1697
-----------------------------------------------------------
@Stack: Note that this is going into our 0.89 branch, so it should receive thorough real-world testing. I would suggest the 0.90 branch. It's actually a bug fix in addition to a feature. With the current compaction logic (pre HBASE-2462), the CompactSplitThread can get overwhlemed without this. You unconditionally compact at least 4 files to be aggressive in times of light load. This is fine until your load picks up, you're congested, and writing too many storefiles. Then, since the PriorityCompactionQueue is essentially operating as a FIFO until the blocking limit, it doesn't make any informed decisions about who to service next and waits until your puts freeze to be intelligent...
- Nicolas
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926370#action_12926370 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Kannan Muthukkaruppan" <ka...@facebook.com>
bq. On 2010-10-29 09:48:53, Kannan Muthukkaruppan wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java, line 149
bq. > <http://review.cloudera.org/r/1103/diff/2/?file=16198#file16198line149>
bq. >
bq. > I think here and in this diff overall (as well as the log outputs) readability will be much better if we flipped the sign of the priority... i.e. higher values imply higher priority.
bq. >
bq. >
bq. >
bq.
bq. Nicolas wrote:
bq. yeah, priority ordering on heaps are always controversial. I think we should keep it as a minheap unless there is a reason to switch (no permanent state = no problem). The nice thing about this ordering is that you know Pri <= 0 is bad, no matter what your blocking limit is.
Yes, ok with as is. I posted a subsequent comment to that effect.
bq. On 2010-10-29 09:48:53, Kannan Muthukkaruppan wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java, line 190
bq. > <http://review.cloudera.org/r/1103/diff/2/?file=16199#file16199line190>
bq. >
bq. > -1 and not Integer.MAX_VALUE?
bq. >
bq. > If blocking store files is not specified (i.e. user can tolerate any number of files), then as per your intent, don't you want PRIORITY_USER to be higher-pri always?
bq.
bq. Nicolas wrote:
bq. agreed. I just used the original value from MemStoreFlusher, but your version makes the most sense. Should this be a seperate jira though? We would need to intestigate any other areas of code that would need to be changed by altering this default...
separate JIRA is fine too. Whatever is easier.
bq. On 2010-10-29 09:48:53, Kannan Muthukkaruppan wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java, line 1330
bq. > <http://review.cloudera.org/r/1103/diff/2/?file=16199#file16199line1330>
bq. >
bq. > I think we should state it here (or somewhere) explicitly that smaller values implies higher priority.
bq.
bq. Nicolas wrote:
bq. should we mention that in the CompactSplitThread?
yes, that seems like a good spot.
- Kannan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1704
-----------------------------------------------------------
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
> Attachments: HBASE-3160.patch
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926363#action_12926363 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Nicolas" <ns...@facebook.com>
bq. On 2010-10-29 09:48:53, Kannan Muthukkaruppan wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java, line 138
bq. > <http://review.cloudera.org/r/1103/diff/2/?file=16198#file16198line138>
bq. >
bq. > the comment around "if it is already present in the queue" needs to be updated.
will update
bq. On 2010-10-29 09:48:53, Kannan Muthukkaruppan wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java, line 149
bq. > <http://review.cloudera.org/r/1103/diff/2/?file=16198#file16198line149>
bq. >
bq. > I think here and in this diff overall (as well as the log outputs) readability will be much better if we flipped the sign of the priority... i.e. higher values imply higher priority.
bq. >
bq. >
bq. >
yeah, priority ordering on heaps are always controversial. I think we should keep it as a minheap unless there is a reason to switch (no permanent state = no problem). The nice thing about this ordering is that you know Pri <= 0 is bad, no matter what your blocking limit is.
bq. On 2010-10-29 09:48:53, Kannan Muthukkaruppan wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java, line 190
bq. > <http://review.cloudera.org/r/1103/diff/2/?file=16199#file16199line190>
bq. >
bq. > -1 and not Integer.MAX_VALUE?
bq. >
bq. > If blocking store files is not specified (i.e. user can tolerate any number of files), then as per your intent, don't you want PRIORITY_USER to be higher-pri always?
agreed. I just used the original value from MemStoreFlusher, but your version makes the most sense. Should this be a seperate jira though? We would need to intestigate any other areas of code that would need to be changed by altering this default...
bq. On 2010-10-29 09:48:53, Kannan Muthukkaruppan wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java, line 1330
bq. > <http://review.cloudera.org/r/1103/diff/2/?file=16199#file16199line1330>
bq. >
bq. > I think we should state it here (or somewhere) explicitly that smaller values implies higher priority.
should we mention that in the CompactSplitThread?
- Nicolas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1704
-----------------------------------------------------------
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
> Attachments: HBASE-3160.patch
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927696#action_12927696 ]
Nicolas Spiegelberg commented on HBASE-3160:
--------------------------------------------
@Jeff: The bug wasn't related to the priority compaction code. The problem was in CompactSplitThread.requestCompaction(). If the user requested a major compaction, then the code would issue HRegion.setForceMajorCompaction(true). however, if that region required a compaction while the HRegion was waiting in the compact queue, then the code would call setForceMajorCompaction(false) and downgrade the compaction request.
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
> Attachments: HBASE-3160.patch
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925960#action_12925960 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Nicolas" <ns...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/
-----------------------------------------------------------
(Updated 2010-10-28 14:48:04.897593)
Review request for hbase.
Changes
-------
Updates after internal review.
1. Removed PRIORITY_BLOCKING. It was a little confusing because blocking is <= 0 and not necessarily the value 0 itself.
2. Cleaned up compaction downgrade conditional.
Questions resolved:
1q. Wondering if we want to go ahead with a split if the compaction priority is PRIORITY_BLOCKING?
1a. Counter-example: storefiles could be very small and compaction will reduce it to one medium file. It was starved because a major compaction occurred immediately beforehand and took forever to finish.
2q. Should we add in logic to interrupt a running compaction if it is lower priority (e.g. Pri >= 5) and we have a PRIORITY_BLOCKING request? We can then re-enqueue the compaction to run at a less pressured time.
2a. Save this problem for another JIRA. Since the master has threadpools, we should give the CompactSplitThread prioritized threadpools and dedicate one thread only to blocking requests.
Summary
-------
Switch to more intelligent priority metric: blockingSize - max(len(s.storefiles) for s in region.stores) . This will allow us to better prioritize, give us faster responsiveness to users, and feel more cavalier about issuing new compaction requests. Note that we also found/fixed a major compaction downgrade bug while writing this code.
This addresses bug HBASE-3160.
http://issues.apache.org/jira/browse/HBASE-3160
Diffs (updated)
-----
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionRequestor.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1027787
trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityCompactionQueue.java 1027787
Diff: http://review.cloudera.org/r/1103/diff
Testing
-------
mvn clean install -Dtest=TestPriorityCompaction
dev cluster tests (on 0.89)
Thanks,
Nicolas
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926373#action_12926373 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1709
-----------------------------------------------------------
Please do as separate JIRA. I committed last version of N's patch. Do as separate JIRA and I'll commit. Thanks lads.
- stack
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
> Attachments: HBASE-3160.patch
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926176#action_12926176 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Nicolas" <ns...@facebook.com>
bq. On 2010-10-28 22:04:28, Ted Yu wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java, line 1333
bq. > <http://review.cloudera.org/r/1103/diff/2/?file=16199#file16199line1333>
bq. >
bq. > It is possible to return a negative priority here.
that is correct. the 0 is centered around the blocking limit. I added a test to ensure that we can properly handle negative priorities and prioritize the most negative value @ TestPriorityCompactionQueue.java:308 . Note that the code to handle priorities in the heap is @ PriorityCompactionQueue.java:89
bq. On 2010-10-28 22:04:28, Ted Yu wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 3258
bq. > <http://review.cloudera.org/r/1103/diff/2/?file=16195#file16195line3258>
bq. >
bq. > Seems to be typo here.
bq. > Should be: this region should have
will fix
- Nicolas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1698
-----------------------------------------------------------
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926181#action_12926181 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: stack@duboce.net
bq. On 2010-10-28 22:04:28, Ted Yu wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 3258
bq. > <http://review.cloudera.org/r/1103/diff/2/?file=16195#file16195line3258>
bq. >
bq. > Seems to be typo here.
bq. > Should be: this region should have
bq.
bq. Nicolas wrote:
bq. will fix
bq.
bq. stack wrote:
bq. OK... put up a new patch N and I'll commit.
Nm nicolas... I'll fix this on commit... Committing now.
- stack
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1698
-----------------------------------------------------------
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3160:
-------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed because its a bug fix.
Thanks for the patch Nicolas and thanks for reviewing Ted.
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
> Attachments: HBASE-3160.patch
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926178#action_12926178 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: stack@duboce.net
bq. On 2010-10-28 22:04:28, Ted Yu wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 3258
bq. > <http://review.cloudera.org/r/1103/diff/2/?file=16195#file16195line3258>
bq. >
bq. > Seems to be typo here.
bq. > Should be: this region should have
bq.
bq. Nicolas wrote:
bq. will fix
OK... put up a new patch N and I'll commit.
- stack
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1698
-----------------------------------------------------------
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926167#action_12926167 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1695
-----------------------------------------------------------
Looks good to me. You fellas think it a bug fix or a feature? Should I punt to 0.92 or include now in TRUNK?
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
<http://review.cloudera.org/r/1103/#comment5605>
Why would we do this -- fiddle w/ our priority if we are going to split? Should we check if we are to split first and if not then mess w/ priority?
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
<http://review.cloudera.org/r/1103/#comment5606>
So, we aggregate these Store counts to come up w/ a Region count and the Region w/ most storefiles gets highest compaction priority?
trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityCompactionQueue.java
<http://review.cloudera.org/r/1103/#comment5607>
There is white space here on end of lines... I can fix on commit.
- stack
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926179#action_12926179 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/
-----------------------------------------------------------
(Updated 2010-10-28 22:47:02.663004)
Review request for hbase.
Changes
-------
Just saying I'll commit after you put up new version Nicolas.
Summary
-------
Switch to more intelligent priority metric: blockingSize - max(len(s.storefiles) for s in region.stores) . This will allow us to better prioritize, give us faster responsiveness to users, and feel more cavalier about issuing new compaction requests. Note that we also found/fixed a major compaction downgrade bug while writing this code.
This addresses bug HBASE-3160.
http://issues.apache.org/jira/browse/HBASE-3160
Diffs (updated)
-----
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionRequestor.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1027787
trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityCompactionQueue.java 1027787
Diff: http://review.cloudera.org/r/1103/diff
Testing
-------
mvn clean install -Dtest=TestPriorityCompaction
dev cluster tests (on 0.89)
Thanks,
Nicolas
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927969#action_12927969 ]
Nicolas Spiegelberg commented on HBASE-3160:
--------------------------------------------
@Jeff: I forgot yesterday, but I also did fix an issue with the compaction priorities. If a flush happened for the store that was being compacted, it would be added to the compaction queue at the pre-compact priority instead of post-compact. We now do:
{code}
if (!this.server.isStopRequested()) {
// requests that were added during compaction will have a
// stale priority. remove and re-insert to update priority
boolean hadCompaction = compactionQueue.remove(r);
if (midKey != null) {
split(r, midKey);
} else if (hadCompaction) {
compactionQueue.add(r);
}
}
{code}
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
> Attachments: HBASE-3160.patch
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925616#action_12925616 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Nicolas" <ns...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
Switch to more intelligent priority metric: blockingSize - max(len(s.storefiles) for s in region.stores) . This will allow us to better prioritize, give us faster responsiveness to users, and feel more cavalier about issuing new compaction requests. Note that we also found/fixed a major compaction downgrade bug while writing this code.
This addresses bug HBASE-3160.
http://issues.apache.org/jira/browse/HBASE-3160
Diffs
-----
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionRequestor.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/PriorityCompactionQueue.java 1027787
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1027787
trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityCompactionQueue.java 1027787
Diff: http://review.cloudera.org/r/1103/diff
Testing
-------
mvn clean install -Dtest=TestPriorityCompaction
dev cluster tests (on 0.89)
Thanks,
Nicolas
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926170#action_12926170 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Nicolas" <ns...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1696
-----------------------------------------------------------
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
<http://review.cloudera.org/r/1103/#comment5608>
@Stack you're completely correct. remember that we have splits turned off, so I wasn't thinking in that vein. I think we want to unconditionally remove and then add only if(just_removed && !splitting)
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
<http://review.cloudera.org/r/1103/#comment5609>
exactly. that is how blockingStoreFile decisions are made right now. this will also make it easy to transition to per-store compactions in the future.
- Nicolas
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925621#action_12925621 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Nicolas" <ns...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1683
-----------------------------------------------------------
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
<http://review.cloudera.org/r/1103/#comment5584>
Does this need to be explicitly PRIORITY_BLOCKING? My first thought was no (thinking that SF count would still handle priorities), but it looks like this really prioritizes META compactions to always be critical. Should this be MIN_INT, so it even prioritizes over true blocked stores?
- Nicolas
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926175#action_12926175 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Ted Yu" <te...@yahoo.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1698
-----------------------------------------------------------
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/1103/#comment5610>
Seems to be typo here.
Should be: this region should have
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
<http://review.cloudera.org/r/1103/#comment5611>
It is possible to return a negative priority here.
- Ted
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg updated HBASE-3160:
---------------------------------------
Attachment: HBASE-3160.patch
I made a minor typo when moving my fixes from internal to 0.90 in the Review Board diff. This is the final version for the commit
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
> Attachments: HBASE-3160.patch
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926180#action_12926180 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1703
-----------------------------------------------------------
Oh, nm, I see you want to fix the typo AND change the split stuff...
Ok, I'll wait on new patch.
- stack
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "Jeff Whiting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927694#action_12927694 ]
Jeff Whiting commented on HBASE-3160:
-------------------------------------
Nicholas you mentioned in the jira that "Note that we also found/fixed a major compaction downgrade bug while writing this code." Being the primary author of HBASE-2646 which added the priority compaction queue, I'm interested in knowing what the bug was! So what was the bug? And what was your fix?
I like the improvement and especially how it makes it much more granular in who should get priority.
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
> Attachments: HBASE-3160.patch
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925631#action_12925631 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Nicolas" <ns...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1684
-----------------------------------------------------------
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
<http://review.cloudera.org/r/1103/#comment5585>
never mind. my fault for inverting the conditional logic. this is the correct logic. if (isTooManyStorefiles() == true) region.getCompactPriority() <= 0, so we're okay
- Nicolas
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3160) Compactions: Use more intelligent
priorities for PriorityCompactionQueue
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926356#action_12926356 ]
HBase Review Board commented on HBASE-3160:
-------------------------------------------
Message from: "Kannan Muthukkaruppan" <ka...@facebook.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1103/#review1705
-----------------------------------------------------------
On second thoughts, I take back my comments on flipping the sign of the priority. I guess it is common to usage ( P0 > P1 > P2). Perhaps the -ve numbers were a little confusing, but what you have seems reasonable.
- Kannan
> Compactions: Use more intelligent priorities for PriorityCompactionQueue
> ------------------------------------------------------------------------
>
> Key: HBASE-3160
> URL: https://issues.apache.org/jira/browse/HBASE-3160
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.89.20100924, 0.90.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
> Fix For: 0.90.0
>
> Attachments: HBASE-3160.patch
>
>
> One of the problems with the current compaction queue is that we have a very low granularity on the importance of the various compactions in the queue. If a StoreFile count exceeds 15 files, only then do we bump via enum change. We should instead look into more intelligent, granular priority metrics for choosing the next compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.