You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/08/18 01:22:14 UTC
[jira] Created: (HBASE-1771) PE sequentialWrite is 7x slower
because of MemStoreFlusher#checkStoreFileCount
PE sequentialWrite is 7x slower because of MemStoreFlusher#checkStoreFileCount
------------------------------------------------------------------------------
Key: HBASE-1771
URL: https://issues.apache.org/jira/browse/HBASE-1771
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1771) PE sequentialWrite is 7x slower
because of MemStoreFlusher#checkStoreFileCount
Posted by "Dave Latham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744579#action_12744579 ]
Dave Latham commented on HBASE-1771:
------------------------------------
Looks like this would be applicable to those of us still running 0.19. Does that sound right?
> PE sequentialWrite is 7x slower because of MemStoreFlusher#checkStoreFileCount
> ------------------------------------------------------------------------------
>
> Key: HBASE-1771
> URL: https://issues.apache.org/jira/browse/HBASE-1771
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.0, 0.21.0
>
> Attachments: HBASE-1771-v1.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1771) PE sequentialWrite is 7x slower
because of MemStoreFlusher#checkStoreFileCount
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744484#action_12744484 ]
Andrew Purtell commented on HBASE-1771:
---------------------------------------
My write intensive environment (4 node Heritrix crawl farm) has needed this set to 20 to avoid blocking. Otherwise there are significant troughs in write throughput at a regular period.
Multithreaded compaction might help.
> PE sequentialWrite is 7x slower because of MemStoreFlusher#checkStoreFileCount
> ------------------------------------------------------------------------------
>
> Key: HBASE-1771
> URL: https://issues.apache.org/jira/browse/HBASE-1771
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.0, 0.21.0
>
> Attachments: HBASE-1771-v1.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1771) PE sequentialWrite is 7x slower
because of MemStoreFlusher#checkStoreFileCount
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray resolved HBASE-1771.
----------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
0.20.0
Assignee: Jonathan Gray
Hadoop Flags: [Reviewed]
Committed to trunk and branch.
> PE sequentialWrite is 7x slower because of MemStoreFlusher#checkStoreFileCount
> ------------------------------------------------------------------------------
>
> Key: HBASE-1771
> URL: https://issues.apache.org/jira/browse/HBASE-1771
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.0, 0.21.0
>
> Attachments: HBASE-1771-v1.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1771) PE sequentialWrite is 7x slower
because of MemStoreFlusher#checkStoreFileCount
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744590#action_12744590 ]
stack commented on HBASE-1771:
------------------------------
@Dave I backported patch to 0.19 branch
> PE sequentialWrite is 7x slower because of MemStoreFlusher#checkStoreFileCount
> ------------------------------------------------------------------------------
>
> Key: HBASE-1771
> URL: https://issues.apache.org/jira/browse/HBASE-1771
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Gray
> Fix For: 0.20.0, 0.21.0
>
> Attachments: HBASE-1771-v1.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1771) PE sequentialWrite is 7x slower
because of MemStoreFlusher#checkStoreFileCount
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1771:
---------------------------------
Attachment: HBASE-1771-v1.patch
Exposes two config parameters into hbase-default.xml (these were already being read in but were defaulting too low). Set at 7 as default (3 is default compaction threshold where we actually trigger compaction in the first place, so means we can take on 4 more flushes while waiting for compaction / before blocking).
{code}
<property>
<name>hbase.hstore.blockingStoreFiles</name>
<value>7</value>
<description>
If more than this number of StoreFiles in any one Store
(one StoreFile is written per flush of MemStore) then updates are
blocked for this HRegion until a compaction is completed, or
until hbase.hstore.blockingWaitTime has been exceeded.
</description>
</property>
<property>
<name>hbase.hstore.blockingWaitTime</name>
<value>90000</value>
<description>
The time an HRegion will block updates for after hitting the StoreFile
limit defined by hbase.hstore.blockingStoreFiles.
After this time has elapsed, the HRegion will stop blocking updates even
if a compaction has not been completed. Default: 90 seconds.
</description>
</property>
{code}
> PE sequentialWrite is 7x slower because of MemStoreFlusher#checkStoreFileCount
> ------------------------------------------------------------------------------
>
> Key: HBASE-1771
> URL: https://issues.apache.org/jira/browse/HBASE-1771
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Attachments: HBASE-1771-v1.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1771) PE sequentialWrite is 7x slower
because of MemStoreFlusher#checkStoreFileCount
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744288#action_12744288 ]
stack commented on HBASE-1771:
------------------------------
+1
Tested. We're back up to where we were with PE... at about 9k writes/sec.
> PE sequentialWrite is 7x slower because of MemStoreFlusher#checkStoreFileCount
> ------------------------------------------------------------------------------
>
> Key: HBASE-1771
> URL: https://issues.apache.org/jira/browse/HBASE-1771
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Attachments: HBASE-1771-v1.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.