You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Ted Yu (JIRA)" <ji...@apache.org> on 2010/06/10 18:11:13 UTC
[jira] Created: (HBASE-2706) Use heuristics to adjust various
parameters based on load characteristics
Use heuristics to adjust various parameters based on load characteristics
-------------------------------------------------------------------------
Key: HBASE-2706
URL: https://issues.apache.org/jira/browse/HBASE-2706
Project: HBase
Issue Type: New Feature
Components: master
Affects Versions: 0.20.4
Reporter: Ted Yu
Fix For: 0.20.6
Refer to 'ideas to improve throughput of the base writing' discussion on hbase user mailing list.
We should use heuristics to adjust various parameters based on load characteristics. Factors include, but not limited to, number of CPU (cores) on each node, amount of physical memory on each node, Heap space allocated to master/region servers.
HBase parameters shown below would be dynamic whose current values can be seen on master.jsp / regionserver.jsp
For JVM parameters such as -Xmx, master.jsp should display recommended value to user.
-------------
Thanks. I will make this change:
<property>
<name>hbase.hregion.memstore.block.multiplier</name>
<value>8</value>
</property>
<property>
<name>hbase.regionserver.msginterval</name>
<value>10000</value>
</property>
<property>
<name>hbase.hstore.compactionThreshold</name>
<value>6</value>
</property>
<property>
<name>hbase.hstore.blockingStoreFiles</name>
<value>18</value>
</property>
and see how it goes.
Jimmy.
With the above change, the write goes faster if it works, but now we have problem that the
insertion doesn't work at all, here is the result:
2010-06-09 22:29:04,330 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io
.IOException: Could not find target position 0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2706) Use heuristics to adjust various
parameters based on load characteristics
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905241#action_12905241 ]
stack commented on HBASE-2706:
------------------------------
Another to adjust is sizes of caches.
> Use heuristics to adjust various parameters based on load characteristics
> -------------------------------------------------------------------------
>
> Key: HBASE-2706
> URL: https://issues.apache.org/jira/browse/HBASE-2706
> Project: HBase
> Issue Type: New Feature
> Components: master
> Affects Versions: 0.20.4
> Reporter: Ted Yu
> Fix For: 0.90.0
>
>
> Refer to 'ideas to improve throughput of the base writing' discussion on hbase user mailing list.
> We should use heuristics to adjust various parameters based on load characteristics. Factors include, but not limited to, number of CPU (cores) on each node, amount of physical memory on each node, Heap space allocated to master/region servers.
> HBase parameters shown below would be dynamic whose current values can be seen on master.jsp / regionserver.jsp
> For JVM parameters such as -Xmx, master.jsp should display recommended value to user.
> -------------
> Thanks. I will make this change:
> <property>
> <name>hbase.hregion.memstore.block.multiplier</name>
> <value>8</value>
> </property>
> <property>
> <name>hbase.regionserver.msginterval</name>
> <value>10000</value>
> </property>
> <property>
> <name>hbase.hstore.compactionThreshold</name>
> <value>6</value>
> </property>
> <property>
> <name>hbase.hstore.blockingStoreFiles</name>
> <value>18</value>
> </property>
> and see how it goes.
> Jimmy.
> With the above change, the write goes faster if it works, but now we have problem that the
> insertion doesn't work at all, here is the result:
> 2010-06-09 22:29:04,330 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io
> .IOException: Could not find target position 0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2706) Use heuristics to adjust various
parameters based on load characteristics
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2706:
-------------------------
Fix Version/s: 0.90.0
(was: 0.20.6)
Moving to 0.90 out of 0.20.6 (not crittical bug fix)
> Use heuristics to adjust various parameters based on load characteristics
> -------------------------------------------------------------------------
>
> Key: HBASE-2706
> URL: https://issues.apache.org/jira/browse/HBASE-2706
> Project: HBase
> Issue Type: New Feature
> Components: master
> Affects Versions: 0.20.4
> Reporter: Ted Yu
> Fix For: 0.90.0
>
>
> Refer to 'ideas to improve throughput of the base writing' discussion on hbase user mailing list.
> We should use heuristics to adjust various parameters based on load characteristics. Factors include, but not limited to, number of CPU (cores) on each node, amount of physical memory on each node, Heap space allocated to master/region servers.
> HBase parameters shown below would be dynamic whose current values can be seen on master.jsp / regionserver.jsp
> For JVM parameters such as -Xmx, master.jsp should display recommended value to user.
> -------------
> Thanks. I will make this change:
> <property>
> <name>hbase.hregion.memstore.block.multiplier</name>
> <value>8</value>
> </property>
> <property>
> <name>hbase.regionserver.msginterval</name>
> <value>10000</value>
> </property>
> <property>
> <name>hbase.hstore.compactionThreshold</name>
> <value>6</value>
> </property>
> <property>
> <name>hbase.hstore.blockingStoreFiles</name>
> <value>18</value>
> </property>
> and see how it goes.
> Jimmy.
> With the above change, the write goes faster if it works, but now we have problem that the
> insertion doesn't work at all, here is the result:
> 2010-06-09 22:29:04,330 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io
> .IOException: Could not find target position 0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2706) Use heuristics to adjust various
parameters based on load characteristics
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-2706:
---------------------------------
Fix Version/s: (was: 0.90.0)
0.92.0
Punting to 0.92
> Use heuristics to adjust various parameters based on load characteristics
> -------------------------------------------------------------------------
>
> Key: HBASE-2706
> URL: https://issues.apache.org/jira/browse/HBASE-2706
> Project: HBase
> Issue Type: New Feature
> Components: master
> Affects Versions: 0.20.4
> Reporter: Ted Yu
> Fix For: 0.92.0
>
>
> Refer to 'ideas to improve throughput of the base writing' discussion on hbase user mailing list.
> We should use heuristics to adjust various parameters based on load characteristics. Factors include, but not limited to, number of CPU (cores) on each node, amount of physical memory on each node, Heap space allocated to master/region servers.
> HBase parameters shown below would be dynamic whose current values can be seen on master.jsp / regionserver.jsp
> For JVM parameters such as -Xmx, master.jsp should display recommended value to user.
> -------------
> Thanks. I will make this change:
> <property>
> <name>hbase.hregion.memstore.block.multiplier</name>
> <value>8</value>
> </property>
> <property>
> <name>hbase.regionserver.msginterval</name>
> <value>10000</value>
> </property>
> <property>
> <name>hbase.hstore.compactionThreshold</name>
> <value>6</value>
> </property>
> <property>
> <name>hbase.hstore.blockingStoreFiles</name>
> <value>18</value>
> </property>
> and see how it goes.
> Jimmy.
> With the above change, the write goes faster if it works, but now we have problem that the
> insertion doesn't work at all, here is the result:
> 2010-06-09 22:29:04,330 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io
> .IOException: Could not find target position 0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.