You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Schubert Zhang (JIRA)" <ji...@apache.org> on 2009/10/12 14:27:31 UTC
[jira] Created: (HBASE-1902) Let PerformanceEvaluation support
setting tableName and compress algorithm
Let PerformanceEvaluation support setting tableName and compress algorithm
--------------------------------------------------------------------------
Key: HBASE-1902
URL: https://issues.apache.org/jira/browse/HBASE-1902
Project: Hadoop HBase
Issue Type: Improvement
Components: test
Affects Versions: 0.20.1
Reporter: Schubert Zhang
Assignee: Schubert Zhang
Priority: Minor
Fix For: 0.21.0
1. Current PerformanceEvaluation.java defaultly does not use compression, and always use table name as "TestTable". Tow comand- arguments are added to support setting tableName and compression algorithm by user. e.g.:
(1) tableName="PE_LZO", compression="lzo"
bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=PE_LZO --compress=lzo sequentialWrite 4
(2) default (tabmeName="TesTable", compression="none"
bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 4
2. Current "public static byte[] generateValue(final Random r)" generates bytes[] which is too random. So the effect of compression is not obvious (e.g. there is not size reduce for LZO).
A new "public static byte[] generateValue(final Random r)" is provided here. We can get 4:1 compression ratio for LZO now.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1902) Let PerformanceEvaluation support
setting tableName and compress algorithm
Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764975#action_12764975 ]
Schubert Zhang commented on HBASE-1902:
---------------------------------------
ok, stack.
The added usage print
System.err.println(" [--miniCluster] [--nomapred] [--rows=ROWS] [--table=NAME] [--compress=TYPE] \\"
miss the end );
System.err.println(" [--miniCluster] [--nomapred] [--rows=ROWS] [--table=NAME] [--compress=TYPE] \\");
> Let PerformanceEvaluation support setting tableName and compress algorithm
> --------------------------------------------------------------------------
>
> Key: HBASE-1902
> URL: https://issues.apache.org/jira/browse/HBASE-1902
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.20.1
> Reporter: Schubert Zhang
> Assignee: Schubert Zhang
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-1902-v1.patch
>
>
> 1. Current PerformanceEvaluation.java defaultly does not use compression, and always use table name as "TestTable". Tow comand- arguments are added to support setting tableName and compression algorithm by user. e.g.:
> (1) tableName="PE_LZO", compression="lzo"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=PE_LZO --compress=lzo sequentialWrite 4
> (2) default (tabmeName="TesTable", compression="none"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 4
> 2. Current "public static byte[] generateValue(final Random r)" generates bytes[] which is too random. So the effect of compression is not obvious (e.g. there is not size reduce for LZO).
> A new "public static byte[] generateValue(final Random r)" is provided here. We can get 4:1 compression ratio for LZO now.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1902) Let PerformanceEvaluation support
setting tableName and compress algorithm
Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Schubert Zhang updated HBASE-1902:
----------------------------------
Attachment: HBASE-1902-v1.patch
> Let PerformanceEvaluation support setting tableName and compress algorithm
> --------------------------------------------------------------------------
>
> Key: HBASE-1902
> URL: https://issues.apache.org/jira/browse/HBASE-1902
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.20.1
> Reporter: Schubert Zhang
> Assignee: Schubert Zhang
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-1902-v1.patch
>
>
> 1. Current PerformanceEvaluation.java defaultly does not use compression, and always use table name as "TestTable". Tow comand- arguments are added to support setting tableName and compression algorithm by user. e.g.:
> (1) tableName="PE_LZO", compression="lzo"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=PE_LZO --compress=lzo sequentialWrite 4
> (2) default (tabmeName="TesTable", compression="none"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 4
> 2. Current "public static byte[] generateValue(final Random r)" generates bytes[] which is too random. So the effect of compression is not obvious (e.g. there is not size reduce for LZO).
> A new "public static byte[] generateValue(final Random r)" is provided here. We can get 4:1 compression ratio for LZO now.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1902) Let PerformanceEvaluation support
setting tableName and compress algorithm
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1902:
-------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Thanks for the patch Schubert. Applied to TRUNK.
I added to the PE usage these new options on commit.
On the randomizing of values, I think in the BT paper, they intentionally made it so values were all over the place so compression had no effect on observed performance. I get the sense that compression was always on in their system hence they did this kinda thing in their test code.
I think your change where you make the values less random is fine because you must ask for compression explicitly in PE. It also makes it so can see how compression changes system performance.
> Let PerformanceEvaluation support setting tableName and compress algorithm
> --------------------------------------------------------------------------
>
> Key: HBASE-1902
> URL: https://issues.apache.org/jira/browse/HBASE-1902
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.20.1
> Reporter: Schubert Zhang
> Assignee: Schubert Zhang
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-1902-v1.patch
>
>
> 1. Current PerformanceEvaluation.java defaultly does not use compression, and always use table name as "TestTable". Tow comand- arguments are added to support setting tableName and compression algorithm by user. e.g.:
> (1) tableName="PE_LZO", compression="lzo"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=PE_LZO --compress=lzo sequentialWrite 4
> (2) default (tabmeName="TesTable", compression="none"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 4
> 2. Current "public static byte[] generateValue(final Random r)" generates bytes[] which is too random. So the effect of compression is not obvious (e.g. there is not size reduce for LZO).
> A new "public static byte[] generateValue(final Random r)" is provided here. We can get 4:1 compression ratio for LZO now.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1902) Let PerformanceEvaluation support
setting tableName and compress algorithm
Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Schubert Zhang updated HBASE-1902:
----------------------------------
Status: Patch Available (was: Open)
> Let PerformanceEvaluation support setting tableName and compress algorithm
> --------------------------------------------------------------------------
>
> Key: HBASE-1902
> URL: https://issues.apache.org/jira/browse/HBASE-1902
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.20.1
> Reporter: Schubert Zhang
> Assignee: Schubert Zhang
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-1902-v1.patch
>
>
> 1. Current PerformanceEvaluation.java defaultly does not use compression, and always use table name as "TestTable". Tow comand- arguments are added to support setting tableName and compression algorithm by user. e.g.:
> (1) tableName="PE_LZO", compression="lzo"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=PE_LZO --compress=lzo sequentialWrite 4
> (2) default (tabmeName="TesTable", compression="none"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 4
> 2. Current "public static byte[] generateValue(final Random r)" generates bytes[] which is too random. So the effect of compression is not obvious (e.g. there is not size reduce for LZO).
> A new "public static byte[] generateValue(final Random r)" is provided here. We can get 4:1 compression ratio for LZO now.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1902) Let PerformanceEvaluation support
setting tableName and compress algorithm
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765083#action_12765083 ]
stack commented on HBASE-1902:
------------------------------
Yeah... should be ok now. Thanks for the patch.
> Let PerformanceEvaluation support setting tableName and compress algorithm
> --------------------------------------------------------------------------
>
> Key: HBASE-1902
> URL: https://issues.apache.org/jira/browse/HBASE-1902
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.20.1
> Reporter: Schubert Zhang
> Assignee: Schubert Zhang
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-1902-v1.patch
>
>
> 1. Current PerformanceEvaluation.java defaultly does not use compression, and always use table name as "TestTable". Tow comand- arguments are added to support setting tableName and compression algorithm by user. e.g.:
> (1) tableName="PE_LZO", compression="lzo"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=PE_LZO --compress=lzo sequentialWrite 4
> (2) default (tabmeName="TesTable", compression="none"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 4
> 2. Current "public static byte[] generateValue(final Random r)" generates bytes[] which is too random. So the effect of compression is not obvious (e.g. there is not size reduce for LZO).
> A new "public static byte[] generateValue(final Random r)" is provided here. We can get 4:1 compression ratio for LZO now.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1902) Let PerformanceEvaluation support
setting tableName and compress algorithm
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764898#action_12764898 ]
stack commented on HBASE-1902:
------------------------------
I'd suggest you make a low priority issue to do it but I'd say that it can wait until someone needs it. Meantime, these additions are kinda sweet; helps get numbers w/ how lzo changes the performance story.
> Let PerformanceEvaluation support setting tableName and compress algorithm
> --------------------------------------------------------------------------
>
> Key: HBASE-1902
> URL: https://issues.apache.org/jira/browse/HBASE-1902
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.20.1
> Reporter: Schubert Zhang
> Assignee: Schubert Zhang
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-1902-v1.patch
>
>
> 1. Current PerformanceEvaluation.java defaultly does not use compression, and always use table name as "TestTable". Tow comand- arguments are added to support setting tableName and compression algorithm by user. e.g.:
> (1) tableName="PE_LZO", compression="lzo"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=PE_LZO --compress=lzo sequentialWrite 4
> (2) default (tabmeName="TesTable", compression="none"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 4
> 2. Current "public static byte[] generateValue(final Random r)" generates bytes[] which is too random. So the effect of compression is not obvious (e.g. there is not size reduce for LZO).
> A new "public static byte[] generateValue(final Random r)" is provided here. We can get 4:1 compression ratio for LZO now.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1902) Let PerformanceEvaluation support
setting tableName and compress algorithm
Posted by "Schubert Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764810#action_12764810 ]
Schubert Zhang commented on HBASE-1902:
---------------------------------------
@Stack,
I double checked the BigTable paper:
"We wrote a single string under each row key. Each string was generated randomly and was therefore uncompressible. In addition, strings
under different row key were distinct, so no cross-row compressionwas possible."
So, to let the PerformanceEvaluation tools can work same as BT paper, I am thinking maybe it is more flexible to implement two types of generateValue(): one default mehod generate random values as the paper, and the other generate less random values.
Do you think above is necessary? If it is necessary, I will do it immediately.
> Let PerformanceEvaluation support setting tableName and compress algorithm
> --------------------------------------------------------------------------
>
> Key: HBASE-1902
> URL: https://issues.apache.org/jira/browse/HBASE-1902
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.20.1
> Reporter: Schubert Zhang
> Assignee: Schubert Zhang
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-1902-v1.patch
>
>
> 1. Current PerformanceEvaluation.java defaultly does not use compression, and always use table name as "TestTable". Tow comand- arguments are added to support setting tableName and compression algorithm by user. e.g.:
> (1) tableName="PE_LZO", compression="lzo"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=PE_LZO --compress=lzo sequentialWrite 4
> (2) default (tabmeName="TesTable", compression="none"
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 4
> 2. Current "public static byte[] generateValue(final Random r)" generates bytes[] which is too random. So the effect of compression is not obvious (e.g. there is not size reduce for LZO).
> A new "public static byte[] generateValue(final Random r)" is provided here. We can get 4:1 compression ratio for LZO now.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.