You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ryan rawson (JIRA)" <ji...@apache.org> on 2011/01/26 07:55:43 UTC
[jira] Created: (HBASE-3480) Reduce the size of Result
serialization
Reduce the size of Result serialization
---------------------------------------
Key: HBASE-3480
URL: https://issues.apache.org/jira/browse/HBASE-3480
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.0
Reporter: ryan rawson
When faced with a gigabit ethernet network connection, things are pretty slow actually. For example, let's take a 2 MB reply, using a 120MB/sec line rate, we are talking about about 16ms to transfer that data across a gige line. This is a pretty significant amount of time.
So this JIRA is about reducing the size of the Result[] serialization. By exploiting family and qualifier and rowkey duplication, I created a simple encoding scheme to use a dictionary instead of literal strings.
in my testing, I am seeing some success with the sizes. Average serialized size is about 1/2 of previous, but time to serialize on the regionserver side is way up, by a factor of 10x. This might be due to the simplistic first implementation however.
Here is the post change size:
grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
377047.1125
Here is the pre change size:
grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
601078.505882353
That is about a 60% improvement in size.
But times are not so good, here are some samples of the old, in (size) (time in ns)
3874599 10685836
5582725 11525888
so that is about 11ms to serialize 3-5mb of data.
In the new implementation:
1898788 118504672
1630058 91133003
this is 118-91ms for serialized sizes of 1.6-1.8 MB.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3480) Reduce the size of Result
serialization
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987418#action_12987418 ]
ryan rawson commented on HBASE-3480:
------------------------------------
I changed the code to use LZF pure java compression from this library: https://github.com/ning/compress
The result size is smaller now:
grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns compressed: (true|false)/ ; print $1, " ", $2, , " ", $3, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
277735.361445783
But the times still arent so great:
1775773 106297860 true
1620568 68043741 true
1334129 98508585 true
1408999 78860459 true
1264817 60595079 true
622714 28482354 true
511205 23480742 true
The 'true' means that this response was compressed. The first column is response size from HRS -> client in bytes, the second column is time it took to serialize, including both the Result.write* code and the compression in nanoseconds.
We may be better off with a C-based compression algorithm, which would be feasible via this mechanism:
- we will only compress responses we know the size for, and if the size > $THRESHOLD
- we can serialize the Result/Result[] into a DirectByteBuffer
- we can then compress the resulting DirectByteBuffer in to a new one
- we can then use nio to directly reply from this DBB.
I think we could dig up some benchmarks for the java vs C implementation of LZF and figure out if this might be worthwhile or not.
> Reduce the size of Result serialization
> ---------------------------------------
>
> Key: HBASE-3480
> URL: https://issues.apache.org/jira/browse/HBASE-3480
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: ryan rawson
> Attachments: HBASE-3480.txt
>
>
> When faced with a gigabit ethernet network connection, things are pretty slow actually. For example, let's take a 2 MB reply, using a 120MB/sec line rate, we are talking about about 16ms to transfer that data across a gige line. This is a pretty significant amount of time.
> So this JIRA is about reducing the size of the Result[] serialization. By exploiting family and qualifier and rowkey duplication, I created a simple encoding scheme to use a dictionary instead of literal strings.
> in my testing, I am seeing some success with the sizes. Average serialized size is about 1/2 of previous, but time to serialize on the regionserver side is way up, by a factor of 10x. This might be due to the simplistic first implementation however.
> Here is the post change size:
> grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
> 377047.1125
> Here is the pre change size:
> grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
> 601078.505882353
> That is about a 60% improvement in size.
> But times are not so good, here are some samples of the old, in (size) (time in ns)
> 3874599 10685836
> 5582725 11525888
> so that is about 11ms to serialize 3-5mb of data.
> In the new implementation:
> 1898788 118504672
> 1630058 91133003
> this is 118-91ms for serialized sizes of 1.6-1.8 MB.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3480) Reduce the size of Result
serialization
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-3480:
-------------------------------
Attachment: HBASE-3480-lzf.txt
> Reduce the size of Result serialization
> ---------------------------------------
>
> Key: HBASE-3480
> URL: https://issues.apache.org/jira/browse/HBASE-3480
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: ryan rawson
> Attachments: HBASE-3480-lzf.txt, HBASE-3480.txt
>
>
> When faced with a gigabit ethernet network connection, things are pretty slow actually. For example, let's take a 2 MB reply, using a 120MB/sec line rate, we are talking about about 16ms to transfer that data across a gige line. This is a pretty significant amount of time.
> So this JIRA is about reducing the size of the Result[] serialization. By exploiting family and qualifier and rowkey duplication, I created a simple encoding scheme to use a dictionary instead of literal strings.
> in my testing, I am seeing some success with the sizes. Average serialized size is about 1/2 of previous, but time to serialize on the regionserver side is way up, by a factor of 10x. This might be due to the simplistic first implementation however.
> Here is the post change size:
> grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
> 377047.1125
> Here is the pre change size:
> grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
> 601078.505882353
> That is about a 60% improvement in size.
> But times are not so good, here are some samples of the old, in (size) (time in ns)
> 3874599 10685836
> 5582725 11525888
> so that is about 11ms to serialize 3-5mb of data.
> In the new implementation:
> 1898788 118504672
> 1630058 91133003
> this is 118-91ms for serialized sizes of 1.6-1.8 MB.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3480) Reduce the size of Result
serialization
Posted by "Kazuki Ohta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003192#comment-13003192 ]
Kazuki Ohta commented on HBASE-3480:
------------------------------------
I really wanna this feature, because I store the the large amount of text data which the compression really matters.
Currently, I use the LZ compression at the application layer. Happy to have this in the Hbase layer.
Should be better to select the compression algorithm (If I store the images, then this optimization is just the overhead).
And how about Put side?
> Reduce the size of Result serialization
> ---------------------------------------
>
> Key: HBASE-3480
> URL: https://issues.apache.org/jira/browse/HBASE-3480
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: ryan rawson
> Attachments: HBASE-3480-lzf.txt, HBASE-3480.txt
>
>
> When faced with a gigabit ethernet network connection, things are pretty slow actually. For example, let's take a 2 MB reply, using a 120MB/sec line rate, we are talking about about 16ms to transfer that data across a gige line. This is a pretty significant amount of time.
> So this JIRA is about reducing the size of the Result[] serialization. By exploiting family and qualifier and rowkey duplication, I created a simple encoding scheme to use a dictionary instead of literal strings.
> in my testing, I am seeing some success with the sizes. Average serialized size is about 1/2 of previous, but time to serialize on the regionserver side is way up, by a factor of 10x. This might be due to the simplistic first implementation however.
> Here is the post change size:
> grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
> 377047.1125
> Here is the pre change size:
> grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
> 601078.505882353
> That is about a 60% improvement in size.
> But times are not so good, here are some samples of the old, in (size) (time in ns)
> 3874599 10685836
> 5582725 11525888
> so that is about 11ms to serialize 3-5mb of data.
> In the new implementation:
> 1898788 118504672
> 1630058 91133003
> this is 118-91ms for serialized sizes of 1.6-1.8 MB.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HBASE-3480) Reduce the size of Result
serialization
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-3480:
-------------------------------
Attachment: HBASE-3480.txt
> Reduce the size of Result serialization
> ---------------------------------------
>
> Key: HBASE-3480
> URL: https://issues.apache.org/jira/browse/HBASE-3480
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: ryan rawson
> Attachments: HBASE-3480.txt
>
>
> When faced with a gigabit ethernet network connection, things are pretty slow actually. For example, let's take a 2 MB reply, using a 120MB/sec line rate, we are talking about about 16ms to transfer that data across a gige line. This is a pretty significant amount of time.
> So this JIRA is about reducing the size of the Result[] serialization. By exploiting family and qualifier and rowkey duplication, I created a simple encoding scheme to use a dictionary instead of literal strings.
> in my testing, I am seeing some success with the sizes. Average serialized size is about 1/2 of previous, but time to serialize on the regionserver side is way up, by a factor of 10x. This might be due to the simplistic first implementation however.
> Here is the post change size:
> grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
> 377047.1125
> Here is the pre change size:
> grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
> 601078.505882353
> That is about a 60% improvement in size.
> But times are not so good, here are some samples of the old, in (size) (time in ns)
> 3874599 10685836
> 5582725 11525888
> so that is about 11ms to serialize 3-5mb of data.
> In the new implementation:
> 1898788 118504672
> 1630058 91133003
> this is 118-91ms for serialized sizes of 1.6-1.8 MB.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3480) Reduce the size of Result
serialization
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003260#comment-13003260 ]
ryan rawson commented on HBASE-3480:
------------------------------------
in my test the cost of serialization was larger than the time savings
of data transmission on the wire.
I think we are going to put a freeze on RPC changes soon, we need to
be thinking next gen.
> Reduce the size of Result serialization
> ---------------------------------------
>
> Key: HBASE-3480
> URL: https://issues.apache.org/jira/browse/HBASE-3480
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0
> Reporter: ryan rawson
> Attachments: HBASE-3480-lzf.txt, HBASE-3480.txt
>
>
> When faced with a gigabit ethernet network connection, things are pretty slow actually. For example, let's take a 2 MB reply, using a 120MB/sec line rate, we are talking about about 16ms to transfer that data across a gige line. This is a pretty significant amount of time.
> So this JIRA is about reducing the size of the Result[] serialization. By exploiting family and qualifier and rowkey duplication, I created a simple encoding scheme to use a dictionary instead of literal strings.
> in my testing, I am seeing some success with the sizes. Average serialized size is about 1/2 of previous, but time to serialize on the regionserver side is way up, by a factor of 10x. This might be due to the simplistic first implementation however.
> Here is the post change size:
> grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
> 377047.1125
> Here is the pre change size:
> grep 'Serialized size' * | perl -ne '/Serialized size: (\d+?) in (\d+?) ns/ ; print $1, " ", $2, "\n" if $1 > 10000;' | cut -f1 -d' ' | perl -ne '$sum += $_; $count++; END {print $sum/$count, "\n"}'
> 601078.505882353
> That is about a 60% improvement in size.
> But times are not so good, here are some samples of the old, in (size) (time in ns)
> 3874599 10685836
> 5582725 11525888
> so that is about 11ms to serialize 3-5mb of data.
> In the new implementation:
> 1898788 118504672
> 1630058 91133003
> this is 118-91ms for serialized sizes of 1.6-1.8 MB.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira