You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2010/09/11 02:03:33 UTC
[jira] Created: (HBASE-2987) Avoid compressing flush files
Avoid compressing flush files
-----------------------------
Key: HBASE-2987
URL: https://issues.apache.org/jira/browse/HBASE-2987
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Attachments: HBASE-2987.patch
I've extended Hadoop compression to use the LZMA algorithm and HFile to provide an option for selecting it. With typical input, the LZMA algorithm produces 30% smaller output than GZIP at max compression (which is currently the best available option for HFiles) and 15% smaller output than BZIP2. I'm aware of the "disk is cheap" mantra but for a multi-peta-scale archival application, where we still want random read and random update capabilities, 30% less disk is a substantial cost savings. LZMA compression speed is ~1 MB/second on a 2 GHz CPU, decompression speed is ~20 MB/second. This is 4x slower than BZIP2 to compress but at least 2x faster to decompress for 15% better results. For an archival application these properties would be acceptable if not for the very significant problem of flushing. Obviously the low throughput of the LZMA compressor means it is unsuitable for foreground processing. In HBase terms, it can be used for compaction but not for flush files.
Attached patch, against 0.20 branch, turns off compression for flushes. This could be implemented as a config option, but I wonder if with the possible exception of LZO should we be compressing flushes at all? Any significant reduction in flush throughput can stall writers during periods of high write activity. Maybe globally disabling compression on flush flies is a good thing?
I have tested this and confirmed the result is the desired behavior: 'file' shows flush files as uncompressed data, compacted files as compressed. Compaction merges files with different compression properties. LZMA provides rather extreme space savings over the other available options without slowing down writers if the regionservers are configured with enough write buffering to ride over the significantly lengthened compaction times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2987) Avoid compressing flush files
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908403#action_12908403 ]
Andrew Purtell commented on HBASE-2987:
---------------------------------------
bq. ... it might make sense to disable compression on flushes *and* minor compactions and to enable it on major compactions (in an archival example like this using LZMA or the like). It could even make sense to use something like LZO on flushes/minors and LZMA on majors.
Makes sense. I'll work it up.
> Avoid compressing flush files
> -----------------------------
>
> Key: HBASE-2987
> URL: https://issues.apache.org/jira/browse/HBASE-2987
> Project: HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Attachments: HBASE-2987.patch
>
>
> I've extended Hadoop compression to use the LZMA algorithm and HFile to provide an option for selecting it. With typical input, the LZMA algorithm produces 30% smaller output than GZIP at max compression (which is currently the best available option for HFiles) and 15% smaller output than BZIP2. I'm aware of the "disk is cheap" mantra but for a multi-peta-scale archival application, where we still want random read and random update capabilities, 30% less disk is a substantial cost savings. LZMA compression speed is ~1 MB/second on a 2 GHz CPU, decompression speed is ~20 MB/second. This is 4x slower than BZIP2 to compress but at least 2x faster to decompress for 15% better results. For an archival application these properties would be acceptable if not for the very significant problem of flushing. Obviously the low throughput of the LZMA compressor means it is unsuitable for foreground processing. In HBase terms, it can be used for compaction but not for flush files.
> Attached patch, against 0.20 branch, turns off compression for flushes. This could be implemented as a config option, but I wonder if with the possible exception of LZO should we be compressing flushes at all? Any significant reduction in flush throughput can stall writers during periods of high write activity. Maybe globally disabling compression on flush flies is a good thing?
> I have tested this and confirmed the result is the desired behavior: 'file' shows flush files as uncompressed data, compacted files as compressed. Compaction merges files with different compression properties. LZMA provides rather extreme space savings over the other available options without slowing down writers if the regionservers are configured with enough write buffering to ride over the significantly lengthened compaction times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2987) Avoid compressing flush files
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908252#action_12908252 ]
Andrew Purtell commented on HBASE-2987:
---------------------------------------
bq. unless you're talking about the first flush not finishing before it's time for the second flush?
Yes, and the write gate comes down because the memstore limit on the region is reached.
> Avoid compressing flush files
> -----------------------------
>
> Key: HBASE-2987
> URL: https://issues.apache.org/jira/browse/HBASE-2987
> Project: HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Attachments: HBASE-2987.patch
>
>
> I've extended Hadoop compression to use the LZMA algorithm and HFile to provide an option for selecting it. With typical input, the LZMA algorithm produces 30% smaller output than GZIP at max compression (which is currently the best available option for HFiles) and 15% smaller output than BZIP2. I'm aware of the "disk is cheap" mantra but for a multi-peta-scale archival application, where we still want random read and random update capabilities, 30% less disk is a substantial cost savings. LZMA compression speed is ~1 MB/second on a 2 GHz CPU, decompression speed is ~20 MB/second. This is 4x slower than BZIP2 to compress but at least 2x faster to decompress for 15% better results. For an archival application these properties would be acceptable if not for the very significant problem of flushing. Obviously the low throughput of the LZMA compressor means it is unsuitable for foreground processing. In HBase terms, it can be used for compaction but not for flush files.
> Attached patch, against 0.20 branch, turns off compression for flushes. This could be implemented as a config option, but I wonder if with the possible exception of LZO should we be compressing flushes at all? Any significant reduction in flush throughput can stall writers during periods of high write activity. Maybe globally disabling compression on flush flies is a good thing?
> I have tested this and confirmed the result is the desired behavior: 'file' shows flush files as uncompressed data, compacted files as compressed. Compaction merges files with different compression properties. LZMA provides rather extreme space savings over the other available options without slowing down writers if the regionservers are configured with enough write buffering to ride over the significantly lengthened compaction times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2987) Avoid compressing flush files
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-2987.
-----------------------------------
Resolution: Duplicate
Continued as HBASE-2988
> Avoid compressing flush files
> -----------------------------
>
> Key: HBASE-2987
> URL: https://issues.apache.org/jira/browse/HBASE-2987
> Project: HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Attachments: HBASE-2987.patch
>
>
> I've extended Hadoop compression to use the LZMA algorithm and HFile to provide an option for selecting it. With typical input, the LZMA algorithm produces 30% smaller output than GZIP at max compression (which is currently the best available option for HFiles) and 15% smaller output than BZIP2. I'm aware of the "disk is cheap" mantra but for a multi-peta-scale archival application, where we still want random read and random update capabilities, 30% less disk is a substantial cost savings. LZMA compression speed is ~1 MB/second on a 2 GHz CPU, decompression speed is ~20 MB/second. This is 4x slower than BZIP2 to compress but at least 2x faster to decompress for 15% better results. For an archival application these properties would be acceptable if not for the very significant problem of flushing. Obviously the low throughput of the LZMA compressor means it is unsuitable for foreground processing. In HBase terms, it can be used for compaction but not for flush files.
> Attached patch, against 0.20 branch, turns off compression for flushes. This could be implemented as a config option, but I wonder if with the possible exception of LZO should we be compressing flushes at all? Any significant reduction in flush throughput can stall writers during periods of high write activity. Maybe globally disabling compression on flush flies is a good thing?
> I have tested this and confirmed the result is the desired behavior: 'file' shows flush files as uncompressed data, compacted files as compressed. Compaction merges files with different compression properties. LZMA provides rather extreme space savings over the other available options without slowing down writers if the regionservers are configured with enough write buffering to ride over the significantly lengthened compaction times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2987) Avoid compressing flush files
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-2987:
----------------------------------
Attachment: HBASE-2987.patch
> Avoid compressing flush files
> -----------------------------
>
> Key: HBASE-2987
> URL: https://issues.apache.org/jira/browse/HBASE-2987
> Project: HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Attachments: HBASE-2987.patch
>
>
> I've extended Hadoop compression to use the LZMA algorithm and HFile to provide an option for selecting it. With typical input, the LZMA algorithm produces 30% smaller output than GZIP at max compression (which is currently the best available option for HFiles) and 15% smaller output than BZIP2. I'm aware of the "disk is cheap" mantra but for a multi-peta-scale archival application, where we still want random read and random update capabilities, 30% less disk is a substantial cost savings. LZMA compression speed is ~1 MB/second on a 2 GHz CPU, decompression speed is ~20 MB/second. This is 4x slower than BZIP2 to compress but at least 2x faster to decompress for 15% better results. For an archival application these properties would be acceptable if not for the very significant problem of flushing. Obviously the low throughput of the LZMA compressor means it is unsuitable for foreground processing. In HBase terms, it can be used for compaction but not for flush files.
> Attached patch, against 0.20 branch, turns off compression for flushes. This could be implemented as a config option, but I wonder if with the possible exception of LZO should we be compressing flushes at all? Any significant reduction in flush throughput can stall writers during periods of high write activity. Maybe globally disabling compression on flush flies is a good thing?
> I have tested this and confirmed the result is the desired behavior: 'file' shows flush files as uncompressed data, compacted files as compressed. Compaction merges files with different compression properties. LZMA provides rather extreme space savings over the other available options without slowing down writers if the regionservers are configured with enough write buffering to ride over the significantly lengthened compaction times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2987) Avoid compressing flush files
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908354#action_12908354 ]
Jonathan Gray commented on HBASE-2987:
--------------------------------------
I think the update wall can come up when memstore reaches it's max limit and you can't flush. You can't flush because you've reached the blockingStoreFiles count and have to wait for a compaction to complete. Or is there another situation where you have snapshotted, waiting for the flush, and the memstore fills before the snapshot gets flushed?
Prioritizing compactions has shown to help the first case. I don't think I've seen the second case in practice.
In any case, I'd still be +1 on making these things configurable.
> Avoid compressing flush files
> -----------------------------
>
> Key: HBASE-2987
> URL: https://issues.apache.org/jira/browse/HBASE-2987
> Project: HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Attachments: HBASE-2987.patch
>
>
> I've extended Hadoop compression to use the LZMA algorithm and HFile to provide an option for selecting it. With typical input, the LZMA algorithm produces 30% smaller output than GZIP at max compression (which is currently the best available option for HFiles) and 15% smaller output than BZIP2. I'm aware of the "disk is cheap" mantra but for a multi-peta-scale archival application, where we still want random read and random update capabilities, 30% less disk is a substantial cost savings. LZMA compression speed is ~1 MB/second on a 2 GHz CPU, decompression speed is ~20 MB/second. This is 4x slower than BZIP2 to compress but at least 2x faster to decompress for 15% better results. For an archival application these properties would be acceptable if not for the very significant problem of flushing. Obviously the low throughput of the LZMA compressor means it is unsuitable for foreground processing. In HBase terms, it can be used for compaction but not for flush files.
> Attached patch, against 0.20 branch, turns off compression for flushes. This could be implemented as a config option, but I wonder if with the possible exception of LZO should we be compressing flushes at all? Any significant reduction in flush throughput can stall writers during periods of high write activity. Maybe globally disabling compression on flush flies is a good thing?
> I have tested this and confirmed the result is the desired behavior: 'file' shows flush files as uncompressed data, compacted files as compressed. Compaction merges files with different compression properties. LZMA provides rather extreme space savings over the other available options without slowing down writers if the regionservers are configured with enough write buffering to ride over the significantly lengthened compaction times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2987) Avoid compressing flush files
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908355#action_12908355 ]
Jonathan Gray commented on HBASE-2987:
--------------------------------------
But I don't think that default behavior should be to override a users compression settings for flushes.
> Avoid compressing flush files
> -----------------------------
>
> Key: HBASE-2987
> URL: https://issues.apache.org/jira/browse/HBASE-2987
> Project: HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Attachments: HBASE-2987.patch
>
>
> I've extended Hadoop compression to use the LZMA algorithm and HFile to provide an option for selecting it. With typical input, the LZMA algorithm produces 30% smaller output than GZIP at max compression (which is currently the best available option for HFiles) and 15% smaller output than BZIP2. I'm aware of the "disk is cheap" mantra but for a multi-peta-scale archival application, where we still want random read and random update capabilities, 30% less disk is a substantial cost savings. LZMA compression speed is ~1 MB/second on a 2 GHz CPU, decompression speed is ~20 MB/second. This is 4x slower than BZIP2 to compress but at least 2x faster to decompress for 15% better results. For an archival application these properties would be acceptable if not for the very significant problem of flushing. Obviously the low throughput of the LZMA compressor means it is unsuitable for foreground processing. In HBase terms, it can be used for compaction but not for flush files.
> Attached patch, against 0.20 branch, turns off compression for flushes. This could be implemented as a config option, but I wonder if with the possible exception of LZO should we be compressing flushes at all? Any significant reduction in flush throughput can stall writers during periods of high write activity. Maybe globally disabling compression on flush flies is a good thing?
> I have tested this and confirmed the result is the desired behavior: 'file' shows flush files as uncompressed data, compacted files as compressed. Compaction merges files with different compression properties. LZMA provides rather extreme space savings over the other available options without slowing down writers if the regionservers are configured with enough write buffering to ride over the significantly lengthened compaction times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2987) Avoid compressing flush files
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908248#action_12908248 ]
Jonathan Gray commented on HBASE-2987:
--------------------------------------
I don't follow why flushes are foreground but compactions are background? We can flush while serving reads and taking writes.
The only difference would be during region movement, in which case, double flush prevents unavailability due to long-running flushes.
Otherwise, it's actually the compactions that we wait for when we put up the update wall not flushes, unless you're talking about the first flush not finishing before it's time for the second flush?
> Avoid compressing flush files
> -----------------------------
>
> Key: HBASE-2987
> URL: https://issues.apache.org/jira/browse/HBASE-2987
> Project: HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Attachments: HBASE-2987.patch
>
>
> I've extended Hadoop compression to use the LZMA algorithm and HFile to provide an option for selecting it. With typical input, the LZMA algorithm produces 30% smaller output than GZIP at max compression (which is currently the best available option for HFiles) and 15% smaller output than BZIP2. I'm aware of the "disk is cheap" mantra but for a multi-peta-scale archival application, where we still want random read and random update capabilities, 30% less disk is a substantial cost savings. LZMA compression speed is ~1 MB/second on a 2 GHz CPU, decompression speed is ~20 MB/second. This is 4x slower than BZIP2 to compress but at least 2x faster to decompress for 15% better results. For an archival application these properties would be acceptable if not for the very significant problem of flushing. Obviously the low throughput of the LZMA compressor means it is unsuitable for foreground processing. In HBase terms, it can be used for compaction but not for flush files.
> Attached patch, against 0.20 branch, turns off compression for flushes. This could be implemented as a config option, but I wonder if with the possible exception of LZO should we be compressing flushes at all? Any significant reduction in flush throughput can stall writers during periods of high write activity. Maybe globally disabling compression on flush flies is a good thing?
> I have tested this and confirmed the result is the desired behavior: 'file' shows flush files as uncompressed data, compacted files as compressed. Compaction merges files with different compression properties. LZMA provides rather extreme space savings over the other available options without slowing down writers if the regionservers are configured with enough write buffering to ride over the significantly lengthened compaction times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2987) Avoid compressing flush files
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908400#action_12908400 ]
Jonathan Gray commented on HBASE-2987:
--------------------------------------
bq. You have not tried to use LZMA compression before, or something like it. I'm not saying that kind of thing would be common.
Agreed (isn't that an argument to not change default?).
Let me try to be more clear on my point. In all tests I've run previously (w/ and w/o LZO compression) it is not the flush that bites you, it's the compaction. If you are tripping over the flush build-up, you are worse off than other tests because normally that is fine and it is the compactions that end up blocking stuff.
Disabling compression for flushes may prevent the flushing from stacking but then you will get hit by the even longer-running compactions.
Thinking through this again, perhaps what I'm really concluding is that it might make sense to disable compression on flushes *and* minor compactions and to enable it on major compactions (in an archival example like this using LZMA or the like). It could even make sense to use something like LZO on flushes/minors and LZMA on majors.
> Avoid compressing flush files
> -----------------------------
>
> Key: HBASE-2987
> URL: https://issues.apache.org/jira/browse/HBASE-2987
> Project: HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Attachments: HBASE-2987.patch
>
>
> I've extended Hadoop compression to use the LZMA algorithm and HFile to provide an option for selecting it. With typical input, the LZMA algorithm produces 30% smaller output than GZIP at max compression (which is currently the best available option for HFiles) and 15% smaller output than BZIP2. I'm aware of the "disk is cheap" mantra but for a multi-peta-scale archival application, where we still want random read and random update capabilities, 30% less disk is a substantial cost savings. LZMA compression speed is ~1 MB/second on a 2 GHz CPU, decompression speed is ~20 MB/second. This is 4x slower than BZIP2 to compress but at least 2x faster to decompress for 15% better results. For an archival application these properties would be acceptable if not for the very significant problem of flushing. Obviously the low throughput of the LZMA compressor means it is unsuitable for foreground processing. In HBase terms, it can be used for compaction but not for flush files.
> Attached patch, against 0.20 branch, turns off compression for flushes. This could be implemented as a config option, but I wonder if with the possible exception of LZO should we be compressing flushes at all? Any significant reduction in flush throughput can stall writers during periods of high write activity. Maybe globally disabling compression on flush flies is a good thing?
> I have tested this and confirmed the result is the desired behavior: 'file' shows flush files as uncompressed data, compacted files as compressed. Compaction merges files with different compression properties. LZMA provides rather extreme space savings over the other available options without slowing down writers if the regionservers are configured with enough write buffering to ride over the significantly lengthened compaction times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2987) Avoid compressing flush files
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908391#action_12908391 ]
Andrew Purtell commented on HBASE-2987:
---------------------------------------
bq. Or is there another situation where you have snapshotted, waiting for the flush, and the memstore fills before the snapshot gets flushed? ... I don't think I've seen the second case in practice.
You have not tried to use LZMA compression before, or something like it. I'm not saying that kind of thing would be common.
bq. But I don't think that default behavior should be to override a users compression settings for flushes.
I'll submit a proposal to RB.
> Avoid compressing flush files
> -----------------------------
>
> Key: HBASE-2987
> URL: https://issues.apache.org/jira/browse/HBASE-2987
> Project: HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Attachments: HBASE-2987.patch
>
>
> I've extended Hadoop compression to use the LZMA algorithm and HFile to provide an option for selecting it. With typical input, the LZMA algorithm produces 30% smaller output than GZIP at max compression (which is currently the best available option for HFiles) and 15% smaller output than BZIP2. I'm aware of the "disk is cheap" mantra but for a multi-peta-scale archival application, where we still want random read and random update capabilities, 30% less disk is a substantial cost savings. LZMA compression speed is ~1 MB/second on a 2 GHz CPU, decompression speed is ~20 MB/second. This is 4x slower than BZIP2 to compress but at least 2x faster to decompress for 15% better results. For an archival application these properties would be acceptable if not for the very significant problem of flushing. Obviously the low throughput of the LZMA compressor means it is unsuitable for foreground processing. In HBase terms, it can be used for compaction but not for flush files.
> Attached patch, against 0.20 branch, turns off compression for flushes. This could be implemented as a config option, but I wonder if with the possible exception of LZO should we be compressing flushes at all? Any significant reduction in flush throughput can stall writers during periods of high write activity. Maybe globally disabling compression on flush flies is a good thing?
> I have tested this and confirmed the result is the desired behavior: 'file' shows flush files as uncompressed data, compacted files as compressed. Compaction merges files with different compression properties. LZMA provides rather extreme space savings over the other available options without slowing down writers if the regionservers are configured with enough write buffering to ride over the significantly lengthened compaction times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2987) Avoid compressing flush files
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908253#action_12908253 ]
Andrew Purtell commented on HBASE-2987:
---------------------------------------
Also I agree slow compactions can trip up writers also. In this case as I experimented I was able to tune the system so compactions were not significant but could do nothing about slow flushes.
> Avoid compressing flush files
> -----------------------------
>
> Key: HBASE-2987
> URL: https://issues.apache.org/jira/browse/HBASE-2987
> Project: HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Attachments: HBASE-2987.patch
>
>
> I've extended Hadoop compression to use the LZMA algorithm and HFile to provide an option for selecting it. With typical input, the LZMA algorithm produces 30% smaller output than GZIP at max compression (which is currently the best available option for HFiles) and 15% smaller output than BZIP2. I'm aware of the "disk is cheap" mantra but for a multi-peta-scale archival application, where we still want random read and random update capabilities, 30% less disk is a substantial cost savings. LZMA compression speed is ~1 MB/second on a 2 GHz CPU, decompression speed is ~20 MB/second. This is 4x slower than BZIP2 to compress but at least 2x faster to decompress for 15% better results. For an archival application these properties would be acceptable if not for the very significant problem of flushing. Obviously the low throughput of the LZMA compressor means it is unsuitable for foreground processing. In HBase terms, it can be used for compaction but not for flush files.
> Attached patch, against 0.20 branch, turns off compression for flushes. This could be implemented as a config option, but I wonder if with the possible exception of LZO should we be compressing flushes at all? Any significant reduction in flush throughput can stall writers during periods of high write activity. Maybe globally disabling compression on flush flies is a good thing?
> I have tested this and confirmed the result is the desired behavior: 'file' shows flush files as uncompressed data, compacted files as compressed. Compaction merges files with different compression properties. LZMA provides rather extreme space savings over the other available options without slowing down writers if the regionservers are configured with enough write buffering to ride over the significantly lengthened compaction times.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.