You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Michele Catasta (JIRA)" <ji...@apache.org> on 2010/06/07 19:07:45 UTC
[jira] Created: (HBASE-2681) Graceful fallback to NONE when the
native compression algo is not found
Graceful fallback to NONE when the native compression algo is not found
-----------------------------------------------------------------------
Key: HBASE-2681
URL: https://issues.apache.org/jira/browse/HBASE-2681
Project: HBase
Issue Type: Improvement
Components: io
Reporter: Michele Catasta
Priority: Minor
Fix For: 0.21.0
stack's word: "Fellas have complained about the way broke lzo manifests itself. HBase will actually take on writes. Its only when it goes to flush that it drops the edits and in a way that is essentially hidden to the client - exceptions are thrown in the regionserver log. So, i'd say, make another issue if you don't mind but its not for you to fix, not unless you are inclined. It'd be about better user experience around choosing a compression that is not supported or not properly installed."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2681) Graceful fallback to NONE when the
native compression algo is not found
Posted by "Dave Latham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876413#action_12876413 ]
Dave Latham commented on HBASE-2681:
------------------------------------
I agree it's not easy as it is right now.
Would it make sense to have a set of expected, "supported" compression schemes in the settings. On regionserver startup, they check to see if the supported algorithms are available, possibly report that to master or just fail to load. Then when creating / altering a table immediately complain if the compression isn't in the supported set?
> Graceful fallback to NONE when the native compression algo is not found
> -----------------------------------------------------------------------
>
> Key: HBASE-2681
> URL: https://issues.apache.org/jira/browse/HBASE-2681
> Project: HBase
> Issue Type: Improvement
> Components: io
> Reporter: Michele Catasta
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2681.patch
>
>
> stack's word: "Fellas have complained about the way broke lzo manifests itself. HBase will actually take on writes. Its only when it goes to flush that it drops the edits and in a way that is essentially hidden to the client - exceptions are thrown in the regionserver log. So, i'd say, make another issue if you don't mind but its not for you to fix, not unless you are inclined. It'd be about better user experience around choosing a compression that is not supported or not properly installed."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2681) Graceful fallback to NONE when the
native compression algo is not found
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876405#action_12876405 ]
stack commented on HBASE-2681:
------------------------------
@Dave Currently, hbase runs and failure happens when we go to flush. Failure is 'loud' if you are looking at logs but if not looking in logs, its not plain what is wrong. You are suggesting Dave that rather than just proceed, that instead we fail even earlier? Like when you try to create a table or alter a schema to add a compression that does not exist, that it fail at this time? Doing this latter is probalby the right thing to do but its kinda tough to do in that you need to ensure the compression is available on all members of the cluster inline w/ the setting of schema. We don't have a mechanism to do this. What do we do also when a new server is added to the cluster only the admins forgot to add compression libs.... should this new node fail loudly or just, log and keep going?
> Graceful fallback to NONE when the native compression algo is not found
> -----------------------------------------------------------------------
>
> Key: HBASE-2681
> URL: https://issues.apache.org/jira/browse/HBASE-2681
> Project: HBase
> Issue Type: Improvement
> Components: io
> Reporter: Michele Catasta
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2681.patch
>
>
> stack's word: "Fellas have complained about the way broke lzo manifests itself. HBase will actually take on writes. Its only when it goes to flush that it drops the edits and in a way that is essentially hidden to the client - exceptions are thrown in the regionserver log. So, i'd say, make another issue if you don't mind but its not for you to fix, not unless you are inclined. It'd be about better user experience around choosing a compression that is not supported or not properly installed."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2681) Graceful fallback to NONE when the
native compression algo is not found
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876416#action_12876416 ]
Andrew Purtell commented on HBASE-2681:
---------------------------------------
I think it makes sense for the region server to refuse to open a region if the schema indicates to use an unavailable codec.
> Graceful fallback to NONE when the native compression algo is not found
> -----------------------------------------------------------------------
>
> Key: HBASE-2681
> URL: https://issues.apache.org/jira/browse/HBASE-2681
> Project: HBase
> Issue Type: Improvement
> Components: io
> Reporter: Michele Catasta
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2681.patch
>
>
> stack's word: "Fellas have complained about the way broke lzo manifests itself. HBase will actually take on writes. Its only when it goes to flush that it drops the edits and in a way that is essentially hidden to the client - exceptions are thrown in the regionserver log. So, i'd say, make another issue if you don't mind but its not for you to fix, not unless you are inclined. It'd be about better user experience around choosing a compression that is not supported or not properly installed."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2681) Graceful fallback to NONE when the
native compression algo is not found
Posted by "Dave Latham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876393#action_12876393 ]
Dave Latham commented on HBASE-2681:
------------------------------------
It's definitely not good to fail silently at a later time.
However, I'd prefer to see it fail loudly and immediately if I try to use a form of compression that is not available rather than a quiet fallback to no compression that I would likely not notice.
> Graceful fallback to NONE when the native compression algo is not found
> -----------------------------------------------------------------------
>
> Key: HBASE-2681
> URL: https://issues.apache.org/jira/browse/HBASE-2681
> Project: HBase
> Issue Type: Improvement
> Components: io
> Reporter: Michele Catasta
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2681.patch
>
>
> stack's word: "Fellas have complained about the way broke lzo manifests itself. HBase will actually take on writes. Its only when it goes to flush that it drops the edits and in a way that is essentially hidden to the client - exceptions are thrown in the regionserver log. So, i'd say, make another issue if you don't mind but its not for you to fix, not unless you are inclined. It'd be about better user experience around choosing a compression that is not supported or not properly installed."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2681) Graceful fallback to NONE when the
native compression algo is not found
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876406#action_12876406 ]
ryan rawson commented on HBASE-2681:
------------------------------------
During a flush/hfile write it is easy to fall back to "NONE", but when you try to read a HFile (which was written with a specific codec) and that codec fails to load then there is no good graceful failure while also returning data.
> Graceful fallback to NONE when the native compression algo is not found
> -----------------------------------------------------------------------
>
> Key: HBASE-2681
> URL: https://issues.apache.org/jira/browse/HBASE-2681
> Project: HBase
> Issue Type: Improvement
> Components: io
> Reporter: Michele Catasta
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2681.patch
>
>
> stack's word: "Fellas have complained about the way broke lzo manifests itself. HBase will actually take on writes. Its only when it goes to flush that it drops the edits and in a way that is essentially hidden to the client - exceptions are thrown in the regionserver log. So, i'd say, make another issue if you don't mind but its not for you to fix, not unless you are inclined. It'd be about better user experience around choosing a compression that is not supported or not properly installed."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2681) Graceful fallback to NONE when the
native compression algo is not found
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876419#action_12876419 ]
ryan rawson commented on HBASE-2681:
------------------------------------
A region may contain multiple files with multiple codecs. Thus even
if the schema says "GZ" the files may be "LZO". Thus the regionserver
must have all the codecs applicable.
But it's a little more insidious, in our case there are 2 failure
modes with LZO:
- missing class exceptions
- missing native library exceptions
Testing for all applicable codecs, including an end to end functional
test would probably be necessary to detect both these cases.
> Graceful fallback to NONE when the native compression algo is not found
> -----------------------------------------------------------------------
>
> Key: HBASE-2681
> URL: https://issues.apache.org/jira/browse/HBASE-2681
> Project: HBase
> Issue Type: Improvement
> Components: io
> Reporter: Michele Catasta
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2681.patch
>
>
> stack's word: "Fellas have complained about the way broke lzo manifests itself. HBase will actually take on writes. Its only when it goes to flush that it drops the edits and in a way that is essentially hidden to the client - exceptions are thrown in the regionserver log. So, i'd say, make another issue if you don't mind but its not for you to fix, not unless you are inclined. It'd be about better user experience around choosing a compression that is not supported or not properly installed."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2681) Graceful fallback to NONE when the
native compression algo is not found
Posted by "Michele Catasta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michele Catasta updated HBASE-2681:
-----------------------------------
Attachment: HBASE-2681.patch
Extracted a method that tries to instantiate a native codec and, if not found, just writes a WARN in the logs instead of throwing a RuntimeException. Trivial test included.
> Graceful fallback to NONE when the native compression algo is not found
> -----------------------------------------------------------------------
>
> Key: HBASE-2681
> URL: https://issues.apache.org/jira/browse/HBASE-2681
> Project: HBase
> Issue Type: Improvement
> Components: io
> Reporter: Michele Catasta
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HBASE-2681.patch
>
>
> stack's word: "Fellas have complained about the way broke lzo manifests itself. HBase will actually take on writes. Its only when it goes to flush that it drops the edits and in a way that is essentially hidden to the client - exceptions are thrown in the regionserver log. So, i'd say, make another issue if you don't mind but its not for you to fix, not unless you are inclined. It'd be about better user experience around choosing a compression that is not supported or not properly installed."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2681) Graceful fallback to NONE when the
native compression algo is not found
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-2681.
--------------------------
Resolution: Won't Fix
We'll go the HBASE-2514 route instead of not loading region if compression not available.
We can't go to NONE if region was written w/ a compression lib -- we won't be able to open the files if absent compression lib.
> Graceful fallback to NONE when the native compression algo is not found
> -----------------------------------------------------------------------
>
> Key: HBASE-2681
> URL: https://issues.apache.org/jira/browse/HBASE-2681
> Project: HBase
> Issue Type: Improvement
> Components: io
> Reporter: Michele Catasta
> Priority: Minor
> Fix For: 0.90.0
>
> Attachments: HBASE-2681.patch
>
>
> stack's word: "Fellas have complained about the way broke lzo manifests itself. HBase will actually take on writes. Its only when it goes to flush that it drops the edits and in a way that is essentially hidden to the client - exceptions are thrown in the regionserver log. So, i'd say, make another issue if you don't mind but its not for you to fix, not unless you are inclined. It'd be about better user experience around choosing a compression that is not supported or not properly installed."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.