You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jean-Daniel Cryans <jd...@apache.org> on 2010/08/25 20:57:56 UTC
Review Request: RegionServer should refuse to be assigned a region that use
LZO when LZO isn't available
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/719/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
Trying to make some progress on this issue, here's an untested patch that adds testing of codecs when the region server starts, then uses that to check the schema of regions it tries to open. That's basically what this jira's scope is about.
Some things I don't like the way I did it:
- For all the users that don't do LZO, they will have a WARN every time a RS starts (in its log)
- Does not cover cases where the files are LZOed, but the schema is NONE or GZ
Finally, there's no way to let user know about the errors unless he checks the logs, but a least we won't throw ugly exceptions.
This addresses bug HBASE-2514.
http://issues.apache.org/jira/browse/HBASE-2514
Diffs
-----
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 989204
/trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 989204
Diff: http://review.cloudera.org/r/719/diff
Testing
-------
Thanks,
Jean-Daniel
Re: Review Request: RegionServer should refuse to be assigned a region that
use LZO when LZO isn't available
Posted by st...@duboce.net.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/719/#review1500
-----------------------------------------------------------
I think we have to do the test elsewhere on open of the storefile -- we read the storefile metadata and there check if the codec available. Thanks for moving this issue along J-D.
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
<http://review.cloudera.org/r/719/#comment5146>
Wouldn't it be better if this tmp dir was under hbase.rootdir? Otherwise, this is the first case of hbase writing outside of hbase.rootdir.
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
<http://review.cloudera.org/r/719/#comment5147>
Can you not iterate over supported codecs list somewhere rather than do this? What if we add a codec -- this code will have to be updated.
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
<http://review.cloudera.org/r/719/#comment5148>
What if the the storefiles we've been passed are lzo'd but schema says gz. It'll work but we could fail to open the lzo'd file if the codec is absent?
- stack
On 2010-08-25 11:57:56, Jean-Daniel Cryans wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/719/
> -----------------------------------------------------------
>
> (Updated 2010-08-25 11:57:56)
>
>
> Review request for hbase.
>
>
> Summary
> -------
>
> Trying to make some progress on this issue, here's an untested patch that adds testing of codecs when the region server starts, then uses that to check the schema of regions it tries to open. That's basically what this jira's scope is about.
>
> Some things I don't like the way I did it:
> - For all the users that don't do LZO, they will have a WARN every time a RS starts (in its log)
> - Does not cover cases where the files are LZOed, but the schema is NONE or GZ
>
> Finally, there's no way to let user know about the errors unless he checks the logs, but a least we won't throw ugly exceptions.
>
>
> This addresses bug HBASE-2514.
> http://issues.apache.org/jira/browse/HBASE-2514
>
>
> Diffs
> -----
>
> /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 989204
> /trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 989204
>
> Diff: http://review.cloudera.org/r/719/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jean-Daniel
>
>
Re: Review Request: RegionServer should refuse to be assigned a
region that use LZO when LZO isn't available
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Sure, but currently there's no infrastructure for that. It could be
part of the master rewrite, but it'd be low priority as we are still
in the process of having a functional master. Also we need that kind
of reporting outside of the web UIs as some production environment may
not have those.
J-D
On Wed, Aug 25, 2010 at 1:11 PM, Ted Yu <yu...@gmail.com> wrote:
> Can we display region server(s) which has this issue in a different color in
> Region Servers table on master.jsp ?
> Or better, can we add a column showing brief error message in Region Servers
> table ?
>
> On Wed, Aug 25, 2010 at 11:57 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>>
>> -----------------------------------------------------------
>> This is an automatically generated e-mail. To reply, visit:
>> http://review.cloudera.org/r/719/
>> -----------------------------------------------------------
>>
>> Review request for hbase.
>>
>>
>> Summary
>> -------
>>
>> Trying to make some progress on this issue, here's an untested patch that
>> adds testing of codecs when the region server starts, then uses that to
>> check the schema of regions it tries to open. That's basically what this
>> jira's scope is about.
>>
>> Some things I don't like the way I did it:
>> - For all the users that don't do LZO, they will have a WARN every time a
>> RS starts (in its log)
>> - Does not cover cases where the files are LZOed, but the schema is NONE
>> or GZ
>>
>> Finally, there's no way to let user know about the errors unless he checks
>> the logs, but a least we won't throw ugly exceptions.
>>
>>
>> This addresses bug HBASE-2514.
>> http://issues.apache.org/jira/browse/HBASE-2514
>>
>>
>> Diffs
>> -----
>>
>> /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
>> 989204
>> /trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java
>> 989204
>>
>> Diff: http://review.cloudera.org/r/719/diff
>>
>>
>> Testing
>> -------
>>
>>
>> Thanks,
>>
>> Jean-Daniel
>>
>>
>
Re: Review Request: RegionServer should refuse to be assigned a
region that use LZO when LZO isn't available
Posted by Ted Yu <yu...@gmail.com>.
Can we display region server(s) which has this issue in a different color in
Region Servers table on master.jsp ?
Or better, can we add a column showing brief error message in Region Servers
table ?
On Wed, Aug 25, 2010 at 11:57 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/719/
> -----------------------------------------------------------
>
> Review request for hbase.
>
>
> Summary
> -------
>
> Trying to make some progress on this issue, here's an untested patch that
> adds testing of codecs when the region server starts, then uses that to
> check the schema of regions it tries to open. That's basically what this
> jira's scope is about.
>
> Some things I don't like the way I did it:
> - For all the users that don't do LZO, they will have a WARN every time a
> RS starts (in its log)
> - Does not cover cases where the files are LZOed, but the schema is NONE
> or GZ
>
> Finally, there's no way to let user know about the errors unless he checks
> the logs, but a least we won't throw ugly exceptions.
>
>
> This addresses bug HBASE-2514.
> http://issues.apache.org/jira/browse/HBASE-2514
>
>
> Diffs
> -----
>
> /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
> 989204
> /trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java
> 989204
>
> Diff: http://review.cloudera.org/r/719/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jean-Daniel
>
>