You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ryan Rawson <ry...@gmail.com> on 2010/10/19 04:30:09 UTC
Review Request: HBASE-2514 RegionServer should refuse to be assigned a
region that use LZO when LZO isn't available
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1046/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
RegionServer should refuse to be assigned a region that use LZO when LZO isn't available
- instead of checking regions during open, i do a check and throw during the construction of the RS that is is able to instantiate the listed codecs.
This addresses bug HBASE-2514.
http://issues.apache.org/jira/browse/HBASE-2514
Diffs
-----
trunk/CHANGES.txt 1024074
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1024074
trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1024073
trunk/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java PRE-CREATION
Diff: http://review.cloudera.org/r/1046/diff
Testing
-------
Thanks,
Ryan
Re: Review Request: HBASE-2514 RegionServer should refuse to be assigned a
region that use LZO when LZO isn't available
Posted by st...@duboce.net.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1046/#review1628
-----------------------------------------------------------
Chatting offline w/ R and J-D, this patch is good if hfiles but no good if no files yet (Ryan pointed this out). So, we need combo of this and J-Ds schema check. RS won't go down. RS just won't deploy a region that requires an uninstalled codec. I think thats the way to go.
trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
<http://review.cloudera.org/r/1046/#comment5483>
Remove TODO
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
<http://review.cloudera.org/r/1046/#comment5484>
Where is 'feature' documented?
- stack
On 2010-10-22 15:43:55, Ryan Rawson wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/1046/
> -----------------------------------------------------------
>
> (Updated 2010-10-22 15:43:55)
>
>
> Review request for hbase.
>
>
> Summary
> -------
>
> RegionServer should refuse to be assigned a region that use LZO when LZO isn't available
>
> - instead of checking regions during open, i do a check and throw during the construction of the RS that is is able to instantiate the listed codecs.
>
>
> This addresses bug HBASE-2514.
> http://issues.apache.org/jira/browse/HBASE-2514
>
>
> Diffs
> -----
>
> trunk/CHANGES.txt 1024074
> trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java 1024073
> trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1024073
> trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1024074
> trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1024073
> trunk/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/1046/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ryan
>
>
Re: Review Request: HBASE-2514 RegionServer should refuse to be assigned a
region that use LZO when LZO isn't available
Posted by Ryan Rawson <ry...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1046/
-----------------------------------------------------------
(Updated 2010-10-22 16:38:10.455573)
Review request for hbase.
Summary
-------
RegionServer should refuse to be assigned a region that use LZO when LZO isn't available
- instead of checking regions during open, i do a check and throw during the construction of the RS that is is able to instantiate the listed codecs.
This addresses bug HBASE-2514.
http://issues.apache.org/jira/browse/HBASE-2514
Diffs (updated)
-----
trunk/CHANGES.txt 1024074
trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java 1024073
trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1024073
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1024073
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1024074
trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1024073
trunk/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java PRE-CREATION
Diff: http://review.cloudera.org/r/1046/diff
Testing
-------
Thanks,
Ryan
Re: Review Request: HBASE-2514 RegionServer should refuse to be assigned a
region that use LZO when LZO isn't available
Posted by Ryan Rawson <ry...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1046/
-----------------------------------------------------------
(Updated 2010-10-22 15:43:55.487649)
Review request for hbase.
Changes
-------
attempt to instantiate the codec inside HFile read trailer so we get IOE on file open not on first read.
Summary
-------
RegionServer should refuse to be assigned a region that use LZO when LZO isn't available
- instead of checking regions during open, i do a check and throw during the construction of the RS that is is able to instantiate the listed codecs.
This addresses bug HBASE-2514.
http://issues.apache.org/jira/browse/HBASE-2514
Diffs (updated)
-----
trunk/CHANGES.txt 1024074
trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java 1024073
trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1024073
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1024074
trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1024073
trunk/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java PRE-CREATION
Diff: http://review.cloudera.org/r/1046/diff
Testing
-------
Thanks,
Ryan
Re: Review Request: HBASE-2514 RegionServer should refuse to be assigned a
region that use LZO when LZO isn't available
Posted by Ryan Rawson <ry...@gmail.com>.
> On 2010-10-20 12:41:06, stack wrote:
> > trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, line 318
> > <http://review.cloudera.org/r/1046/diff/1/?file=14924#file14924line318>
> >
> > So this String is not defined so we don't do any compression test with this patch, right?
> >
> > Is this patch what we want? Issue is about not loading a region that needs lzo. Thats one of two things: a new table that has lzo specified though we don't have lzo loaded; or its an HRS being asked to open a region that was written by another HRS that had lzo installed -- and this does not. In the issue we thought the region should just not be opened. If this HRS is errant being the only one w/ no LZO installed, then he'll open non-lzo'd regions. As is, HRSs will go down... a whole cluster could go down just because a silly admin enabled lzo on a table that had a region per HRS in a cluster.
i think this is the patch we want. If you look at JD's patch, he points out that it only checks the schema, not the store files. If any of the store files are compressed with LZO but the schema is something else, then his patch will give it a 'pass' but it will fail to read. Checking all the store files is pretty time consuming and expensive.
Instead, we can give operators tools to keep themselves out of trouble. If they intend to run LZO, but a single machine is misconfigured, refuse to run instead of logging some difficult to find error.
I dont think it's a common case to run uneven configured machines and expect the LZO-ed regions to 'flow' to the 'correct' regionservers. If we really want to do that, we should add capabilities to regionservers and requirements to regions and stop the assignment in the master. Seems excessive though.
- Ryan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1046/#review1574
-----------------------------------------------------------
On 2010-10-18 19:30:09, Ryan Rawson wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/1046/
> -----------------------------------------------------------
>
> (Updated 2010-10-18 19:30:09)
>
>
> Review request for hbase.
>
>
> Summary
> -------
>
> RegionServer should refuse to be assigned a region that use LZO when LZO isn't available
>
> - instead of checking regions during open, i do a check and throw during the construction of the RS that is is able to instantiate the listed codecs.
>
>
> This addresses bug HBASE-2514.
> http://issues.apache.org/jira/browse/HBASE-2514
>
>
> Diffs
> -----
>
> trunk/CHANGES.txt 1024074
> trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1024074
> trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1024073
> trunk/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/1046/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ryan
>
>
Re: Review Request: HBASE-2514 RegionServer should refuse to be assigned a
region that use LZO when LZO isn't available
Posted by st...@duboce.net.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1046/#review1574
-----------------------------------------------------------
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
<http://review.cloudera.org/r/1046/#comment5291>
So this String is not defined so we don't do any compression test with this patch, right?
Is this patch what we want? Issue is about not loading a region that needs lzo. Thats one of two things: a new table that has lzo specified though we don't have lzo loaded; or its an HRS being asked to open a region that was written by another HRS that had lzo installed -- and this does not. In the issue we thought the region should just not be opened. If this HRS is errant being the only one w/ no LZO installed, then he'll open non-lzo'd regions. As is, HRSs will go down... a whole cluster could go down just because a silly admin enabled lzo on a table that had a region per HRS in a cluster.
- stack
On 2010-10-18 19:30:09, Ryan Rawson wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/1046/
> -----------------------------------------------------------
>
> (Updated 2010-10-18 19:30:09)
>
>
> Review request for hbase.
>
>
> Summary
> -------
>
> RegionServer should refuse to be assigned a region that use LZO when LZO isn't available
>
> - instead of checking regions during open, i do a check and throw during the construction of the RS that is is able to instantiate the listed codecs.
>
>
> This addresses bug HBASE-2514.
> http://issues.apache.org/jira/browse/HBASE-2514
>
>
> Diffs
> -----
>
> trunk/CHANGES.txt 1024074
> trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1024074
> trunk/src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1024073
> trunk/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java PRE-CREATION
>
> Diff: http://review.cloudera.org/r/1046/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ryan
>
>