You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hbase.apache.org by Ashish Shinde <as...@strandls.com> on 2011/02/03 16:46:25 UTC

Exception reading from Hbase table with LZO compression

Hi,

I get the following exception when reading from a table with LZO
compression (using a M/R job as well as shell) on hbase 0.90.0. 

Is this hbase related of is there a big in hadoop-gpl-compression. 

Thanks and regards,
 - Ashish


java.io.IOException: java.io.IOException:
java.lang.IllegalArgumentException at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:988)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:976)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1767)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)
Caused by: java.lang.IllegalArgumentException at
java.nio.Buffer.limit(Buffer.java:249) at
com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:289)
at
org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:74)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at
java.io.BufferedInputStream.read(BufferedInputStream.java:317) at
org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1433)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:139)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(Storenner.java:96)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:77)
at
org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1338)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2229)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1119)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1111)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1095)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1765) ...
5 more

Re: Exception reading from Hbase table with LZO compression

Posted by Ashish Shinde <as...@strandls.com>.

hi,

This was my mistake on one of the cluster machines. It had
hadoop-gpl-compression jar from google code. After removing this jar,
this error no longer came up. 

Sorry about the false alarm.

Thanks and regards,
 - Ashish



On Fri, 4 Feb 2011 10:37:33 +0530
Ashish Shinde <as...@strandls.com> wrote:

> hi Todd,
> 
> Sorry forgot to mention that.
> 
> I am using your version 0.4.9 build from 
> https://github.com/toddlipcon/hadoop-lzo
> 
> Thanks and regrads, 
>  - Ashish
> 
> 
> 
> On Thu, 3 Feb 2011 11:25:09 -0800
> Todd Lipcon <to...@cloudera.com> wrote:
> 
> > Hi Ashish,
> > 
> > Which version of the LZO libraries are you using?
> > 
> > -Todd
> > 
> > On Thu, Feb 3, 2011 at 7:46 AM, Ashish Shinde <as...@strandls.com>
> > wrote:
> > 
> > > Hi,
> > >
> > > I get the following exception when reading from a table with LZO
> > > compression (using a M/R job as well as shell) on hbase 0.90.0.
> > >
> > > Is this hbase related of is there a big in hadoop-gpl-compression.
> > >
> > > Thanks and regards,
> > >  - Ashish
> > >
> > >
> > > java.io.IOException: java.io.IOException:
> > > java.lang.IllegalArgumentException at
> > >
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:988)
> > > at
> > >
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:976)
> > > at
> > >
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1767)
> > > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at
> > >
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597) at
> > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> > > at
> > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)
> > > Caused by: java.lang.IllegalArgumentException at
> > > java.nio.Buffer.limit(Buffer.java:249) at
> > >
> > > com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:289)
> > > at
> > >
> > > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:74)
> > > at
> > >
> > > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> > > at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> > > at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> > > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at
> > > org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094)
> > > at
> > > org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> > > at
> > >
> > > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1433)
> > > at
> > >
> > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:139)
> > > at
> > >
> > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(Storenner.java:96)
> > > at
> > >
> > > org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:77)
> > > at
> > > org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1338)
> > > at
> > >
> > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2229)
> > > at
> > >
> > > org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1119)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1111)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1095)
> > > at
> > > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1765)
> > > ...
> > > 5 more
> > >
> > >
> > 
> >

Re: Exception reading from Hbase table with LZO compression

Posted by Ashish Shinde <as...@strandls.com>.

hi Todd,

Sorry forgot to mention that.

I am using your version 0.4.9 build from 
https://github.com/toddlipcon/hadoop-lzo

Thanks and regrads, 
 - Ashish



On Thu, 3 Feb 2011 11:25:09 -0800
Todd Lipcon <to...@cloudera.com> wrote:

> Hi Ashish,
> 
> Which version of the LZO libraries are you using?
> 
> -Todd
> 
> On Thu, Feb 3, 2011 at 7:46 AM, Ashish Shinde <as...@strandls.com>
> wrote:
> 
> > Hi,
> >
> > I get the following exception when reading from a table with LZO
> > compression (using a M/R job as well as shell) on hbase 0.90.0.
> >
> > Is this hbase related of is there a big in hadoop-gpl-compression.
> >
> > Thanks and regards,
> >  - Ashish
> >
> >
> > java.io.IOException: java.io.IOException:
> > java.lang.IllegalArgumentException at
> >
> > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:988)
> > at
> >
> > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:976)
> > at
> >
> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1767)
> > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at
> >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597) at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)
> > Caused by: java.lang.IllegalArgumentException at
> > java.nio.Buffer.limit(Buffer.java:249) at
> >
> > com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:289)
> > at
> >
> > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:74)
> > at
> >
> > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> > at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at
> > org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at
> > org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094)
> > at
> > org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> > at
> >
> > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1433)
> > at
> >
> > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:139)
> > at
> >
> > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(Storenner.java:96)
> > at
> >
> > org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:77)
> > at
> > org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1338)
> > at
> >
> > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2229)
> > at
> >
> > org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1119)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1111)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1095)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1765)
> > ...
> > 5 more
> >
> >
> 
>

Re: Exception reading from Hbase table with LZO compression

Posted by Todd Lipcon <to...@cloudera.com>.

Hi Ashish,

Which version of the LZO libraries are you using?

-Todd

On Thu, Feb 3, 2011 at 7:46 AM, Ashish Shinde <as...@strandls.com> wrote:

> Hi,
>
> I get the following exception when reading from a table with LZO
> compression (using a M/R job as well as shell) on hbase 0.90.0.
>
> Is this hbase related of is there a big in hadoop-gpl-compression.
>
> Thanks and regards,
>  - Ashish
>
>
> java.io.IOException: java.io.IOException:
> java.lang.IllegalArgumentException at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:988)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:976)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1767)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597) at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)
> Caused by: java.lang.IllegalArgumentException at
> java.nio.Buffer.limit(Buffer.java:249) at
>
> com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:289)
> at
>
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:74)
> at
>
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at
> java.io.BufferedInputStream.read(BufferedInputStream.java:317) at
> org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1433)
> at
>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:139)
> at
>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(Storenner.java:96)
> at
>
> org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:77)
> at
> org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1338)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2229)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1119)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1111)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1095)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1765)
> ...
> 5 more
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera