You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Amit Sela <am...@infolinks.com> on 2013/09/03 17:37:19 UTC
Bulk load from OSGi running client
Hi all,
I'm running on Hadoop 1.0.4 with HBase 0.94.2 and I've bundled both (for
client side use only) so that I could support execution of MR and/or HBase
queries (and other client operations) from an OSGi environment (in my case
Felix).
So far, I've managed (context class loader adjustments) to execute MR jobs
and to query HBase (gert, put...) with no problem.
*I'm trying to execute Bulk Load into HBase and I seem to encounter a
strange NullPointerException:*
Caused by: java.lang.NullPointerException: null
at
org.apache.felix.framework.BundleRevisionImpl.getResourceLocal(BundleRevisionImpl.java:474)
at
org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1432)
at
org.apache.felix.framework.BundleWiringImpl.getResourceByDelegation(BundleWiringImpl.java:1360)
at
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.getResource(BundleWiringImpl.java:2256)
at org.apache.hadoop.conf.Configuration.getResource(Configuration.java:1002)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1112)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1056)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:401)
at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:471)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:131)
at
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.createDecompressionStream(Compression.java:223)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.decompress(HFileBlock.java:1392)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1897)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1286)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1294)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.<init>(HFileReaderV2.java:126)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:552)
at
org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:589)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:603)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:402)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:323)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:321)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
*I get this every time I try to bulk load after server restart following
with a bundle update (update is done after restart, so the update calls for
refresh packages).*
*Strangely, if I immediately try again, success. Any following attempts
succeed as well.*
*
*
*Any ideas anyone ?*
*
*
*Thanks, *
*
*
*Amit. *
Re: Bulk load from OSGi running client
Posted by Amit Sela <am...@infolinks.com>.
I opened a ticket HBASE-9682<https://issues.apache.org/jira/browse/HBASE-9682>.
I will try to provide some sort of sample project.
Thanks.
On Fri, Sep 27, 2013 at 8:06 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> Hi Amit,
>
> Would you be able to open a ticket summarizing your findings? Can you
> provide a sample project that demonstrates the behavior you're seeing? We
> could use that to provide a fix and, I hope, some kind of unit or
> integration test.
>
> Thanks,
> Nick
>
>
> On Sun, Sep 22, 2013 at 6:10 AM, Amit Sela <am...@infolinks.com> wrote:
>
> > I think I got it.
> > Algorithm has a Configuration member that is instantiated when Algorithm
> is
> > created. Since Algorithm is static, when I update a jar that doesn't
> cause
> > the hbase bundle to update, configuration still holds to the old CL.
> > I tested with GZ and changed the code to call new Configuration (in
> > getCodec(Configuration conf) GZ override) and it works. Makes sense
> because
> > it probably takes the current TCCL.
> >
> > Since Algorithm.getCodec(Configuration conf) is only called from inside
> > Algorithm and the only (apparent) reason to have configuration as a
> member
> > of Algorithm is to avoid code duplication of
> > *conf.setBoolean("hadoop.native.lib",
> > true)* for every codec, I think it would be better to duplicate code here
> > even if it's not as pretty as it is now, because on top of cloning the
> > configuration properties, the clone constructor also clones the CL and
> that
> > may cause problems where CL changes like in OSGi environment.
> >
> > The NPE causing recovery is still a mystery to me :)
> >
> >
> > On Wed, Sep 11, 2013 at 3:34 PM, Amit Sela <am...@infolinks.com> wrote:
> >
> > > I did some more digging and I got this:
> > >
> > > When the HBase bundle is loaded (system start) the
> Compression.Algorithm
> > > is probably created for the first time and the constructor calls new
> > > Configuration().
> > > When I update a bundle (but not the HBase bundle) it refreshes the
> > > relevant packages, which doesn't include the HBase bundle, and once I
> try
> > > to use getCodec() (GZ in my case) it creates a
> > > new ReusableStreamGzipCodec() and sets new Configuration(*conf*) -
> where
> > *conf
> > > *is a private final member in Algorithm. Since *conf *holds the old
> class
> > > loader (referring to the pre-update bundle) it passes that CL to the
> new
> > > configuration created for the codec.
> > >
> > > I still have NO IDEA why NPE would cause Compression.Algorithm to
> > > re-instantiate itself...
> > > I think that calling new Configuration() and
> > > setting this.conf.setBoolean("hadoop.native.lib", true) for each codec
> > > would solve it since the class loader that would be set is the TCCL.
> > >
> > > I'll give it a try an keep updating.
> > >
> > > Thanks,
> > > Amit.
> > >
> > >
> > >
> > > On Mon, Sep 9, 2013 at 9:12 PM, Stack <st...@duboce.net> wrote:
> > >
> > >> On Mon, Sep 9, 2013 at 12:14 AM, Amit Sela <am...@infolinks.com>
> wrote:
> > >> ...
> > >>
> > >> > The main issue still remains, it looks like Compression.Algortihm
> > >> > configuration's class loader had reference to the bundle in
> revision 0
> > >> > (before jar update) instead of revision 1 (after jar update). This
> > >> could be
> > >> > because of caching (or static) but then, why should it work after I
> > get
> > >> > NullPointerException (it does, immediately, no restarts or bundle
> > >> updates).
> > >> >
> > >>
> > >> When you say configuration above, you mean Compression.Algorithm's
> > >> reference to an Hadoop Configuration instance? I've not looked at
> code.
> > >> Is it coming in via static?
> > >>
> > >> I am not sure what it would then start working after NPE. I would
> > expect
> > >> that it would stay broke rather than recover.
> > >>
> > >> St.Ack
> > >>
> > >
> > >
> >
>
Re: Bulk load from OSGi running client
Posted by Nick Dimiduk <nd...@gmail.com>.
Hi Amit,
Would you be able to open a ticket summarizing your findings? Can you
provide a sample project that demonstrates the behavior you're seeing? We
could use that to provide a fix and, I hope, some kind of unit or
integration test.
Thanks,
Nick
On Sun, Sep 22, 2013 at 6:10 AM, Amit Sela <am...@infolinks.com> wrote:
> I think I got it.
> Algorithm has a Configuration member that is instantiated when Algorithm is
> created. Since Algorithm is static, when I update a jar that doesn't cause
> the hbase bundle to update, configuration still holds to the old CL.
> I tested with GZ and changed the code to call new Configuration (in
> getCodec(Configuration conf) GZ override) and it works. Makes sense because
> it probably takes the current TCCL.
>
> Since Algorithm.getCodec(Configuration conf) is only called from inside
> Algorithm and the only (apparent) reason to have configuration as a member
> of Algorithm is to avoid code duplication of
> *conf.setBoolean("hadoop.native.lib",
> true)* for every codec, I think it would be better to duplicate code here
> even if it's not as pretty as it is now, because on top of cloning the
> configuration properties, the clone constructor also clones the CL and that
> may cause problems where CL changes like in OSGi environment.
>
> The NPE causing recovery is still a mystery to me :)
>
>
> On Wed, Sep 11, 2013 at 3:34 PM, Amit Sela <am...@infolinks.com> wrote:
>
> > I did some more digging and I got this:
> >
> > When the HBase bundle is loaded (system start) the Compression.Algorithm
> > is probably created for the first time and the constructor calls new
> > Configuration().
> > When I update a bundle (but not the HBase bundle) it refreshes the
> > relevant packages, which doesn't include the HBase bundle, and once I try
> > to use getCodec() (GZ in my case) it creates a
> > new ReusableStreamGzipCodec() and sets new Configuration(*conf*) - where
> *conf
> > *is a private final member in Algorithm. Since *conf *holds the old class
> > loader (referring to the pre-update bundle) it passes that CL to the new
> > configuration created for the codec.
> >
> > I still have NO IDEA why NPE would cause Compression.Algorithm to
> > re-instantiate itself...
> > I think that calling new Configuration() and
> > setting this.conf.setBoolean("hadoop.native.lib", true) for each codec
> > would solve it since the class loader that would be set is the TCCL.
> >
> > I'll give it a try an keep updating.
> >
> > Thanks,
> > Amit.
> >
> >
> >
> > On Mon, Sep 9, 2013 at 9:12 PM, Stack <st...@duboce.net> wrote:
> >
> >> On Mon, Sep 9, 2013 at 12:14 AM, Amit Sela <am...@infolinks.com> wrote:
> >> ...
> >>
> >> > The main issue still remains, it looks like Compression.Algortihm
> >> > configuration's class loader had reference to the bundle in revision 0
> >> > (before jar update) instead of revision 1 (after jar update). This
> >> could be
> >> > because of caching (or static) but then, why should it work after I
> get
> >> > NullPointerException (it does, immediately, no restarts or bundle
> >> updates).
> >> >
> >>
> >> When you say configuration above, you mean Compression.Algorithm's
> >> reference to an Hadoop Configuration instance? I've not looked at code.
> >> Is it coming in via static?
> >>
> >> I am not sure what it would then start working after NPE. I would
> expect
> >> that it would stay broke rather than recover.
> >>
> >> St.Ack
> >>
> >
> >
>
Re: Bulk load from OSGi running client
Posted by Amit Sela <am...@infolinks.com>.
I think I got it.
Algorithm has a Configuration member that is instantiated when Algorithm is
created. Since Algorithm is static, when I update a jar that doesn't cause
the hbase bundle to update, configuration still holds to the old CL.
I tested with GZ and changed the code to call new Configuration (in
getCodec(Configuration conf) GZ override) and it works. Makes sense because
it probably takes the current TCCL.
Since Algorithm.getCodec(Configuration conf) is only called from inside
Algorithm and the only (apparent) reason to have configuration as a member
of Algorithm is to avoid code duplication of
*conf.setBoolean("hadoop.native.lib",
true)* for every codec, I think it would be better to duplicate code here
even if it's not as pretty as it is now, because on top of cloning the
configuration properties, the clone constructor also clones the CL and that
may cause problems where CL changes like in OSGi environment.
The NPE causing recovery is still a mystery to me :)
On Wed, Sep 11, 2013 at 3:34 PM, Amit Sela <am...@infolinks.com> wrote:
> I did some more digging and I got this:
>
> When the HBase bundle is loaded (system start) the Compression.Algorithm
> is probably created for the first time and the constructor calls new
> Configuration().
> When I update a bundle (but not the HBase bundle) it refreshes the
> relevant packages, which doesn't include the HBase bundle, and once I try
> to use getCodec() (GZ in my case) it creates a
> new ReusableStreamGzipCodec() and sets new Configuration(*conf*) - where *conf
> *is a private final member in Algorithm. Since *conf *holds the old class
> loader (referring to the pre-update bundle) it passes that CL to the new
> configuration created for the codec.
>
> I still have NO IDEA why NPE would cause Compression.Algorithm to
> re-instantiate itself...
> I think that calling new Configuration() and
> setting this.conf.setBoolean("hadoop.native.lib", true) for each codec
> would solve it since the class loader that would be set is the TCCL.
>
> I'll give it a try an keep updating.
>
> Thanks,
> Amit.
>
>
>
> On Mon, Sep 9, 2013 at 9:12 PM, Stack <st...@duboce.net> wrote:
>
>> On Mon, Sep 9, 2013 at 12:14 AM, Amit Sela <am...@infolinks.com> wrote:
>> ...
>>
>> > The main issue still remains, it looks like Compression.Algortihm
>> > configuration's class loader had reference to the bundle in revision 0
>> > (before jar update) instead of revision 1 (after jar update). This
>> could be
>> > because of caching (or static) but then, why should it work after I get
>> > NullPointerException (it does, immediately, no restarts or bundle
>> updates).
>> >
>>
>> When you say configuration above, you mean Compression.Algorithm's
>> reference to an Hadoop Configuration instance? I've not looked at code.
>> Is it coming in via static?
>>
>> I am not sure what it would then start working after NPE. I would expect
>> that it would stay broke rather than recover.
>>
>> St.Ack
>>
>
>
Re: Bulk load from OSGi running client
Posted by Amit Sela <am...@infolinks.com>.
I did some more digging and I got this:
When the HBase bundle is loaded (system start) the Compression.Algorithm is
probably created for the first time and the constructor calls new
Configuration().
When I update a bundle (but not the HBase bundle) it refreshes the relevant
packages, which doesn't include the HBase bundle, and once I try to use
getCodec() (GZ in my case) it creates a new ReusableStreamGzipCodec() and
sets new Configuration(*conf*) - where *conf *is a private final member in
Algorithm. Since *conf *holds the old class loader (referring to the
pre-update bundle) it passes that CL to the new configuration created for
the codec.
I still have NO IDEA why NPE would cause Compression.Algorithm to
re-instantiate itself...
I think that calling new Configuration() and
setting this.conf.setBoolean("hadoop.native.lib", true) for each codec
would solve it since the class loader that would be set is the TCCL.
I'll give it a try an keep updating.
Thanks,
Amit.
On Mon, Sep 9, 2013 at 9:12 PM, Stack <st...@duboce.net> wrote:
> On Mon, Sep 9, 2013 at 12:14 AM, Amit Sela <am...@infolinks.com> wrote:
> ...
>
> > The main issue still remains, it looks like Compression.Algortihm
> > configuration's class loader had reference to the bundle in revision 0
> > (before jar update) instead of revision 1 (after jar update). This could
> be
> > because of caching (or static) but then, why should it work after I get
> > NullPointerException (it does, immediately, no restarts or bundle
> updates).
> >
>
> When you say configuration above, you mean Compression.Algorithm's
> reference to an Hadoop Configuration instance? I've not looked at code.
> Is it coming in via static?
>
> I am not sure what it would then start working after NPE. I would expect
> that it would stay broke rather than recover.
>
> St.Ack
>
Re: Bulk load from OSGi running client
Posted by Stack <st...@duboce.net>.
On Mon, Sep 9, 2013 at 12:14 AM, Amit Sela <am...@infolinks.com> wrote:
...
> The main issue still remains, it looks like Compression.Algortihm
> configuration's class loader had reference to the bundle in revision 0
> (before jar update) instead of revision 1 (after jar update). This could be
> because of caching (or static) but then, why should it work after I get
> NullPointerException (it does, immediately, no restarts or bundle updates).
>
When you say configuration above, you mean Compression.Algorithm's
reference to an Hadoop Configuration instance? I've not looked at code.
Is it coming in via static?
I am not sure what it would then start working after NPE. I would expect
that it would stay broke rather than recover.
St.Ack
Re: Bulk load from OSGi running client
Posted by Amit Sela <am...@infolinks.com>.
First issue I found was that I didn't bundle the libhadoop.so in my hadoop
bundle (I saw a lot of "Got brand new decompressor" in the log), that is
fixed now.
The main issue still remains, it looks like Compression.Algortihm
configuration's class loader had reference to the bundle in revision 0
(before jar update) instead of revision 1 (after jar update). This could be
because of caching (or static) but then, why should it work after I get
NullPointerException (it does, immediately, no restarts or bundle updates).
If anyone has any idea please share, I will keep posting my findings.
Thanks,
Amit.
On Sun, Sep 8, 2013 at 3:57 AM, Stack <st...@duboce.net> wrote:
> On Sun, Sep 8, 2013 at 4:19 AM, Amit Sela <am...@infolinks.com> wrote:
>
> > I did some debug and I have more input about my issue. The Configuration
> in
> > Compression.Algorithm has a class loader that holds reference to the
> > original package (loaded at restart) and not to e current one (loaded
> after
> > package update). Is the compression algorithm cached somewhere such that
> > after a first time read (get, scan) from hbase, the following uses use a
> > cached instance ?
>
>
> Yes. Does this rather than reload each time.
>
> Let me know if you need more help getting this all up and running. Am
> interested in your findings.
>
> St.Ack
>
Re: Bulk load from OSGi running client
Posted by Stack <st...@duboce.net>.
On Sun, Sep 8, 2013 at 4:19 AM, Amit Sela <am...@infolinks.com> wrote:
> I did some debug and I have more input about my issue. The Configuration in
> Compression.Algorithm has a class loader that holds reference to the
> original package (loaded at restart) and not to e current one (loaded after
> package update). Is the compression algorithm cached somewhere such that
> after a first time read (get, scan) from hbase, the following uses use a
> cached instance ?
Yes. Does this rather than reload each time.
Let me know if you need more help getting this all up and running. Am
interested in your findings.
St.Ack
Re: Bulk load from OSGi running client
Posted by Amit Sela <am...@infolinks.com>.
I did some debug and I have more input about my issue. The Configuration in
Compression.Algorithm has a class loader that holds reference to the
original package (loaded at restart) and not to e current one (loaded after
package update). Is the compression algorithm cached somewhere such that
after a first time read (get, scan) from hbase, the following uses use a
cached instance ?
On Sep 3, 2013 6:37 PM, "Amit Sela" <am...@infolinks.com> wrote:
> Hi all,
>
> I'm running on Hadoop 1.0.4 with HBase 0.94.2 and I've bundled both (for
> client side use only) so that I could support execution of MR and/or HBase
> queries (and other client operations) from an OSGi environment (in my case
> Felix).
>
> So far, I've managed (context class loader adjustments) to execute MR jobs
> and to query HBase (gert, put...) with no problem.
>
> *I'm trying to execute Bulk Load into HBase and I seem to encounter a
> strange NullPointerException:*
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.felix.framework.BundleRevisionImpl.getResourceLocal(BundleRevisionImpl.java:474)
> at
> org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1432)
> at
> org.apache.felix.framework.BundleWiringImpl.getResourceByDelegation(BundleWiringImpl.java:1360)
> at
> org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.getResource(BundleWiringImpl.java:2256)
> at
> org.apache.hadoop.conf.Configuration.getResource(Configuration.java:1002)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1112)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1056)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:401)
> at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:471)
> at
> org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:131)
> at
> org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.createDecompressionStream(Compression.java:223)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.decompress(HFileBlock.java:1392)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1897)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1286)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1294)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.<init>(HFileReaderV2.java:126)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:552)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:589)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:603)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:402)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:323)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:321)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> *I get this every time I try to bulk load after server restart following
> with a bundle update (update is done after restart, so the update calls for
> refresh packages).*
> *Strangely, if I immediately try again, success. Any following attempts
> succeed as well.*
> *
> *
> *Any ideas anyone ?*
> *
> *
> *Thanks, *
> *
> *
> *Amit. *
>