You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Amit Sela <am...@infolinks.com> on 2014/03/19 18:53:00 UTC
Class loading in Hadoop and HBase
Hi all,
I'm running with Hadoop 1.0.4 and HBase 0.94.12 bundled (OSGi) versions I
built.
Most issues I encountered are related to class loaders.
One of the patterns I noticed in both projects is:
ClassLoader cl = Thread.currentThread().getContextClassLoader();
if(cl == null) {
cl = Clazz.class.getClassLoader();
}
Where Clazz is the Class containing this code.
I was wondering about this choice... Why not go the other way around:
ClassLoader cl = Clazz.class.getClassLoader();
if(cl == null) {
cl = Thread.currentThread().getContextClassLoader();
}
And in a more general note, why not always use Configuration (and let it's
cl be this.getClass().getClassLoader()) to load classes ?
That would surely help in integration with modularity frameworks.
Thanks,
Amit.
Re: Class loading in Hadoop and HBase
Posted by Amit Sela <am...@infolinks.com>.
I'm just trying to understand if making these changes in my client
Hadoop/HBase JARs could cause trouble...
On Thu, Mar 20, 2014 at 11:19 AM, Amit Sela <am...@infolinks.com> wrote:
> And what's wrong with: ClassLoader cl = Clazz.class.getClassLoader();
>
>
>
> On Thu, Mar 20, 2014 at 11:03 AM, haosdent <ha...@gmail.com> wrote:
>
>> > should prefer the Class's CL because the client's application affects
>> the
>> TCCL.
>> You could call "Thread.currentThread().setContextClassLoader(classLoader)"
>> before the classloader load "Configuration" or "HbaseObjectWritable"
>> class.
>>
>> You consider the logic below should be better:
>> ClassLoader cl = Clazz.class.getClassLoader();
>> if (cl == null) {
>> cl = Thread.currentThread().getContextClassLoader();
>> }
>>
>> But variable "cl" couln't be null here. So the code snippet above is
>> equivalent to the following code:
>>
>> ClassLoader cl = Clazz.class.getClassLoader();
>>
>>
>> On Thu, Mar 20, 2014 at 4:39 PM, Amit Sela <am...@infolinks.com> wrote:
>>
>> > I understand the benefits of TCCL, but I think, and I would like to hear
>> > the "why not", that code that may run in the client side (like in the
>> case
>> > of Configuration and HbaseObjectWritable examples I gave) should prefer
>> the
>> > Class's CL because the client's application affects the TCCL.
>> >
>> >
>> > On Thu, Mar 20, 2014 at 8:56 AM, haosdent <ha...@gmail.com> wrote:
>> >
>> > > Because we could change the value of "Thread.currentThread().
>> > > getContextClassLoader()" by
>> > "Thread.currentThread().setContextClassLoader(
>> > > classLoader)". But we couldn't change the value of
>> > > "this.getClass().getClassLoader()".
>> > > Maybe a little confused, TCCL is more flexible than the class's CL.
>> > >
>> > >
>> > > On Thu, Mar 20, 2014 at 2:43 PM, Amit Sela <am...@infolinks.com>
>> wrote:
>> > >
>> > > > I understand that. But why ? Why prefer the TCCL over the class's
>> CL ?
>> > > > On Mar 20, 2014 3:00 AM, "haosdent" <ha...@gmail.com> wrote:
>> > > >
>> > > > > Sometimes we would use
>> "Thread.currentThread().setContextClassLoader(
>> > > > > classLoader);" to specify classloader. So
>> > > > > "Thread.currentThread().getContextClassLoader();"
>> > > > > should be the first choice.
>> > > > >
>> > > > >
>> > > > > On Thu, Mar 20, 2014 at 1:53 AM, Amit Sela <am...@infolinks.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > > I'm running with Hadoop 1.0.4 and HBase 0.94.12 bundled (OSGi)
>> > > > versions I
>> > > > > > built.
>> > > > > > Most issues I encountered are related to class loaders.
>> > > > > >
>> > > > > > One of the patterns I noticed in both projects is:
>> > > > > >
>> > > > > > ClassLoader cl =
>> > Thread.currentThread().getContextClassLoader();
>> > > > > > if(cl == null) {
>> > > > > > cl = Clazz.class.getClassLoader();
>> > > > > > }
>> > > > > >
>> > > > > > Where Clazz is the Class containing this code.
>> > > > > >
>> > > > > > I was wondering about this choice... Why not go the other way
>> > around:
>> > > > > >
>> > > > > > ClassLoader cl = Clazz.class.getClassLoader();
>> > > > > > if(cl == null) {
>> > > > > > cl = Thread.currentThread().getContextClassLoader();
>> > > > > > }
>> > > > > >
>> > > > > > And in a more general note, why not always use Configuration
>> (and
>> > let
>> > > > > it's
>> > > > > > cl be this.getClass().getClassLoader()) to load classes ?
>> > > > > >
>> > > > > > That would surely help in integration with modularity
>> frameworks.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Amit.
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Best Regards,
>> > > > > Haosdent Huang
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Best Regards,
>> > > Haosdent Huang
>> > >
>> >
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>
Re: Class loading in Hadoop and HBase
Posted by Amit Sela <am...@infolinks.com>.
And what's wrong with: ClassLoader cl = Clazz.class.getClassLoader();
On Thu, Mar 20, 2014 at 11:03 AM, haosdent <ha...@gmail.com> wrote:
> > should prefer the Class's CL because the client's application affects the
> TCCL.
> You could call "Thread.currentThread().setContextClassLoader(classLoader)"
> before the classloader load "Configuration" or "HbaseObjectWritable" class.
>
> You consider the logic below should be better:
> ClassLoader cl = Clazz.class.getClassLoader();
> if (cl == null) {
> cl = Thread.currentThread().getContextClassLoader();
> }
>
> But variable "cl" couln't be null here. So the code snippet above is
> equivalent to the following code:
>
> ClassLoader cl = Clazz.class.getClassLoader();
>
>
> On Thu, Mar 20, 2014 at 4:39 PM, Amit Sela <am...@infolinks.com> wrote:
>
> > I understand the benefits of TCCL, but I think, and I would like to hear
> > the "why not", that code that may run in the client side (like in the
> case
> > of Configuration and HbaseObjectWritable examples I gave) should prefer
> the
> > Class's CL because the client's application affects the TCCL.
> >
> >
> > On Thu, Mar 20, 2014 at 8:56 AM, haosdent <ha...@gmail.com> wrote:
> >
> > > Because we could change the value of "Thread.currentThread().
> > > getContextClassLoader()" by
> > "Thread.currentThread().setContextClassLoader(
> > > classLoader)". But we couldn't change the value of
> > > "this.getClass().getClassLoader()".
> > > Maybe a little confused, TCCL is more flexible than the class's CL.
> > >
> > >
> > > On Thu, Mar 20, 2014 at 2:43 PM, Amit Sela <am...@infolinks.com>
> wrote:
> > >
> > > > I understand that. But why ? Why prefer the TCCL over the class's CL
> ?
> > > > On Mar 20, 2014 3:00 AM, "haosdent" <ha...@gmail.com> wrote:
> > > >
> > > > > Sometimes we would use
> "Thread.currentThread().setContextClassLoader(
> > > > > classLoader);" to specify classloader. So
> > > > > "Thread.currentThread().getContextClassLoader();"
> > > > > should be the first choice.
> > > > >
> > > > >
> > > > > On Thu, Mar 20, 2014 at 1:53 AM, Amit Sela <am...@infolinks.com>
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > > I'm running with Hadoop 1.0.4 and HBase 0.94.12 bundled (OSGi)
> > > > versions I
> > > > > > built.
> > > > > > Most issues I encountered are related to class loaders.
> > > > > >
> > > > > > One of the patterns I noticed in both projects is:
> > > > > >
> > > > > > ClassLoader cl =
> > Thread.currentThread().getContextClassLoader();
> > > > > > if(cl == null) {
> > > > > > cl = Clazz.class.getClassLoader();
> > > > > > }
> > > > > >
> > > > > > Where Clazz is the Class containing this code.
> > > > > >
> > > > > > I was wondering about this choice... Why not go the other way
> > around:
> > > > > >
> > > > > > ClassLoader cl = Clazz.class.getClassLoader();
> > > > > > if(cl == null) {
> > > > > > cl = Thread.currentThread().getContextClassLoader();
> > > > > > }
> > > > > >
> > > > > > And in a more general note, why not always use Configuration (and
> > let
> > > > > it's
> > > > > > cl be this.getClass().getClassLoader()) to load classes ?
> > > > > >
> > > > > > That would surely help in integration with modularity frameworks.
> > > > > >
> > > > > > Thanks,
> > > > > > Amit.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best Regards,
> > > > > Haosdent Huang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Haosdent Huang
> > >
> >
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Class loading in Hadoop and HBase
Posted by haosdent <ha...@gmail.com>.
> should prefer the Class's CL because the client's application affects the
TCCL.
You could call "Thread.currentThread().setContextClassLoader(classLoader)"
before the classloader load "Configuration" or "HbaseObjectWritable" class.
You consider the logic below should be better:
ClassLoader cl = Clazz.class.getClassLoader();
if (cl == null) {
cl = Thread.currentThread().getContextClassLoader();
}
But variable "cl" couln't be null here. So the code snippet above is
equivalent to the following code:
ClassLoader cl = Clazz.class.getClassLoader();
On Thu, Mar 20, 2014 at 4:39 PM, Amit Sela <am...@infolinks.com> wrote:
> I understand the benefits of TCCL, but I think, and I would like to hear
> the "why not", that code that may run in the client side (like in the case
> of Configuration and HbaseObjectWritable examples I gave) should prefer the
> Class's CL because the client's application affects the TCCL.
>
>
> On Thu, Mar 20, 2014 at 8:56 AM, haosdent <ha...@gmail.com> wrote:
>
> > Because we could change the value of "Thread.currentThread().
> > getContextClassLoader()" by
> "Thread.currentThread().setContextClassLoader(
> > classLoader)". But we couldn't change the value of
> > "this.getClass().getClassLoader()".
> > Maybe a little confused, TCCL is more flexible than the class's CL.
> >
> >
> > On Thu, Mar 20, 2014 at 2:43 PM, Amit Sela <am...@infolinks.com> wrote:
> >
> > > I understand that. But why ? Why prefer the TCCL over the class's CL ?
> > > On Mar 20, 2014 3:00 AM, "haosdent" <ha...@gmail.com> wrote:
> > >
> > > > Sometimes we would use "Thread.currentThread().setContextClassLoader(
> > > > classLoader);" to specify classloader. So
> > > > "Thread.currentThread().getContextClassLoader();"
> > > > should be the first choice.
> > > >
> > > >
> > > > On Thu, Mar 20, 2014 at 1:53 AM, Amit Sela <am...@infolinks.com>
> > wrote:
> > > >
> > > > > Hi all,
> > > > > I'm running with Hadoop 1.0.4 and HBase 0.94.12 bundled (OSGi)
> > > versions I
> > > > > built.
> > > > > Most issues I encountered are related to class loaders.
> > > > >
> > > > > One of the patterns I noticed in both projects is:
> > > > >
> > > > > ClassLoader cl =
> Thread.currentThread().getContextClassLoader();
> > > > > if(cl == null) {
> > > > > cl = Clazz.class.getClassLoader();
> > > > > }
> > > > >
> > > > > Where Clazz is the Class containing this code.
> > > > >
> > > > > I was wondering about this choice... Why not go the other way
> around:
> > > > >
> > > > > ClassLoader cl = Clazz.class.getClassLoader();
> > > > > if(cl == null) {
> > > > > cl = Thread.currentThread().getContextClassLoader();
> > > > > }
> > > > >
> > > > > And in a more general note, why not always use Configuration (and
> let
> > > > it's
> > > > > cl be this.getClass().getClassLoader()) to load classes ?
> > > > >
> > > > > That would surely help in integration with modularity frameworks.
> > > > >
> > > > > Thanks,
> > > > > Amit.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards,
> > > > Haosdent Huang
> > > >
> > >
> >
> >
> >
> > --
> > Best Regards,
> > Haosdent Huang
> >
>
--
Best Regards,
Haosdent Huang
Re: Class loading in Hadoop and HBase
Posted by Amit Sela <am...@infolinks.com>.
I understand the benefits of TCCL, but I think, and I would like to hear
the "why not", that code that may run in the client side (like in the case
of Configuration and HbaseObjectWritable examples I gave) should prefer the
Class's CL because the client's application affects the TCCL.
On Thu, Mar 20, 2014 at 8:56 AM, haosdent <ha...@gmail.com> wrote:
> Because we could change the value of "Thread.currentThread().
> getContextClassLoader()" by "Thread.currentThread().setContextClassLoader(
> classLoader)". But we couldn't change the value of
> "this.getClass().getClassLoader()".
> Maybe a little confused, TCCL is more flexible than the class's CL.
>
>
> On Thu, Mar 20, 2014 at 2:43 PM, Amit Sela <am...@infolinks.com> wrote:
>
> > I understand that. But why ? Why prefer the TCCL over the class's CL ?
> > On Mar 20, 2014 3:00 AM, "haosdent" <ha...@gmail.com> wrote:
> >
> > > Sometimes we would use "Thread.currentThread().setContextClassLoader(
> > > classLoader);" to specify classloader. So
> > > "Thread.currentThread().getContextClassLoader();"
> > > should be the first choice.
> > >
> > >
> > > On Thu, Mar 20, 2014 at 1:53 AM, Amit Sela <am...@infolinks.com>
> wrote:
> > >
> > > > Hi all,
> > > > I'm running with Hadoop 1.0.4 and HBase 0.94.12 bundled (OSGi)
> > versions I
> > > > built.
> > > > Most issues I encountered are related to class loaders.
> > > >
> > > > One of the patterns I noticed in both projects is:
> > > >
> > > > ClassLoader cl = Thread.currentThread().getContextClassLoader();
> > > > if(cl == null) {
> > > > cl = Clazz.class.getClassLoader();
> > > > }
> > > >
> > > > Where Clazz is the Class containing this code.
> > > >
> > > > I was wondering about this choice... Why not go the other way around:
> > > >
> > > > ClassLoader cl = Clazz.class.getClassLoader();
> > > > if(cl == null) {
> > > > cl = Thread.currentThread().getContextClassLoader();
> > > > }
> > > >
> > > > And in a more general note, why not always use Configuration (and let
> > > it's
> > > > cl be this.getClass().getClassLoader()) to load classes ?
> > > >
> > > > That would surely help in integration with modularity frameworks.
> > > >
> > > > Thanks,
> > > > Amit.
> > > >
> > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Haosdent Huang
> > >
> >
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Class loading in Hadoop and HBase
Posted by haosdent <ha...@gmail.com>.
Because we could change the value of "Thread.currentThread().
getContextClassLoader()" by "Thread.currentThread().setContextClassLoader(
classLoader)". But we couldn't change the value of
"this.getClass().getClassLoader()".
Maybe a little confused, TCCL is more flexible than the class's CL.
On Thu, Mar 20, 2014 at 2:43 PM, Amit Sela <am...@infolinks.com> wrote:
> I understand that. But why ? Why prefer the TCCL over the class's CL ?
> On Mar 20, 2014 3:00 AM, "haosdent" <ha...@gmail.com> wrote:
>
> > Sometimes we would use "Thread.currentThread().setContextClassLoader(
> > classLoader);" to specify classloader. So
> > "Thread.currentThread().getContextClassLoader();"
> > should be the first choice.
> >
> >
> > On Thu, Mar 20, 2014 at 1:53 AM, Amit Sela <am...@infolinks.com> wrote:
> >
> > > Hi all,
> > > I'm running with Hadoop 1.0.4 and HBase 0.94.12 bundled (OSGi)
> versions I
> > > built.
> > > Most issues I encountered are related to class loaders.
> > >
> > > One of the patterns I noticed in both projects is:
> > >
> > > ClassLoader cl = Thread.currentThread().getContextClassLoader();
> > > if(cl == null) {
> > > cl = Clazz.class.getClassLoader();
> > > }
> > >
> > > Where Clazz is the Class containing this code.
> > >
> > > I was wondering about this choice... Why not go the other way around:
> > >
> > > ClassLoader cl = Clazz.class.getClassLoader();
> > > if(cl == null) {
> > > cl = Thread.currentThread().getContextClassLoader();
> > > }
> > >
> > > And in a more general note, why not always use Configuration (and let
> > it's
> > > cl be this.getClass().getClassLoader()) to load classes ?
> > >
> > > That would surely help in integration with modularity frameworks.
> > >
> > > Thanks,
> > > Amit.
> > >
> >
> >
> >
> > --
> > Best Regards,
> > Haosdent Huang
> >
>
--
Best Regards,
Haosdent Huang
Re: Class loading in Hadoop and HBase
Posted by Amit Sela <am...@infolinks.com>.
I understand that. But why ? Why prefer the TCCL over the class's CL ?
On Mar 20, 2014 3:00 AM, "haosdent" <ha...@gmail.com> wrote:
> Sometimes we would use "Thread.currentThread().setContextClassLoader(
> classLoader);" to specify classloader. So
> "Thread.currentThread().getContextClassLoader();"
> should be the first choice.
>
>
> On Thu, Mar 20, 2014 at 1:53 AM, Amit Sela <am...@infolinks.com> wrote:
>
> > Hi all,
> > I'm running with Hadoop 1.0.4 and HBase 0.94.12 bundled (OSGi) versions I
> > built.
> > Most issues I encountered are related to class loaders.
> >
> > One of the patterns I noticed in both projects is:
> >
> > ClassLoader cl = Thread.currentThread().getContextClassLoader();
> > if(cl == null) {
> > cl = Clazz.class.getClassLoader();
> > }
> >
> > Where Clazz is the Class containing this code.
> >
> > I was wondering about this choice... Why not go the other way around:
> >
> > ClassLoader cl = Clazz.class.getClassLoader();
> > if(cl == null) {
> > cl = Thread.currentThread().getContextClassLoader();
> > }
> >
> > And in a more general note, why not always use Configuration (and let
> it's
> > cl be this.getClass().getClassLoader()) to load classes ?
> >
> > That would surely help in integration with modularity frameworks.
> >
> > Thanks,
> > Amit.
> >
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: Class loading in Hadoop and HBase
Posted by haosdent <ha...@gmail.com>.
Sometimes we would use "Thread.currentThread().setContextClassLoader(
classLoader);" to specify classloader. So
"Thread.currentThread().getContextClassLoader();"
should be the first choice.
On Thu, Mar 20, 2014 at 1:53 AM, Amit Sela <am...@infolinks.com> wrote:
> Hi all,
> I'm running with Hadoop 1.0.4 and HBase 0.94.12 bundled (OSGi) versions I
> built.
> Most issues I encountered are related to class loaders.
>
> One of the patterns I noticed in both projects is:
>
> ClassLoader cl = Thread.currentThread().getContextClassLoader();
> if(cl == null) {
> cl = Clazz.class.getClassLoader();
> }
>
> Where Clazz is the Class containing this code.
>
> I was wondering about this choice... Why not go the other way around:
>
> ClassLoader cl = Clazz.class.getClassLoader();
> if(cl == null) {
> cl = Thread.currentThread().getContextClassLoader();
> }
>
> And in a more general note, why not always use Configuration (and let it's
> cl be this.getClass().getClassLoader()) to load classes ?
>
> That would surely help in integration with modularity frameworks.
>
> Thanks,
> Amit.
>
--
Best Regards,
Haosdent Huang