You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by "Segel, Mike" <ms...@navteq.com> on 2010/04/14 15:39:50 UTC
Custom Class Loader for Hadoop M/R jobs?
Hi,
Ok, here's a bit of a bizarre issue...
How do you handle class collisions between Hadoop and your m/r job which calls other 3rd party classes.
An example: Hadoop has an older version of an open source jar in its /lib directory. You're interfacing with a 3rd party OS tool that uses a later release of the same jar.
You can modify the classpath, and that might work. But the better way is to create a Custom Class Loader. (Non-trivial)
Looking at the Configuration class, it looks like there are a couple of methods that deal with loading a class in to the configuration so that the m/r jobs can have access to them on each node.
Is this the correct intended use, or am I missing something?
Has anyone done something like this?
Thx
-Mike
Michael Segel
Architect, R&D
NAVTEQ
425 West Randolph Street
Chicago, IL 60606
(T) +1 312-780-3432
(C) +1 312-952-8175
www.navteq.com<http://www.navteq.com/>
The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Custom Class Loader for Hadoop M/R jobs?
Posted by Tom White <to...@cloudera.com>.
I agree that it writing a custom classloader is non-trivial (and I
don't know if anyone has done this in the MapReduce context), so this
would best be handled by the framework. I've opened
https://issues.apache.org/jira/browse/MAPREDUCE-1700 for this.
Tom
On Wed, Apr 14, 2010 at 6:39 AM, Segel, Mike <ms...@navteq.com> wrote:
> Hi,
>
> Ok, here's a bit of a bizarre issue...
>
> How do you handle class collisions between Hadoop and your m/r job which calls other 3rd party classes.
>
> An example: Hadoop has an older version of an open source jar in its /lib directory. You're interfacing with a 3rd party OS tool that uses a later release of the same jar.
>
> You can modify the classpath, and that might work. But the better way is to create a Custom Class Loader. (Non-trivial)
>
> Looking at the Configuration class, it looks like there are a couple of methods that deal with loading a class in to the configuration so that the m/r jobs can have access to them on each node.
>
> Is this the correct intended use, or am I missing something?
> Has anyone done something like this?
>
> Thx
>
> -Mike
>
> Michael Segel
> Architect, R&D
> NAVTEQ
> 425 West Randolph Street
> Chicago, IL 60606
> (T) +1 312-780-3432
> (C) +1 312-952-8175
> www.navteq.com<http://www.navteq.com/>
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
>
Re: Custom Class Loader for Hadoop M/R jobs?
Posted by Caspar MacRae <ea...@gmail.com>.
ServiceMix and ActiveMQ are also using OSGi in their core AFAIK.
The way this (these) problems have been solved is more of an afterthought,
and regardless is quite simplistic (thinking of classloader isolation in web
containers, app servers, and the monolithic WARs produced as a result),
however OSGi's approach to modularity provides a truly elegant solution.
wrt to the ticket https://issues.apache.org/jira/browse/MAPREDUCE-243, Dave
Savage is a nice chap who works for Paremus (an enterprise OSGi provider) I
spoke with him at the recent OSGi devcon in London and he did quite a bit of
work to get this working for private clients - I'm sure with his expertise
it would not be a huge effort to get this rolling again.
On 15 April 2010 18:23, Segel, Mike <ms...@navteq.com> wrote:
> Ok, so are you using OSGi's code, or just writing your own code to
> implement a published API/specification?
>
> I realize this is getting off topic, so you may just want to e-mail me
> directly.
>
> Thx
>
> -Mike
>
> -----Original Message-----
> From: Marcel May [mailto:marcel.may@consol.de]
> Sent: Thursday, April 15, 2010 11:35 AM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>
>
> On Apr 15, 2010, at 6:27 PM, Segel, Mike wrote:
>
> > Caspar,
> >
> > While IANAL, I did a first blush read of OSGi's Specification License,
> Version 1.0. on the website link:
> http://www.osgi.org/Specifications/Licensing as well as the license on the
> download page: http://www.osgi.org/Download/Release4V42?info=nothanks.
> >
> > In a nutshell, OSGi's Specification License, Version 1.0. is incompatible
> with Apache's.
> >
> > That alone can explain why Apache projects won't touch OSGi.
> >
>
> http://felix.apache.org/
>
> > From a more practical issue, this problem is solved in other Apache
> projects, so why not borrow from another Apache project like Tomcat?
> >
> > Does that make sense? It appears OSGi is a non-starter.
> >
> > HTH
> >
> > -Mike
> ...
>
> Cheers,
> Marcel
>
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above. If you are not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited. If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>
RE: Custom Class Loader for Hadoop M/R jobs?
Posted by "Segel, Mike" <ms...@navteq.com>.
Ok, so are you using OSGi's code, or just writing your own code to implement a published API/specification?
I realize this is getting off topic, so you may just want to e-mail me directly.
Thx
-Mike
-----Original Message-----
From: Marcel May [mailto:marcel.may@consol.de]
Sent: Thursday, April 15, 2010 11:35 AM
To: general@hadoop.apache.org
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
On Apr 15, 2010, at 6:27 PM, Segel, Mike wrote:
> Caspar,
>
> While IANAL, I did a first blush read of OSGi's Specification License, Version 1.0. on the website link: http://www.osgi.org/Specifications/Licensing as well as the license on the download page: http://www.osgi.org/Download/Release4V42?info=nothanks.
>
> In a nutshell, OSGi's Specification License, Version 1.0. is incompatible with Apache's.
>
> That alone can explain why Apache projects won't touch OSGi.
>
http://felix.apache.org/
> From a more practical issue, this problem is solved in other Apache projects, so why not borrow from another Apache project like Tomcat?
>
> Does that make sense? It appears OSGi is a non-starter.
>
> HTH
>
> -Mike
...
Cheers,
Marcel
The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Custom Class Loader for Hadoop M/R jobs?
Posted by Marcel May <ma...@consol.de>.
On Apr 15, 2010, at 6:27 PM, Segel, Mike wrote:
> Caspar,
>
> While IANAL, I did a first blush read of OSGi's Specification License, Version 1.0. on the website link: http://www.osgi.org/Specifications/Licensing as well as the license on the download page: http://www.osgi.org/Download/Release4V42?info=nothanks.
>
> In a nutshell, OSGi's Specification License, Version 1.0. is incompatible with Apache's.
>
> That alone can explain why Apache projects won't touch OSGi.
>
http://felix.apache.org/
> From a more practical issue, this problem is solved in other Apache projects, so why not borrow from another Apache project like Tomcat?
>
> Does that make sense? It appears OSGi is a non-starter.
>
> HTH
>
> -Mike
...
Cheers,
Marcel
RE: Custom Class Loader for Hadoop M/R jobs?
Posted by "Segel, Mike" <ms...@navteq.com>.
Caspar,
While IANAL, I did a first blush read of OSGi's Specification License, Version 1.0. on the website link: http://www.osgi.org/Specifications/Licensing as well as the license on the download page: http://www.osgi.org/Download/Release4V42?info=nothanks.
In a nutshell, OSGi's Specification License, Version 1.0. is incompatible with Apache's.
That alone can explain why Apache projects won't touch OSGi.
>From a more practical issue, this problem is solved in other Apache projects, so why not borrow from another Apache project like Tomcat?
Does that make sense? It appears OSGi is a non-starter.
HTH
-Mike
-----Original Message-----
From: Caspar MacRae [mailto:earcam@gmail.com]
Sent: Thursday, April 15, 2010 10:17 AM
To: general@hadoop.apache.org
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
Hi all,
I think the problem(s) lies deeper and should be solved at more fundamental
level: OSGi (http://www.osgi.org)
This has the *classloader*, *modularity* and *distribution* management
maturity that IMHO Hadoop clearly needs (from what I know, albeit circa
1.9).
It's 10 years old, not headline app-server-tastic, nor flavour of the month
http://java.dzone.com/articles/osgi-feast-or-famine - but that's the point,
this proven tech. And it'll eventually be present at the lowest levels of
Java, eg. project jigsaw: http://openjdk.java.net/projects/jigsaw/ which is
a precursor to completing JSR-291
<http://markmail.org/message/5fjx7pzq6kmwagch> A number of people have tried
to introduce OSGi to Hadoop but it seems their efforts *may* have been
ignored by those in the meritocratic circle of power - this is a real shame,
perhaps Jira voting is the way to draw attention to this? You could vote on
this ticket that's over 2 years old...
https://issues.apache.org/jira/browse/MAPREDUCE-243
OSGi can easily solve the classloading and lifecycle management issues in
Hadoop, and brings a lot more besides. Can someone please explain to me the
rationale for continuing to ignore such an obvious and elegant solution?
Best regards,
Caspar
<http://techdistrict.kirkk.com/2010/02/26/osgi-devcon-slides/>
On 14 April 2010 19:19, Cooper, Chris <ch...@navteq.com> wrote:
> Scott,
>
> I think the direction your comments in
> https://issues.apache.org/jira/browse/MAPREDUCE-1700 is spot on. You
> should be looking at J2EE container class loader hierarchies. I've attached
> a couple of good links that cover this approach.
>
>
>
> http://www.ibm.com/developerworks/websphere/library/techarticles/0112_deboer/deboer.html
> http://www.objectsource.com/j2eechapters/Ch21-ClassLoaders_and_J2EE.htm
>
> I'm sure Mike and I would both be willing to work with you to contribute a
> solution if you're interested.
>
> Best regards,
>
> CC
>
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]
> Sent: Wednesday, April 14, 2010 1:08 PM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>
> My long term suggestions are in
> https://issues.apache.org/jira/browse/MAPREDUCE-1700. The framework
> definitely needs to handle this and not place the burden on users, IMO. But
> that won't help you in the short term.
>
> Whether removing or replacing a Hadoop jar is an acceptable option to you
> (or others) in the short term is up to you. Obviously, its not a great long
> term solution but if you (or someone else) has to make it work ASAP, it
> might be the only option. In our case, we package our own rpm and have a
> few custom patches to Hadoop so removing one jar is a trivial thing to do in
> the short / medium term.
>
> -Scott
>
> On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote:
>
> > Scott,
> >
> > While that may work for a quick fix. Its not a good long term solution
> and you then run in to a problem where you upgrade your hadoop release and
> the removed jar is replaced or if you replace the jar, it possible to get
> overwritten.
> >
> > In this specific instance, the Jackson libraries are not that important
> and they can be replaced.
> > But that doesn't mean that this issue won't come up again and its
> something you can't easily pop out and replace.
> >
> > This is why I'm looking at custom class loading and trying to understand
> what can be accomplished with the methods in the Configuration class.
> >
> > Thx
> >
> > -Mike
> >
> >
> > -----Original Message-----
> > From: Scott Carey [mailto:scott@richrelevance.com]
> > Sent: Wednesday, April 14, 2010 12:02 PM
> > To: general@hadoop.apache.org
> > Subject: Re: Custom Class Loader for Hadoop M/R jobs?
> >
> > Depending on what the dependency is, you might be able to just remove it
> from hadoop's lib directory on your cluster.
> >
> > For me, Hadoop's later versions has jackson-1.0.1 in its lib directory
> and that breaks usage of Avro in a M/R job among other things. However, the
> feature that uses this library is unimportant to me (configuration dump in
> JSON format) so I just removed the jar.
> >
> > -Scott
> >
> > On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
> >
> >> Hi,
> >>
> >> Ok, here's a bit of a bizarre issue...
> >>
> >> How do you handle class collisions between Hadoop and your m/r job which
> calls other 3rd party classes.
> >>
> >> An example: Hadoop has an older version of an open source jar in its
> /lib directory. You're interfacing with a 3rd party OS tool that uses a
> later release of the same jar.
> >>
> >> You can modify the classpath, and that might work. But the better way is
> to create a Custom Class Loader. (Non-trivial)
> >>
> >> Looking at the Configuration class, it looks like there are a couple of
> methods that deal with loading a class in to the configuration so that the
> m/r jobs can have access to them on each node.
> >>
> >> Is this the correct intended use, or am I missing something?
> >> Has anyone done something like this?
> >>
> >> Thx
> >>
> >> -Mike
> >>
> >> Michael Segel
> >> Architect, R&D
> >> NAVTEQ
> >> 425 West Randolph Street
> >> Chicago, IL 60606
> >> (T) +1 312-780-3432
> >> (C) +1 312-952-8175
> >> www.navteq.com<http://www.navteq.com/>
> >>
> >>
> >>
> >> The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above. If you are
> not the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited. If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
> >
> >
> >
> > The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above. If you are
> not the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited. If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above. If you are not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited. If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>
The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Custom Class Loader for Hadoop M/R jobs?
Posted by Caspar MacRae <ea...@gmail.com>.
Hi all,
I think the problem(s) lies deeper and should be solved at more fundamental
level: OSGi (http://www.osgi.org)
This has the *classloader*, *modularity* and *distribution* management
maturity that IMHO Hadoop clearly needs (from what I know, albeit circa
1.9).
It's 10 years old, not headline app-server-tastic, nor flavour of the month
http://java.dzone.com/articles/osgi-feast-or-famine - but that's the point,
this proven tech. And it'll eventually be present at the lowest levels of
Java, eg. project jigsaw: http://openjdk.java.net/projects/jigsaw/ which is
a precursor to completing JSR-291
<http://markmail.org/message/5fjx7pzq6kmwagch> A number of people have tried
to introduce OSGi to Hadoop but it seems their efforts *may* have been
ignored by those in the meritocratic circle of power - this is a real shame,
perhaps Jira voting is the way to draw attention to this? You could vote on
this ticket that's over 2 years old...
https://issues.apache.org/jira/browse/MAPREDUCE-243
OSGi can easily solve the classloading and lifecycle management issues in
Hadoop, and brings a lot more besides. Can someone please explain to me the
rationale for continuing to ignore such an obvious and elegant solution?
Best regards,
Caspar
<http://techdistrict.kirkk.com/2010/02/26/osgi-devcon-slides/>
On 14 April 2010 19:19, Cooper, Chris <ch...@navteq.com> wrote:
> Scott,
>
> I think the direction your comments in
> https://issues.apache.org/jira/browse/MAPREDUCE-1700 is spot on. You
> should be looking at J2EE container class loader hierarchies. I've attached
> a couple of good links that cover this approach.
>
>
>
> http://www.ibm.com/developerworks/websphere/library/techarticles/0112_deboer/deboer.html
> http://www.objectsource.com/j2eechapters/Ch21-ClassLoaders_and_J2EE.htm
>
> I'm sure Mike and I would both be willing to work with you to contribute a
> solution if you're interested.
>
> Best regards,
>
> CC
>
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]
> Sent: Wednesday, April 14, 2010 1:08 PM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>
> My long term suggestions are in
> https://issues.apache.org/jira/browse/MAPREDUCE-1700. The framework
> definitely needs to handle this and not place the burden on users, IMO. But
> that won't help you in the short term.
>
> Whether removing or replacing a Hadoop jar is an acceptable option to you
> (or others) in the short term is up to you. Obviously, its not a great long
> term solution but if you (or someone else) has to make it work ASAP, it
> might be the only option. In our case, we package our own rpm and have a
> few custom patches to Hadoop so removing one jar is a trivial thing to do in
> the short / medium term.
>
> -Scott
>
> On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote:
>
> > Scott,
> >
> > While that may work for a quick fix. Its not a good long term solution
> and you then run in to a problem where you upgrade your hadoop release and
> the removed jar is replaced or if you replace the jar, it possible to get
> overwritten.
> >
> > In this specific instance, the Jackson libraries are not that important
> and they can be replaced.
> > But that doesn't mean that this issue won't come up again and its
> something you can't easily pop out and replace.
> >
> > This is why I'm looking at custom class loading and trying to understand
> what can be accomplished with the methods in the Configuration class.
> >
> > Thx
> >
> > -Mike
> >
> >
> > -----Original Message-----
> > From: Scott Carey [mailto:scott@richrelevance.com]
> > Sent: Wednesday, April 14, 2010 12:02 PM
> > To: general@hadoop.apache.org
> > Subject: Re: Custom Class Loader for Hadoop M/R jobs?
> >
> > Depending on what the dependency is, you might be able to just remove it
> from hadoop's lib directory on your cluster.
> >
> > For me, Hadoop's later versions has jackson-1.0.1 in its lib directory
> and that breaks usage of Avro in a M/R job among other things. However, the
> feature that uses this library is unimportant to me (configuration dump in
> JSON format) so I just removed the jar.
> >
> > -Scott
> >
> > On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
> >
> >> Hi,
> >>
> >> Ok, here's a bit of a bizarre issue...
> >>
> >> How do you handle class collisions between Hadoop and your m/r job which
> calls other 3rd party classes.
> >>
> >> An example: Hadoop has an older version of an open source jar in its
> /lib directory. You're interfacing with a 3rd party OS tool that uses a
> later release of the same jar.
> >>
> >> You can modify the classpath, and that might work. But the better way is
> to create a Custom Class Loader. (Non-trivial)
> >>
> >> Looking at the Configuration class, it looks like there are a couple of
> methods that deal with loading a class in to the configuration so that the
> m/r jobs can have access to them on each node.
> >>
> >> Is this the correct intended use, or am I missing something?
> >> Has anyone done something like this?
> >>
> >> Thx
> >>
> >> -Mike
> >>
> >> Michael Segel
> >> Architect, R&D
> >> NAVTEQ
> >> 425 West Randolph Street
> >> Chicago, IL 60606
> >> (T) +1 312-780-3432
> >> (C) +1 312-952-8175
> >> www.navteq.com<http://www.navteq.com/>
> >>
> >>
> >>
> >> The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above. If you are
> not the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited. If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
> >
> >
> >
> > The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above. If you are
> not the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited. If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above. If you are not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited. If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>
RE: Custom Class Loader for Hadoop M/R jobs?
Posted by "Cooper, Chris" <ch...@navteq.com>.
Scott,
I think the direction your comments in https://issues.apache.org/jira/browse/MAPREDUCE-1700 is spot on. You should be looking at J2EE container class loader hierarchies. I've attached a couple of good links that cover this approach.
http://www.ibm.com/developerworks/websphere/library/techarticles/0112_deboer/deboer.html
http://www.objectsource.com/j2eechapters/Ch21-ClassLoaders_and_J2EE.htm
I'm sure Mike and I would both be willing to work with you to contribute a solution if you're interested.
Best regards,
CC
-----Original Message-----
From: Scott Carey [mailto:scott@richrelevance.com]
Sent: Wednesday, April 14, 2010 1:08 PM
To: general@hadoop.apache.org
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
My long term suggestions are in https://issues.apache.org/jira/browse/MAPREDUCE-1700. The framework definitely needs to handle this and not place the burden on users, IMO. But that won't help you in the short term.
Whether removing or replacing a Hadoop jar is an acceptable option to you (or others) in the short term is up to you. Obviously, its not a great long term solution but if you (or someone else) has to make it work ASAP, it might be the only option. In our case, we package our own rpm and have a few custom patches to Hadoop so removing one jar is a trivial thing to do in the short / medium term.
-Scott
On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote:
> Scott,
>
> While that may work for a quick fix. Its not a good long term solution and you then run in to a problem where you upgrade your hadoop release and the removed jar is replaced or if you replace the jar, it possible to get overwritten.
>
> In this specific instance, the Jackson libraries are not that important and they can be replaced.
> But that doesn't mean that this issue won't come up again and its something you can't easily pop out and replace.
>
> This is why I'm looking at custom class loading and trying to understand what can be accomplished with the methods in the Configuration class.
>
> Thx
>
> -Mike
>
>
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]
> Sent: Wednesday, April 14, 2010 12:02 PM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>
> Depending on what the dependency is, you might be able to just remove it from hadoop's lib directory on your cluster.
>
> For me, Hadoop's later versions has jackson-1.0.1 in its lib directory and that breaks usage of Avro in a M/R job among other things. However, the feature that uses this library is unimportant to me (configuration dump in JSON format) so I just removed the jar.
>
> -Scott
>
> On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
>
>> Hi,
>>
>> Ok, here's a bit of a bizarre issue...
>>
>> How do you handle class collisions between Hadoop and your m/r job which calls other 3rd party classes.
>>
>> An example: Hadoop has an older version of an open source jar in its /lib directory. You're interfacing with a 3rd party OS tool that uses a later release of the same jar.
>>
>> You can modify the classpath, and that might work. But the better way is to create a Custom Class Loader. (Non-trivial)
>>
>> Looking at the Configuration class, it looks like there are a couple of methods that deal with loading a class in to the configuration so that the m/r jobs can have access to them on each node.
>>
>> Is this the correct intended use, or am I missing something?
>> Has anyone done something like this?
>>
>> Thx
>>
>> -Mike
>>
>> Michael Segel
>> Architect, R&D
>> NAVTEQ
>> 425 West Randolph Street
>> Chicago, IL 60606
>> (T) +1 312-780-3432
>> (C) +1 312-952-8175
>> www.navteq.com<http://www.navteq.com/>
>>
>>
>>
>> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Custom Class Loader for Hadoop M/R jobs?
Posted by Steve Loughran <st...@apache.org>.
Scott Carey wrote:
> In our case, we package our own rpm and have a few custom patches to Hadoop so removing one jar is a trivial thing to do in the short / medium term.
Added my thoughts on CLs to the MAPREDUCE-1700 bugrep.
I do my own RPMs too, Ivy driven, tested by installing onto local VMs.
There is scope for having a hadoop-redist project that contains
everything needed to create RPMs or debs of Hadoop releases, with your
own JARs in. volunteers to do this are welcome...
Re: Custom Class Loader for Hadoop M/R jobs?
Posted by Scott Carey <sc...@richrelevance.com>.
My long term suggestions are in https://issues.apache.org/jira/browse/MAPREDUCE-1700. The framework definitely needs to handle this and not place the burden on users, IMO. But that won't help you in the short term.
Whether removing or replacing a Hadoop jar is an acceptable option to you (or others) in the short term is up to you. Obviously, its not a great long term solution but if you (or someone else) has to make it work ASAP, it might be the only option. In our case, we package our own rpm and have a few custom patches to Hadoop so removing one jar is a trivial thing to do in the short / medium term.
-Scott
On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote:
> Scott,
>
> While that may work for a quick fix. Its not a good long term solution and you then run in to a problem where you upgrade your hadoop release and the removed jar is replaced or if you replace the jar, it possible to get overwritten.
>
> In this specific instance, the Jackson libraries are not that important and they can be replaced.
> But that doesn't mean that this issue won't come up again and its something you can't easily pop out and replace.
>
> This is why I'm looking at custom class loading and trying to understand what can be accomplished with the methods in the Configuration class.
>
> Thx
>
> -Mike
>
>
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]
> Sent: Wednesday, April 14, 2010 12:02 PM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>
> Depending on what the dependency is, you might be able to just remove it from hadoop's lib directory on your cluster.
>
> For me, Hadoop's later versions has jackson-1.0.1 in its lib directory and that breaks usage of Avro in a M/R job among other things. However, the feature that uses this library is unimportant to me (configuration dump in JSON format) so I just removed the jar.
>
> -Scott
>
> On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
>
>> Hi,
>>
>> Ok, here's a bit of a bizarre issue...
>>
>> How do you handle class collisions between Hadoop and your m/r job which calls other 3rd party classes.
>>
>> An example: Hadoop has an older version of an open source jar in its /lib directory. You're interfacing with a 3rd party OS tool that uses a later release of the same jar.
>>
>> You can modify the classpath, and that might work. But the better way is to create a Custom Class Loader. (Non-trivial)
>>
>> Looking at the Configuration class, it looks like there are a couple of methods that deal with loading a class in to the configuration so that the m/r jobs can have access to them on each node.
>>
>> Is this the correct intended use, or am I missing something?
>> Has anyone done something like this?
>>
>> Thx
>>
>> -Mike
>>
>> Michael Segel
>> Architect, R&D
>> NAVTEQ
>> 425 West Randolph Street
>> Chicago, IL 60606
>> (T) +1 312-780-3432
>> (C) +1 312-952-8175
>> www.navteq.com<http://www.navteq.com/>
>>
>>
>>
>> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Custom Class Loader for Hadoop M/R jobs?
Posted by Julia Smith <in...@mac.com>.
This might be of use to you, as I had a related problem years ago:
http://www.developertutorials.com/tutorials/java/simplify-application-delivery-one-jar-050422/page5.html
The trick is a one-jar solution allows you to embed jar files and load classes from them. Rolling up all your jar variants in a single jar and then selectively loading the desired version might provide you a useful solution. The source is there, so you could start with a working classloader and perhaps apply a few tweaks.
On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote:
> Scott,
>
> While that may work for a quick fix. Its not a good long term solution and you then run in to a problem where you upgrade your hadoop release and the removed jar is replaced or if you replace the jar, it possible to get overwritten.
>
> In this specific instance, the Jackson libraries are not that important and they can be replaced.
> But that doesn't mean that this issue won't come up again and its something you can't easily pop out and replace.
>
> This is why I'm looking at custom class loading and trying to understand what can be accomplished with the methods in the Configuration class.
>
> Thx
>
> -Mike
>
>
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]
> Sent: Wednesday, April 14, 2010 12:02 PM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>
> Depending on what the dependency is, you might be able to just remove it from hadoop's lib directory on your cluster.
>
> For me, Hadoop's later versions has jackson-1.0.1 in its lib directory and that breaks usage of Avro in a M/R job among other things. However, the feature that uses this library is unimportant to me (configuration dump in JSON format) so I just removed the jar.
>
> -Scott
>
> On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
>
>> Hi,
>>
>> Ok, here's a bit of a bizarre issue...
>>
>> How do you handle class collisions between Hadoop and your m/r job which calls other 3rd party classes.
>>
>> An example: Hadoop has an older version of an open source jar in its /lib directory. You're interfacing with a 3rd party OS tool that uses a later release of the same jar.
>>
>> You can modify the classpath, and that might work. But the better way is to create a Custom Class Loader. (Non-trivial)
>>
>> Looking at the Configuration class, it looks like there are a couple of methods that deal with loading a class in to the configuration so that the m/r jobs can have access to them on each node.
>>
>> Is this the correct intended use, or am I missing something?
>> Has anyone done something like this?
>>
>> Thx
>>
>> -Mike
>>
>> Michael Segel
>> Architect, R&D
>> NAVTEQ
>> 425 West Randolph Street
>> Chicago, IL 60606
>> (T) +1 312-780-3432
>> (C) +1 312-952-8175
>> www.navteq.com<http://www.navteq.com/>
>>
>>
>>
>> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
RE: Custom Class Loader for Hadoop M/R jobs?
Posted by "Segel, Mike" <ms...@navteq.com>.
Scott,
While that may work for a quick fix. Its not a good long term solution and you then run in to a problem where you upgrade your hadoop release and the removed jar is replaced or if you replace the jar, it possible to get overwritten.
In this specific instance, the Jackson libraries are not that important and they can be replaced.
But that doesn't mean that this issue won't come up again and its something you can't easily pop out and replace.
This is why I'm looking at custom class loading and trying to understand what can be accomplished with the methods in the Configuration class.
Thx
-Mike
-----Original Message-----
From: Scott Carey [mailto:scott@richrelevance.com]
Sent: Wednesday, April 14, 2010 12:02 PM
To: general@hadoop.apache.org
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
Depending on what the dependency is, you might be able to just remove it from hadoop's lib directory on your cluster.
For me, Hadoop's later versions has jackson-1.0.1 in its lib directory and that breaks usage of Avro in a M/R job among other things. However, the feature that uses this library is unimportant to me (configuration dump in JSON format) so I just removed the jar.
-Scott
On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
> Hi,
>
> Ok, here's a bit of a bizarre issue...
>
> How do you handle class collisions between Hadoop and your m/r job which calls other 3rd party classes.
>
> An example: Hadoop has an older version of an open source jar in its /lib directory. You're interfacing with a 3rd party OS tool that uses a later release of the same jar.
>
> You can modify the classpath, and that might work. But the better way is to create a Custom Class Loader. (Non-trivial)
>
> Looking at the Configuration class, it looks like there are a couple of methods that deal with loading a class in to the configuration so that the m/r jobs can have access to them on each node.
>
> Is this the correct intended use, or am I missing something?
> Has anyone done something like this?
>
> Thx
>
> -Mike
>
> Michael Segel
> Architect, R&D
> NAVTEQ
> 425 West Randolph Street
> Chicago, IL 60606
> (T) +1 312-780-3432
> (C) +1 312-952-8175
> www.navteq.com<http://www.navteq.com/>
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Custom Class Loader for Hadoop M/R jobs?
Posted by Scott Carey <sc...@richrelevance.com>.
Depending on what the dependency is, you might be able to just remove it from hadoop's lib directory on your cluster.
For me, Hadoop's later versions has jackson-1.0.1 in its lib directory and that breaks usage of Avro in a M/R job among other things. However, the feature that uses this library is unimportant to me (configuration dump in JSON format) so I just removed the jar.
-Scott
On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
> Hi,
>
> Ok, here's a bit of a bizarre issue...
>
> How do you handle class collisions between Hadoop and your m/r job which calls other 3rd party classes.
>
> An example: Hadoop has an older version of an open source jar in its /lib directory. You're interfacing with a 3rd party OS tool that uses a later release of the same jar.
>
> You can modify the classpath, and that might work. But the better way is to create a Custom Class Loader. (Non-trivial)
>
> Looking at the Configuration class, it looks like there are a couple of methods that deal with loading a class in to the configuration so that the m/r jobs can have access to them on each node.
>
> Is this the correct intended use, or am I missing something?
> Has anyone done something like this?
>
> Thx
>
> -Mike
>
> Michael Segel
> Architect, R&D
> NAVTEQ
> 425 West Randolph Street
> Chicago, IL 60606
> (T) +1 312-780-3432
> (C) +1 312-952-8175
> www.navteq.com<http://www.navteq.com/>
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.