[jira] [Commented] (HADOOP-12406) AbstractMapWritable.readFields throws ClassNotFoundException with custom writables

["Nadeem Douba (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-12406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14986295#comment-14986295 ] 

Nadeem Douba commented on HADOOP-12406:
---------------------------------------

Hi Vinod,

I was running this on top of YARN with the default configuration enabled on top of Java 1.8 from Oracle. I think the new isolation strategy used in the newer versions of the JVM for class loaders prevents the system class loader from loading a job jar class. I didn't do anything fancy with configuration options or anything else. The only thing I added were the remote debugging options to allow me to attach to the map and reduce threads remotely and debug them step by step. I can tell you that the job jar definitely isn't in the system's class loader but in the current thread's context class loader. I verified this observation by manually inspecting the class loader object from the system and the one that comes with the thread context.

Cheers,

Nadeem

> AbstractMapWritable.readFields throws ClassNotFoundException with custom writables
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-12406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12406
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 2.7.1
>         Environment: Ubuntu Linux 14.04 LTS amd64
>            Reporter: Nadeem Douba
>            Priority: Blocker
>              Labels: bug, hadoop, io, newbie, patch-available
>         Attachments: HADOOP-12406.patch
>
>
> Note: I am not an expert at JAVA, Class loaders, or Hadoop. I am just a hacker. My solution might be entirely wrong.
> AbstractMapWritable.readFields throws a ClassNotFoundException when reading custom writables. Debugging the job using remote debugging in IntelliJ revealed that the class loader being used in Class.forName() is different than that used by the Thread's current context (Thread.currentThread().getContextClassLoader()). The class path for the system class loader does not include the libraries of the job jar. However, the class path for the context class loader does. The proposed patch changes the class loading mechanism in readFields to use the Thread's context class loader instead of the system's default class loader.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)