You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org> on 2016/04/10 22:37:25 UTC

[jira] [Updated] (HADOOP-12406) AbstractMapWritable.readFields throws ClassNotFoundException with custom writables

     [ https://issues.apache.org/jira/browse/HADOOP-12406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod Kumar Vavilapalli updated HADOOP-12406:
---------------------------------------------
    Assignee: Nadeem Douba
      Status: Open  (was: Patch Available)

Hi [~ndouba],

I'm about to do a 2.7.3 Apache Hadoop release and finally got around to this again.

h4. Analysis
To make progress, I had to read up a bit on nutch and about how to run this so that I can reproduce the bug in order to rationalize your patch. I finally succeeded in doing so! Tested this with 2.7.2 release and nutch 1.11 and using the URL feed [given at NUTCH-1084|https://issues.apache.org/jira/browse/NUTCH-1084?focusedCommentId=13882771&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13882771]
{code}
~/tmp/common/hadoop-common-2.7.2/bin/hadoop jar apache-nutch-1.11.job org.apache.nutch.crawl.CrawlDbReader file:///tmp/nutch/apache-nutch-1.11/runtime/local/crawl/crawldb/ -url http://bappenas.go.id/
{code}

I can reproduce all the problems listed at NUTCH-1084 - with readdb, MR local-job-runner based job for crawling etc.

The real issue is that Nutch's readdb is client-only and *not* running a MapReduce job which was my question before. For regular MR jobs, the job-jar *is* on the system class-loader. For the client-only invocations using "hadoop jar" and local-job-runner, the job-jar is actually *not* on the system-classpath - that is why you are running into the issue.

h4. Summary
Your patch looks good to me. Clearly, the thread context-loader falls back to system class-loader where it is not overridden - so we are fine for all the ways of loading the classes in readFields.

I'll resubmit your patch with minor commenting related changes to Jenkins and commit if Mr.Jenkins is also fine.

> AbstractMapWritable.readFields throws ClassNotFoundException with custom writables
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-12406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12406
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 2.7.1
>         Environment: Ubuntu Linux 14.04 LTS amd64
>            Reporter: Nadeem Douba
>            Assignee: Nadeem Douba
>            Priority: Blocker
>              Labels: bug, hadoop, io, newbie, patch-available
>         Attachments: HADOOP-12406.patch
>
>
> Note: I am not an expert at JAVA, Class loaders, or Hadoop. I am just a hacker. My solution might be entirely wrong.
> AbstractMapWritable.readFields throws a ClassNotFoundException when reading custom writables. Debugging the job using remote debugging in IntelliJ revealed that the class loader being used in Class.forName() is different than that used by the Thread's current context (Thread.currentThread().getContextClassLoader()). The class path for the system class loader does not include the libraries of the job jar. However, the class path for the context class loader does. The proposed patch changes the class loading mechanism in readFields to use the Thread's context class loader instead of the system's default class loader.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)