hack fix to "RE: IsolationRunner job.jar file not being loaded into classloader?"

["Hiller, Dean (Contractor)" <de...@broadridge.com>]

Well, I hacked in my own fix and it works(I also put a
$HADOOP_HOME/deanTmpLib directory where I put the modified class files
of hadoop and modified hadoopscript to put that first on the classpath).
Here is my changes to IsolationRunner that fixed it(not sure if the is
the right way as it shouldn't be broken, should it?)...

 

  private static ClassLoader makeClassLoader(JobConf conf, 

                                             File workDir) throws
IOException {

    List<URL> cp = new ArrayList<URL>();

 

    String jar = conf.getJar();

    if (jar != null) {                      // if jar exists, it into
workDir

      File[] libs = new File(workDir, "lib").listFiles();

      if (libs != null) {

        for (int i = 0; i < libs.length; i++) {

          cp.add(new URL("file:" + libs[i].toString()));

        }

      }

      cp.add(new URL("file:" + new File(workDir,
"classes/").toString()));

      cp.add(new URL("file:" + workDir.toString() + "/"));

    }

    String url = "file:"+workDir.toString()+"/../../jars/";

    cp.add(new URL(url));

    return new URLClassLoader(cp.toArray(new URL[cp.size()]));

  }

 

 

From: Hiller, Dean (Contractor) 
Sent: Sunday, January 02, 2011 3:50 PM
To: 'mapreduce-user@hadoop.apache.org'
Subject: IsolationRunner job.jar file not being loaded into classloader?

 

IsolationRunner ends up calling the following code below, and the
"String jar" variable ends up correctly containing the path to job.jar,
and the working Dir(I am in eclipse remote debug mode) contains the
jobxxxx/attemptxxxx/work directory.  There is no lib dir in that work
dir so listFiles returns null below into the libs variable.

 

But as you can see from the below code, the job.jar is not added to the
URLClassLoader.  I then later get a ClassNotFoundException because
naturally my jar is not in the hadoop classpath and needs to be loaded
like a normal M/R job.  Is there a different location where the jar
should be loaded.  I can obviously cheat and stick my jar in the hadoop
classpath temporarily but that is really not the way to go here....(any
ideas?)  (I can also copy IsolationRunner.java and hack it but would
rather work with what is already there or supply a quick fix as it
doesn't look hard to fix).

 

  private static ClassLoader makeClassLoader(JobConf conf, 

                                             File workDir) throws
IOException {

    List<URL> cp = new ArrayList<URL>();

 

    String jar = conf.getJar();

    if (jar != null) {                      // if jar exists, it into
workDir

      File[] libs = new File(workDir, "lib").listFiles();

      if (libs != null) {

        for (int i = 0; i < libs.length; i++) {

          cp.add(new URL("file:" + libs[i].toString()));

        }

      }

      cp.add(new URL("file:" + new File(workDir,
"classes/").toString()));

      cp.add(new URL("file:" + workDir.toString() + "/"));

    }

    return new URLClassLoader(cp.toArray(new URL[cp.size()]));

  }

 

 


This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the 
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.