You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Alex Baranau <al...@gmail.com> on 2013/03/21 01:44:32 UTC

Bug in LocalJobRunner?

Hi!

In a test I tried to run a job which is in separate jar with
MiniMRYarnCluster and got ClassNotFoundException for the Mapper class that
is in that jar.

I went a bit deeper and found possible bug in LocalJobRunner:

currently:

      this.job = new JobConf(localJobFile);

      // Job (the current object) is a Thread, so we wrap its class loader.
      if (localDistributedCacheManager.hasLocalClasspaths()) {
        setContextClassLoader(localDistributedCacheManager.makeClassLoader(
                getContextClassLoader()));
      }

should be:

      this.job = new JobConf(localJobFile);
      // Job (the current object) is a Thread, so we wrap its class loader.
      if (localDistributedCacheManager.hasLocalClasspaths()) {
        ClassLoader classLoader =
localDistributedCacheManager.makeClassLoader(getContextClassLoader());
        setContextClassLoader(classLoader);
        this.job.setClassLoader(classLoader);
      }

I.e. we need to set classloader for job configuration so that it can load
classes from the jar.

If the above makes sense I will file JIRA with patch, otherwise, what am I
missing?


Thank you,
Alex Baranau

Re: Bug in LocalJobRunner?

Posted by Alex Baranau <al...@gmail.com>.
Hi Harsh J,

Thanx for taking a look. I created
https://issues.apache.org/jira/browse/MAPREDUCE-5097 and attached patch. I
also provided (ugly, sorry) example of how to get the error.

Alex Baranau

On Thu, Mar 21, 2013 at 5:58 AM, Harsh J <ha...@cloudera.com> wrote:

> Hi Alex,
>
> This seems to make sense. Could you post your patch with a possible
> (even if detached) test case?
>
> On Thu, Mar 21, 2013 at 6:14 AM, Alex Baranau <al...@gmail.com>
> wrote:
> > Hi!
> >
> > In a test I tried to run a job which is in separate jar with
> > MiniMRYarnCluster and got ClassNotFoundException for the Mapper class
> that
> > is in that jar.
> >
> > I went a bit deeper and found possible bug in LocalJobRunner:
> >
> > currently:
> >
> >       this.job = new JobConf(localJobFile);
> >
> >       // Job (the current object) is a Thread, so we wrap its class
> loader.
> >       if (localDistributedCacheManager.hasLocalClasspaths()) {
> >
> setContextClassLoader(localDistributedCacheManager.makeClassLoader(
> >                 getContextClassLoader()));
> >       }
> >
> > should be:
> >
> >       this.job = new JobConf(localJobFile);
> >       // Job (the current object) is a Thread, so we wrap its class
> loader.
> >       if (localDistributedCacheManager.hasLocalClasspaths()) {
> >         ClassLoader classLoader =
> > localDistributedCacheManager.makeClassLoader(getContextClassLoader());
> >         setContextClassLoader(classLoader);
> >         this.job.setClassLoader(classLoader);
> >       }
> >
> > I.e. we need to set classloader for job configuration so that it can load
> > classes from the jar.
> >
> > If the above makes sense I will file JIRA with patch, otherwise, what am
> I
> > missing?
> >
> >
> > Thank you,
> > Alex Baranau
>
>
>
> --
> Harsh J
>

Re: Bug in LocalJobRunner?

Posted by Harsh J <ha...@cloudera.com>.
Hi Alex,

This seems to make sense. Could you post your patch with a possible
(even if detached) test case?

On Thu, Mar 21, 2013 at 6:14 AM, Alex Baranau <al...@gmail.com> wrote:
> Hi!
>
> In a test I tried to run a job which is in separate jar with
> MiniMRYarnCluster and got ClassNotFoundException for the Mapper class that
> is in that jar.
>
> I went a bit deeper and found possible bug in LocalJobRunner:
>
> currently:
>
>       this.job = new JobConf(localJobFile);
>
>       // Job (the current object) is a Thread, so we wrap its class loader.
>       if (localDistributedCacheManager.hasLocalClasspaths()) {
>         setContextClassLoader(localDistributedCacheManager.makeClassLoader(
>                 getContextClassLoader()));
>       }
>
> should be:
>
>       this.job = new JobConf(localJobFile);
>       // Job (the current object) is a Thread, so we wrap its class loader.
>       if (localDistributedCacheManager.hasLocalClasspaths()) {
>         ClassLoader classLoader =
> localDistributedCacheManager.makeClassLoader(getContextClassLoader());
>         setContextClassLoader(classLoader);
>         this.job.setClassLoader(classLoader);
>       }
>
> I.e. we need to set classloader for job configuration so that it can load
> classes from the jar.
>
> If the above makes sense I will file JIRA with patch, otherwise, what am I
> missing?
>
>
> Thank you,
> Alex Baranau



-- 
Harsh J