You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Joan <jo...@gmail.com> on 2011/01/13 13:18:56 UTC

Solr + Hadoop

Hi,

I'm trying build solr index with MapReduce (Hadoop) and I'm using
https://issues.apache.org/jira/browse/SOLR-1301 but I've a problem with
hadoop version and this patch.

When I compile this patch, I use 0.21.0 hadoop version, I don't have any
problem but when I'm trying to run my job in Hadoop (0.0.21) I get some
error like this:

*Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
interface org.apache.hadoop.mapreduce.JobContext, but class was expected*
 at
org.apache.solr.hadoop.SolrOutputFormat.checkOutputSpecs(SolrOutputFormat.java:147)
        at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:373)
        at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:334)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:960)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:976)



I try to override the next method:

  *public void checkOutputSpecs(JobContext job) throws IOException {
    super.checkOutputSpecs(job);
    if (job.getConfiguration().get(SETUP_OK) == null) {
      throw new IOException("Solr home cache not set up!");
    }
  }*

by

 * public void checkOutputSpecs(Job job) throws IOException {
    super.checkOutputSpecs(job);
    if (job.getConfiguration().get(SETUP_OK) == null) {
      throw new IOException("Solr home cache not set up!");
    }
  }*

but I continue receive some error:
*
        java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.solr.hadoop.SolrOutputFormat
        at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1128)
        at
org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:203)
        at org.apache.hadoop.mapred.Task.initialize(Task.java:487)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
        at java.security.AccessController.doPrivileged(Native Method)*



Please, Is someone using this patch with 0.21.0 Version Hadoop?. Can someone
help me?

Thanks,

Joan

Re: Solr + Hadoop

Posted by Harsh J <qw...@gmail.com>.
The second issue is cause your Mapper code is unable to find the solr
and other user classes (It does not have a JAR on its classpath to
load one from). You can avoid this by making sure that (either):

1. Your submission code is doing a Job.setJarByClass(...) or
Job.setJar(...) --> This will send the JAR via the DistributedCache to
all the TaskTrackers that will run the Mappers. You can also manually
add required files to DistributedCache via it's API.

2. Add your Solr + other library jars to Hadoop's lib/
directory/classpath across all machines and restart the cluster.

The latter is the way to go if you will always need some classes in your setup.

On Thu, Jan 13, 2011 at 5:48 PM, Joan <jo...@gmail.com> wrote:
> Hi,
>
> I'm trying build solr index with MapReduce (Hadoop) and I'm using
> https://issues.apache.org/jira/browse/SOLR-1301 but I've a problem with
> hadoop version and this patch.
>
> When I compile this patch, I use 0.21.0 hadoop version, I don't have any
> problem but when I'm trying to run my job in Hadoop (0.0.21) I get some
> error like this:
>
> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
>  at
> org.apache.solr.hadoop.SolrOutputFormat.checkOutputSpecs(SolrOutputFormat.java:147)
>         at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:373)
>         at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:334)
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:960)
>         at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:976)
>
>
>
> I try to override the next method:
>
>   public void checkOutputSpecs(JobContext job) throws IOException {
>     super.checkOutputSpecs(job);
>     if (job.getConfiguration().get(SETUP_OK) == null) {
>       throw new IOException("Solr home cache not set up!");
>     }
>   }
>
> by
>
>   public void checkOutputSpecs(Job job) throws IOException {
>     super.checkOutputSpecs(job);
>     if (job.getConfiguration().get(SETUP_OK) == null) {
>       throw new IOException("Solr home cache not set up!");
>     }
>   }
>
> but I continue receive some error:
>
>         java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.solr.hadoop.SolrOutputFormat
>         at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1128)
>         at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:203)
>         at org.apache.hadoop.mapred.Task.initialize(Task.java:487)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>         at java.security.AccessController.doPrivileged(Native Method)
>
>
>
> Please, Is someone using this patch with 0.21.0 Version Hadoop?. Can someone
> help me?
>
> Thanks,
>
> Joan
>



-- 
Harsh J
www.harshj.com