[jira] [Updated] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases

["Benjamin Coverston (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Benjamin Coverston updated CASSANDRA-5201:
------------------------------------------

    Attachment: hadoopCompat.patch

Poking around at other projects this generally gets solved in one of two ways: Ship two versions of their Hadoop integrations (one compiled for the old, and one compiled for the new), or use a little reflection to make things work across the board.

I'm attaching a patch that uses the hadoopCompat subproject of elephantbird. This will allow us to compile a single binary and run with the new and old context objects.

I've tested this patch with HDP 2.0, and Apache Hadoop 1.0.4 and it works fine with both (including Hive in DSE). With Pig I needed to compile our (optional) pig dependency with:

bq. ant clean jar-withouthadoop -Dhadoopversion=23

Only really needed if you're using one of the current versions of thrift with the new JobContext.


> Cassandra/Hadoop does not support current Hadoop releases
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-5201
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5201
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Hadoop
>    Affects Versions: 1.2.0
>            Reporter: Brian Jeltema
>            Assignee: Dave Brosius
>         Attachments: 5201_a.txt, hadoopCompat.patch
>
>
> Using Hadoop 0.22.0 with Cassandra results in the stack trace below.
> It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext
> from a class to an interface.
> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
> 	at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357)
> 	at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045)
> 	at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
> 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042)
> 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062)
> 	at MyHadoopApp.run(MyHadoopApp.java:163)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> 	at MyHadoopApp.main(MyHadoopApp.java:82)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:192)



--
This message was sent by Atlassian JIRA
(v6.1#6144)