[jira] [Commented] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions

["Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197514#comment-13197514 ] 

Alejandro Abdelnur commented on MAPREDUCE-3697:
-----------------------------------------------

While (in my case Oozie) code compiles and works fine with Hadoop 1 or Hadoop 0.23, the problem shows up when you compile Oozie using a version of Hadoop (i.e. 1.0.0) and then run Oozie with another version of Hadoop (ie 0.23/trunk).

*Compiled with Hadoop 1.0.0, run with Hadoop 0.24.0-SNAPSHOT:*

{code}
	at org.apache.oozie.action.hadoop.TestMapReduceActionExecutor.testStreaming(TestMapReduceActionExecutor.java:404)

testSetExecutionStats_when_user_has_specified_stats_write_TRUE(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)  Time elapsed: 31.487 sec  <<< ERROR!
java.lang.NoSuchMethodError: org.apache.hadoop.mapred.Counters.getGroup(Ljava/lang/String;)Lorg/apache/hadoop/mapred/Counters$Group;
	at org.apache.oozie.action.hadoop.LauncherMapper.hasIdSwap(LauncherMapper.java:273)
	at org.apache.oozie.action.hadoop.TestMapReduceActionExecutor.testSetExecutionStats_when_user_has_specified_stats_write_TRUE(TestMapReduceActionExecutor.java:488)
{code}

As Tom' mentioned earlier this is because Counters changed from abstract class to interface. Seems that the generated bytecode in Oozieland differs if Counters is an abstract class or an interface.

Because of this, I would this qualifies as an incompatible change in the Hadoop API.

IMO we should revert to the old Counters API i we want to maintain backwards compatibility here.

Else, downstream projects will have to either use shims or reflection if they want to be binary compatible.

                
> Hadoop Counters API limits Oozie's working across different hadoop versions
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3697
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.1
>            Reporter: John George
>            Assignee: Mahadev konar
>            Priority: Blocker
>
> Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in
> hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). Its
> parent class AbstractCounters implements it. This is different from hadoop20X.
> As a result, Oozie compiled with either hadoop version does not work with the
> other version.
> A specific scenario, Oozie compiled with .23 and run against 205, does not
> update job status owing to a Counters API exception.
> Will explicit re-compilation against the relevant hadoop jars be required each
> time? This will prevent launching a uniform Oozie version across different
> clusters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira