- Oozie 4.0.1 on AWS EMR ...

[Naga Vijay <na...@gmail.com>]

Hi,

I have Oozie 4.0.1 working fine for Hive action on AWS EC2 instance by
using Apache Hadoop distribution.

I am facing issues with Oozie 4.0.1 on AWS EMR with Amazon Hadoop
distribution.

The nature of issue is - there's an out-dated libthrift-0.7.0.jar in amazon
hadoop lib dir.

On the Task Tracker to which I have no access, that jar is ahead of hive's
libthrift-0.9.0.jar in the classpath, which I think is dynamically
generated.

How can I work around this?  That is how can have the hive job execution on
Task Tracker use hive's libthrift-0.9.0.jar ajhead of hadoop's
libthrift-0.7.0.jar ?

I cannot replace amazon hadoop libthrift-0.7.0.jar, as that leads to losing
the cluster upon reboot (tried).

Placing libthrift-0.9.0.jar renamed as libthrift-0.6.0.jar in amazon hadoop
lib dir doesn't help, as the hive job execution is on task tracker node to
which I have no accesss.

I have tried placing libthrift-0.9.0.jar in sharelib, and also tried
placing libthrift-0.9.0.jar renamed as libthrift-0.6.0.jar in sharelib.
None of this helps.

Here's the stack trace ...

-----

Failing Oozie Launcher, Main class
[org.apache.oozie.action.hadoop.HiveMain], main() threw exception,
org.apache.thrift.EncodingUtil
s.setBit(BIZ)B
java.lang.NoSuchMethodError: org.apache.thrift.EncodingUtils.setBit(BIZ)B
        at
org.apache.hadoop.hive.ql.plan.api.Query.setStartedIsSet(Query.java:487)
        at
org.apache.hadoop.hive.ql.plan.api.Query.setStarted(Query.java:474)
        at
org.apache.hadoop.hive.ql.QueryPlan.updateCountersInQueryPlan(QueryPlan.java:309)
        at
org.apache.hadoop.hive.ql.QueryPlan.getQueryPlan(QueryPlan.java:450)
        at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:622)
        at
org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:504)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1073)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:945)

-----

I tried with this as well in hive script ...

set mapreduce.task.classpath.user.precedence=true;

It didn't help.

Please advice on how to go about.

Thanks
Naga