RE: Hive action in Oozie failing intermittently

["Raja.Aravapalli" <Ra...@target.com>]

Hi Sathi,

Thanks a lot for your reply.

We have a java action(which is our MR application) where the respective jar is built with jar with dependencies, and kept in the "lib" directory of deploy folder. And ofcourse, we had some hive related jars in that!

But,  if you ask me to remove that jar with dependencies jar file  and execute our hive script, my java action in workflow fails....!!!! then how to handle this scenario... again !!!


Also, FYI, we recently upgraded our cluster from 4.x to 5.x, when we are with 4.x the workflow worked absolutely fine and we did not face any issues (does't the problem you stated with jar with dependencies doesn't create issue with CDH 4.x and it only creates issue with CDH 5.x ??? ) ...!! 

Only when we upgraded to 5.x and configured cluster with YARN we are facing hive intermittent failure i.e., successfully runs for a while and suddenly fails and runs successfully next time for a while and suddenly fails again!!!


One possible issue we are suspecting is that Distributed cache might be having conflicting "serde" jars!!! Do you have any advice on this thought, please let us know, we will try to implement that and see ???


Thank you.


Regards,
Raja.

-----Original Message-----
From: Sathi Chowdhury [mailto:sathi.chowdhury@lithium.com] 
Sent: Tuesday, September 30, 2014 9:33 PM
To: user@oozie.apache.org; Mohammad Islam
Cc: Sathi Chowdhury
Subject: RE: Hive action in Oozie failing intermittently

Hi Raja,
One clue...do you have any jar with dependencies that you are using which has hadoop and hive jars in it...so cluster jars may be clashing with it...I had faced issues due to this.
The
Sathi


-------- Original message --------
From: Mohammad Islam
Date:2014/09/30 1:15 AM (GMT-08:00)
To: user@oozie.apache.org
Subject: Re: Hive action in Oozie failing intermittently

Raja,
What version of Hive and Oozie are you using? In particular, what is in the shared lib for Hive. It might help.

I'm also confused that the same sql is working in some cases and in other cases, it is failing.
Regards,
Mohammad


On Tuesday, September 30, 2014 12:35 AM, Raja. Aravapalli <Ra...@target.com> wrote:




Hi  All,

We had an oozie workflow with simple "create" and "alter" stmts., with "create" stmt using "RCFILE" file format in Hive Action.

The challenge we are facing is...

This Hive action is executing successfully sometimes and failing sometimes... we weren't able to fix this.

It is throwing "NoSuchMethodError"  exception in regard to "serde".


Log Type: stderr

Log Length: 4289

14/09/30 01:10:37 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.getPrimitiveTypeInfo(Ljava/lang/String;)Lorg/apache/hadoop/hive/serde2/typeinfo/TypeInfo;
java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.getPrimitiveTypeInfo(Ljava/lang/String;)Lorg/apache/hadoop/hive/serde2/typeinfo/TypeInfo;
    at org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerNumericType(FunctionRegistry.java:630)
    at org.apache.hadoop.hive.ql.exec.FunctionRegistry.<clinit>(FunctionRegistry.java:636)
    at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:208)
    at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:78)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:645)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
    at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:318)
    at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:279)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
    at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)



Can someone please guide me in fixing this...  ???


Thanks.
Raja.