[jira] [Created] (PIG-5075) Pig ORCStorage with Snappy Compression will fail with NoClassDefFoundError org/iq80/snappy/Snappy

["Prabhu Joseph (JIRA)" <ji...@apache.org>]

Prabhu Joseph created PIG-5075:
----------------------------------

             Summary: Pig ORCStorage with Snappy Compression will fail with NoClassDefFoundError org/iq80/snappy/Snappy
                 Key: PIG-5075
                 URL: https://issues.apache.org/jira/browse/PIG-5075
             Project: Pig
          Issue Type: Bug
          Components: impl
    Affects Versions: 0.16.0
            Reporter: Prabhu Joseph


Pig Script to store a text file into ORC with Snappy compression enabled fails with java.lang.NoClassDefFoundError: org/iq80/snappy/Snappy

The hive-exec jar which comes with Pig does not have snappy jars whereas the hive-exec jar comes with hive has that.

{code}
[root@prabhuSpark3 lib]# jar tvf /usr/hdp/2.4.2.0-258/pig/lib/hive-exec-1.2.1000.2.4.2.0-258-core.jar | grep iq80
[root@prabhuSpark3 lib]#
[root@prabhuSpark3 lib]# jar tvf /usr/hdp/2.4.2.0-258/hive/lib/hive-exec-1.2.1000.2.4.2.0-258.jar | grep iq80
0 Mon Apr 25 06:49:28 UTC 2016 org/iq80/
0 Mon Apr 25 06:49:28 UTC 2016 org/iq80/snappy/
1577 Mon Apr 25 06:49:28 UTC 2016 org/iq80/snappy/Snappy.class 
{code}

Repro:
{code}
[root@prabhuSpark3 lib]# hadoop fs -cat /tmp/data
hadoop,5
hive,4
pig,3
tez,2
hawq,1

MYFILE = LOAD '/tmp/data' using PigStorage(',') As (name:chararray,age:int);
Store MYFILE into '/tmp/orcsnappydata' using OrcStorage('-c SNAPPY');

2016-09-22 03:29:06,830 [main] ERROR org.apache.pig.backend.hadoop.executionengine.Launcher - Backend error message
Error: org/iq80/snappy/Snappy
2016-09-22 03:29:06,831 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 0: org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backed error: Error: org/iq80/snappy/Snappy
2016-09-22 03:29:06,831 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
{code}

Workaround:
Register /usr/hdp/2.4.2.0-258/hive/lib/hive-exec-1.2.1000.2.4.2.0-258.jar; 

As part of this Bug, we want to include the Snappy jars which comes into Pig hive-exec jar.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)