You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by S Malligarjunan <sm...@yahoo.com.INVALID> on 2014/08/26 21:53:18 UTC

Spark 1.1. doesn't work with hive context

Hello all,

I have just checked out branch-1.1 
and executed below command
./bin/spark-shell --driver-memory 1G
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)

val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) hiveContext.hql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)") hiveContext.hql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src") // Queries are expressed in HiveQL hiveContext.hql("FROM src SELECT key, value").collect().foreach(println)

I am getting the following exception

Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
... 72 more
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found

 
Thanks and Regards,
Sankar S.

Re: Spark 1.1. doesn't work with hive context

Posted by S Malligarjunan <sm...@yahoo.com.INVALID>.

It is my mistake, some how I have added the io.compression.codec property value as the above mentioned class. Resolved the problem now
 
Thanks and Regards,
Sankar S.  



On Wednesday, 27 August 2014, 1:23, S Malligarjunan <sm...@yahoo.com> wrote:
 


Hello all,

I have just checked out branch-1.1 
and executed below command
./bin/spark-shell --driver-memory 1G
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)

val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) hiveContext.hql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)") hiveContext.hql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src") // Queries are expressed in HiveQL hiveContext.hql("FROM src SELECT key, value").collect().foreach(println)

I am getting the following exception

Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
... 72 more
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found

 
Thanks and Regards,
Sankar S.