You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by ZhuGe <tc...@outlook.com> on 2015/07/27 09:35:26 UTC

hive.contrib.serde2.RegexSerDe not found

Hi all:I am testing the performance of hive on spark sql.The existing table is created with ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'  WITH SERDEPROPERTIES  ( 'input.regex' = '(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)','output.format.string' = '%1$s %2$s %3$s %4$s %5$s %16$s %7$s %8$s %9$s %10$s %11$s %12$s %13$s %14$s %15$s %16$s %17$s ')STORED AS TEXTFILElocation '/data/BaseData/wx/xx/xx/xx/xx';
When i use spark sql(spark-shell) to query the existing table, got exception like this:Caused by: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found)	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:382)	at org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:249)
I add the jar dependency in the spark-shell command, still do not work.SPARK_SUBMIT_OPTS="-XX:MaxPermSize=256m" ./bin/spark-shell --jars /data/dbcenter/cdh5/spark-1.4.0-bin-hadoop2.4/hive-contrib-0.13.1-cdh5.2.0.jar,postgresql-9.2-1004-jdbc41.jar
How should i fix the problem?Cheers

Re: hive.contrib.serde2.RegexSerDe not found

Posted by Gianluca Privitera <gi...@studio.unibo.it>.

Try use: org.apache.hadoop.hive.serde2.RegexSerDe

GP

On 27 Jul 2015, at 09:35, ZhuGe <tc...@outlook.com>> wrote:

Hi all:
I am testing the performance of hive on spark sql.
The existing table is created with
ROW FORMAT
SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES
( 'input.regex' = '(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)\\|\\^\\|(.*?)'
,'output.format.string' = '%1$s %2$s %3$s %4$s %5$s %16$s %7$s %8$s %9$s %10$s %11$s %12$s %13$s %14$s %15$s %16$s %17$s ')
STORED AS TEXTFILE
location '/data/BaseData/wx/xx/xx/xx/xx';

When i use spark sql(spark-shell) to query the existing table, got exception like this:
Caused by: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:382)
at org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:249)

I add the jar dependency in the spark-shell command, still do not work.
SPARK_SUBMIT_OPTS="-XX:MaxPermSize=256m" ./bin/spark-shell --jars /data/dbcenter/cdh5/spark-1.4.0-bin-hadoop2.4/hive-contrib-0.13.1-cdh5.2.0.jar,postgresql-9.2-1004-jdbc41.jar

How should i fix the problem?
Cheers