You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Harish Butani (JIRA)" <ji...@apache.org> on 2014/03/28 00:49:15 UTC
[jira] [Commented] (HIVE-6336) Issue is hive 12 datanucleus
incompatability with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
[ https://issues.apache.org/jira/browse/HIVE-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950160#comment-13950160 ]
Harish Butani commented on HIVE-6336:
-------------------------------------
[~nigeljvm] this one doesn't seem critical for the hive 0.13 release. Can this be downgraded from blocker priority
> Issue is hive 12 datanucleus incompatability with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-6336
> URL: https://issues.apache.org/jira/browse/HIVE-6336
> Project: Hive
> Issue Type: Wish
> Components: HiveServer2
> Affects Versions: 0.12.0
> Environment: Hadoop 2.2 local derby Meatastore embedded
> Reporter: Nigel Savage
> Priority: Blocker
> Labels: HADOOP
>
> There is an with hive 12 datanucleus incompatability which seems to have invompatibility with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> The main question:
> *IF hive 0.12.0 and datanucleus are compatabile, then what is the version of datanucleus I should be using with Hive 12 and Hadoop 2.2?*
> The error which Im getting (this blocks me from properly running hive queries invoked from the "test" phase of a maven project)
> *To reproduce*
> I have hadoop and hive running as a pseudo cluster local mode and derby as the metastore
> I have the following environment variables
> {noformat}
> HADOOP_HOME=/home/ubu/hadoop
> JAVA_HOME=/usr/lib/jvm/java-7-oracle
> {noformat}
> I have the RegexSerDe declared in the hive-site.xml
> {noformat}
> <property>
> <name>hive.aux.jars.path</name>
> <value>file:///home/ubu/hadoop/lib/hive-contrib-0.12.0.jar </value>
> <description>This JAR file available to all users for alljobs</description>
> </property>
> {noformat}
> If I run with
> {noformat}
> <datanucleus.version>3.0.2</datanucleus.version>
> {noformat}
> I get the following 1 exception only
> 'java.lang.ClassNotFoundException...org.datanucleus.store.types.backed.Ma'
> HOWEVER, If I run with
> {noformat}
> <datanucleus.version>3.2.0-release</datanucleus.version>
> {noformat}
> I get the following 1 exception exception only
> java.lang.ClassNotFoundException:
> org/apache/hadoop/hive/contrib/serde2/RegexSerDe
> EXPLANATION
> The RegexSerDe class is picked up at run time but the datanucleus Map class is not available, I have checked in the datanucleus-core 3.0.2 jar and it is missing, Upgrading to the first datanucleus above 3.0.2 that includes the Map class throws the ClassNotFoundException for RegexSerDe.
> The earlier *3.0.2* datanucleus, code fails with the missing Map class but the RegexSerDe class is found, then when I upgrade to the
> 3.2.0-release the Map class is found but for some unkown reason the code/Hive no longer finds the RegexSerDe class
> I started using the same datanucleus dependencies found in this hive pom
> http://maven-repository.com/artifact/org.apache.hive/hive-metastore/0.12.0/pom
> below are the dependencies my latest attempts to get a functioning pom
> {noformat}
> <dependency>
> <groupId>org.apache.hbase</groupId>
> <artifactId>hbase-server</artifactId>
> <version>0.96.0-hadoop2</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hbase</groupId>
> <artifactId>hbase-client</artifactId>
> <version>0.96.0-hadoop2</version>
> </dependency>
> <!-- misc -->
> <dependency>
> <groupId>org.apache.commons</groupId>
> <artifactId>commons-lang3</artifactId>
> <version>3.1</version>
> </dependency>
> <dependency>
> <groupId>com.google.guava</groupId>
> <artifactId>guava</artifactId>
> <version>${guava.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.derby</groupId>
> <artifactId>derby</artifactId>
> <version>${derby.version}</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-core</artifactId>
> <version>${datanucleus.version}</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-rdbms</artifactId>
> <version>${datanucleus-rdbms.version}</version>
> </dependency>
> <dependency>
> <groupId>javax.jdo</groupId>
> <artifactId>jdo-api</artifactId>
> <version>3.0.1</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-api-jdo</artifactId>
> <version>${datanucleus.jdo.version}</version>
> <exclusions>
> <exclusion>
> <groupId>javax.jdo</groupId>
> <artifactId>jdo2-api</artifactId>
> </exclusion>
> <exclusion>
> <groupId>junit</groupId>
> <artifactId>junit</artifactId>
> </exclusion>
> <exclusion>
> <groupId>log4j</groupId>
> <artifactId>log4j</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <!-- hadoop -->
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${hadoop.version}</version>
> </dependency>
> <!-- hive -->
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-common</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-serde</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-exec</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-jdbc</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-contrib</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-metastore</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-cli</artifactId>
> <version>${hive.version}</version>
> <exclusions>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-core</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-api-jdo</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-rdbms</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-api</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-log4j12</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <dependency>
> <groupId>com.jolbox</groupId>
> <artifactId>bonecp</artifactId>
> <version>${bonecp.version}</version>
> </dependency>
> <!-- logging -->
> <dependency>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-api</artifactId>
> <version>${slf4j.version}</version>
> </dependency>
> <!-- SL4J Binding provided at runtime -->
> <dependency>
> <groupId>log4j</groupId>
> <artifactId>log4j</artifactId>
> <version>1.2.12</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-log4j12</artifactId>
> <version>${slf4j.version}</version>
> <scope>provided</scope>
> </dependency>
> <!-- Unit test artifacts -->
> <dependency>
> <groupId>junit</groupId>
> <artifactId>junit</artifactId>
> <version>4.11</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.hamcrest</groupId>
> <artifactId>hamcrest-all</artifactId>
> <version>1.3</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.mrunit</groupId>
> <artifactId>mrunit</artifactId>
> <version>1.0.0</version>
> <classifier>hadoop2</classifier>
> </dependency>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)