You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jay Vyas <ja...@gmail.com> on 2013/12/30 13:17:23 UTC
RegExSerde over JDBC... is it possible>
Hi folks..
Is there an JDBC or other API driven, programmatic way to launch a Hive Job
using SerDe's from the contrib library?
I've been attempting to run a query using Hive's JDBC driver which uses the
regex SerDe, however, it appears that hive doesn;t see this class. As Im
launching the job from java via JDBC, im not sure where the class is not
found (i added it to HIVE_HOME/lib AND HADOOP_HOME/lib, so this is *really*
odd ).
2013-12-30 07:00:58.795 java[5829:1c03] Unable to load realm info from
SCDynamicStore
org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is
not allowed in prolog.
Continuing ...
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/serde2/AbstractSerDe
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:64)
at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:106)
at com.sun.beans.finder.ClassFinder.resolveClass(ClassFinder.java:166)
at
com.sun.beans.decoder.DocumentHandler.findClass(DocumentHandler.java:382)
at
com.sun.beans.decoder.ClassElementHandler.getValue(ClassElementHandler.java:60)
at
com.sun.beans.decoder.StringElementHandler.getValueObject(StringElementHandler.java:92)
at
com.sun.beans.decoder.ElementHandler.endElement(ElementHandler.java:169)
at
com.sun.beans.decoder.DocumentHandler.endElement(DocumentHandler.java:305)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:606)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1741)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2898)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:302)
at com.sun.beans.decoder.DocumentHandler.parse(DocumentHandler.java:356)
at java.beans.XMLDecoder.parsingComplete(XMLDecoder.java:192)
at java.beans.XMLDecoder.readObject(XMLDecoder.java:238)
at
org.apache.hadoop.hive.ql.exec.Utilities.deserializeMapRedWork(Utilities.java:542)
at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:679)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.serde2.AbstractSerDe
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 44 more
I've seen a JIRA alluding that custom serdes may *not* work in Hive JDBC...
Is that still the case https://issues.apache.org/jira/browse/HIVE-2685 ?
--
Jay Vyas
http://jayunit100.blogspot.com
Re: RegExSerde over JDBC... is it possible>
Posted by Jay Vyas <ja...@gmail.com>.
Ah... i see whats going on.
When hive finally issues a "load" statement that actually USES the SerDe,
hadoop takes over.
At that point, hadoop/lib needs to have the hive serde libraries copied
into it, otherwise it cant find the AbstractSerDe classes.
I wonder if there is a more elegant way to integrate hive/hadoop that im
not thinking of (hive_test maybe :))
On Mon, Dec 30, 2013 at 7:17 AM, Jay Vyas <ja...@gmail.com> wrote:
> Hi folks..
>
>
> Is there an JDBC or other API driven, programmatic way to launch a Hive
> Job using SerDe's from the contrib library?
>
> I've been attempting to run a query using Hive's JDBC driver which uses
> the regex SerDe, however, it appears that hive doesn;t see this class. As
> Im launching the job from java via JDBC, im not sure where the class is not
> found (i added it to HIVE_HOME/lib AND HADOOP_HOME/lib, so this is *really*
> odd ).
>
> 2013-12-30 07:00:58.795 java[5829:1c03] Unable to load realm info from
> SCDynamicStore
> org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is
> not allowed in prolog.
> Continuing ...
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/serde2/AbstractSerDe
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:64)
> at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:106)
> at com.sun.beans.finder.ClassFinder.resolveClass(ClassFinder.java:166)
> at
> com.sun.beans.decoder.DocumentHandler.findClass(DocumentHandler.java:382)
> at
> com.sun.beans.decoder.ClassElementHandler.getValue(ClassElementHandler.java:60)
> at
> com.sun.beans.decoder.StringElementHandler.getValueObject(StringElementHandler.java:92)
> at
> com.sun.beans.decoder.ElementHandler.endElement(ElementHandler.java:169)
> at
> com.sun.beans.decoder.DocumentHandler.endElement(DocumentHandler.java:305)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:606)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1741)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2898)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:302)
> at
> com.sun.beans.decoder.DocumentHandler.parse(DocumentHandler.java:356)
> at java.beans.XMLDecoder.parsingComplete(XMLDecoder.java:192)
> at java.beans.XMLDecoder.readObject(XMLDecoder.java:238)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializeMapRedWork(Utilities.java:542)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:679)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.serde2.AbstractSerDe
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> ... 44 more
>
>
> I've seen a JIRA alluding that custom serdes may *not* work in Hive
> JDBC... Is that still the case
> https://issues.apache.org/jira/browse/HIVE-2685 ?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
--
Jay Vyas
http://jayunit100.blogspot.com