You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jay Vyas <ja...@gmail.com> on 2013/12/30 13:17:23 UTC

RegExSerde over JDBC... is it possible>

Hi folks..


Is there an JDBC or other API driven, programmatic way to launch a Hive Job
using SerDe's from the contrib library?

I've been attempting to run a query using Hive's JDBC driver which uses the
regex SerDe, however, it appears that hive doesn;t see this class.  As Im
launching the job from java via JDBC, im not sure where the class is not
found (i added it to HIVE_HOME/lib AND HADOOP_HOME/lib, so this is *really*
odd ).

2013-12-30 07:00:58.795 java[5829:1c03] Unable to load realm info from
SCDynamicStore
org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is
not allowed in prolog.
Continuing ...
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/serde2/AbstractSerDe
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
    at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:64)
    at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:106)
    at com.sun.beans.finder.ClassFinder.resolveClass(ClassFinder.java:166)
    at
com.sun.beans.decoder.DocumentHandler.findClass(DocumentHandler.java:382)
    at
com.sun.beans.decoder.ClassElementHandler.getValue(ClassElementHandler.java:60)
    at
com.sun.beans.decoder.StringElementHandler.getValueObject(StringElementHandler.java:92)
    at
com.sun.beans.decoder.ElementHandler.endElement(ElementHandler.java:169)
    at
com.sun.beans.decoder.DocumentHandler.endElement(DocumentHandler.java:305)
    at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:606)
    at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1741)
    at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2898)
    at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
    at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488)
    at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
    at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
    at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
    at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568)
    at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:302)
    at com.sun.beans.decoder.DocumentHandler.parse(DocumentHandler.java:356)
    at java.beans.XMLDecoder.parsingComplete(XMLDecoder.java:192)
    at java.beans.XMLDecoder.readObject(XMLDecoder.java:238)
    at
org.apache.hadoop.hive.ql.exec.Utilities.deserializeMapRedWork(Utilities.java:542)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:679)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.serde2.AbstractSerDe
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
    ... 44 more


I've seen a JIRA alluding that custom serdes may *not* work in Hive JDBC...
Is that still the case https://issues.apache.org/jira/browse/HIVE-2685 ?

-- 
Jay Vyas
http://jayunit100.blogspot.com

Re: RegExSerde over JDBC... is it possible>

Posted by Jay Vyas <ja...@gmail.com>.
Ah... i see whats going on.

When hive finally issues a "load" statement that actually USES the SerDe,
hadoop takes over.

At that point, hadoop/lib needs to have the hive serde libraries copied
into it, otherwise it cant find the AbstractSerDe classes.

I wonder if there is a more elegant way to integrate hive/hadoop that im
not thinking of (hive_test maybe :))


On Mon, Dec 30, 2013 at 7:17 AM, Jay Vyas <ja...@gmail.com> wrote:

> Hi folks..
>
>
> Is there an JDBC or other API driven, programmatic way to launch a Hive
> Job using SerDe's from the contrib library?
>
> I've been attempting to run a query using Hive's JDBC driver which uses
> the regex SerDe, however, it appears that hive doesn;t see this class.  As
> Im launching the job from java via JDBC, im not sure where the class is not
> found (i added it to HIVE_HOME/lib AND HADOOP_HOME/lib, so this is *really*
> odd ).
>
> 2013-12-30 07:00:58.795 java[5829:1c03] Unable to load realm info from
> SCDynamicStore
> org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is
> not allowed in prolog.
> Continuing ...
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/serde2/AbstractSerDe
>     at java.lang.ClassLoader.defineClass1(Native Method)
>     at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
>     at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>     at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>     at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:264)
>     at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:64)
>     at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:106)
>     at com.sun.beans.finder.ClassFinder.resolveClass(ClassFinder.java:166)
>     at
> com.sun.beans.decoder.DocumentHandler.findClass(DocumentHandler.java:382)
>     at
> com.sun.beans.decoder.ClassElementHandler.getValue(ClassElementHandler.java:60)
>     at
> com.sun.beans.decoder.StringElementHandler.getValueObject(StringElementHandler.java:92)
>     at
> com.sun.beans.decoder.ElementHandler.endElement(ElementHandler.java:169)
>     at
> com.sun.beans.decoder.DocumentHandler.endElement(DocumentHandler.java:305)
>     at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:606)
>     at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1741)
>     at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2898)
>     at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
>     at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488)
>     at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
>     at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
>     at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
>     at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
>     at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568)
>     at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:302)
>     at
> com.sun.beans.decoder.DocumentHandler.parse(DocumentHandler.java:356)
>     at java.beans.XMLDecoder.parsingComplete(XMLDecoder.java:192)
>     at java.beans.XMLDecoder.readObject(XMLDecoder.java:238)
>     at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializeMapRedWork(Utilities.java:542)
>     at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:679)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:601)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.serde2.AbstractSerDe
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>     ... 44 more
>
>
> I've seen a JIRA alluding that custom serdes may *not* work in Hive
> JDBC... Is that still the case
> https://issues.apache.org/jira/browse/HIVE-2685 ?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>



-- 
Jay Vyas
http://jayunit100.blogspot.com