You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Atul Paldhikar (apaldhik)" <ap...@cisco.com> on 2014/11/27 08:58:19 UTC
Can't integrate Elasticsearch with Hive
Hi All,
I am using Hive 0.13.1 and Hadoop 2.5.1 and trying to create an external table so data can me loaded from Hive to Elasticsearch. However I keep getting the following error. I have tried with following jars but same error. I will really appreciate for any pointers.
Thanks
- Atul
<property>
<name>hive.aux.jars.path</name>
<!--
<value>/apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar</value>
-->
<value>/apps/sas/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.Beta3.jar</value>
<description>A comma separated list (with no spaces) of the jar files</description>
</property>
ERROR :
2014-11-26 23:09:22,069 ERROR [main]: exec.DDLTask (DDLTask.java:execute(478)) - java.lang.IllegalAccessError: tried to access class org.elasticsearch.hadoop.hive.HiveUtils from class org.elasticsearch.hadoop.hive.EsSerDe
at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:81)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:339)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:288)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:281)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:631)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:593)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4189)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
2014-11-26 23:09:22,069 ERROR [main]: ql.Driver (SessionState.java:printError(545)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. tried to access class org.elasticsearch.hadoop.hive.HiveUtils from class org.elasticsearch.hadoop.hive.EsSerDe
RE: Can't integrate Elasticsearch with Hive
Posted by "Atul Paldhikar (apaldhik)" <ap...@cisco.com>.
Actually even that issue is resolved ☺ There is spelling difference in the sample available on the web, all of them have the storage class as “EsStorageHandler” however only Costin Lieu’s post says it is “ESStorageHandler” which is right !
Now the next problem, the MapReduce job is failing for some reason. I am still a beginner in Hadoop so not exactly sure where to debug. Here are some logs, looks some bad character “&#” in the job.xml file. But I that is generated by Hive right ?
Hive Log :------------------------------------------------
hive> insert overwrite table ex_address select name, st_no, st_name, city, state, zip from employee.address;
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1417158738771_0001, Tracking URL = http://finattr-comp-dev-01:8088/proxy/application_1417158738771_0001/
Kill Command = /apps/hadoop-2.5.1/bin/hadoop job -kill job_1417158738771_0001
Hadoop job information for Stage-0: number of mappers: 0; number of reducers: 0
2014-11-27 23:13:37,547 Stage-0 map = 0%, reduce = 0%
Ended Job = job_1417158738771_0001 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
Container Job Logs:
Stderr:---------------------------------------------
[sas@finattr-comp-dev-01 container_1417158738771_0001_02_000001]$ cat stderr
[Fatal Error] job.xml:606:51: Character reference "&#
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Syslog:---------------------------------------------
[sas@finattr-comp-dev-01 container_1417158738771_0001_02_000001]$ cat syslog
2014-11-27 23:13:36,023 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1417158738771_0001_000002
2014-11-27 23:13:36,334 FATAL [main] org.apache.hadoop.conf.Configuration: error parsing conf job.xml
org.xml.sax.SAXParseException; systemId: file:///tmp/hadoop-sas/nm-local-dir/usercache/sas/appcache/application_1417158738771_0001/container_1417158738771_0001_02_000001/job.xml; lineNumber: 606; columnNumber: 51; Character reference "&#
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2183)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2252)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2205)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2112)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1078)
at org.apache.hadoop.mapreduce.v2.util.MRWebAppUtil.initialize(MRWebAppUtil.java:50)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
2014-11-27 23:13:36,337 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:///tmp/hadoop-sas/nm-local-dir/usercache/sas/appcache/application_1417158738771_0001/container_1417158738771_0001_02_000001/job.xml; lineNumber: 606; columnNumber: 51; Character reference "&#
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2348)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2205)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2112)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1078)
at org.apache.hadoop.mapreduce.v2.util.MRWebAppUtil.initialize(MRWebAppUtil.java:50)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
Caused by: org.xml.sax.SAXParseException; systemId: file:///tmp/hadoop-sas/nm-local-dir/usercache/sas/appcache/application_1417158738771_0001/container_1417158738771_0001_02_000001/job.xml; lineNumber: 606; columnNumber: 51; Character reference "&#
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2183)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2252)
... 5 more
2014-11-27 23:13:36,340 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1
Thanks
- Atul
From: Muthu Pandi [mailto:muthu1086@gmail.com]
Sent: Thursday, November 27, 2014 11:20 PM
To: user@hive.apache.org
Subject: Re: Can't integrate Elasticsearch with Hive
How did you get elasticsearch jar??
Try to build the jar against your version and use that jar.
Regards
Muthupandi.K
[Image removed by sender.] Think before you print.
On Fri, Nov 28, 2014 at 12:23 PM, Atul Paldhikar (apaldhik) <ap...@cisco.com>> wrote:
Thanks Muthu for pointing out the issue.
However unfortunately I am back to the same old issue while accessing the external table. I am getting the following exception again, there wasn’t any change in the environment.
I tried all the cleanup but still no luck :-( Here are the steps I tried
1. Removed the other jar for es-hadoop 2.1.x completely from the server
2. Updated the hive-site.xml as below, but didn't work
<property>
<name>hive.aux.jars.path</name>
<value>/apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar</value>
<description>A comma separated list (with no spaces) of the jar files</description>
</property>
3. Added the jar file in the hiveconf as below, still the same issue
hive --hiveconf hive.aux.jars.path=/apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar
4. Tried adding the jar file in the hive session, still didn't work
add jar /apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar;
Thanks
- Atul
ERROR:
2014-11-26 23:09:22,069 ERROR [main]: exec.DDLTask (DDLTask.java:execute(478)) - java.lang.IllegalAccessError: tried to access class org.elasticsearch.hadoop.hive.HiveUtils from class org.elasticsearch.hadoop.hive.EsSerDe
at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:81)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:339)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:288)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:281)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:631)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:593)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4189)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
2014-11-26 23:09:22,069 ERROR [main]: ql.Driver (SessionState.java:printError(545)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. tried to access class org.elasticsearch.hadoop.hive.HiveUtils from class org.elasticsearch.hadoop.hive.EsSerDe
From: Muthu Pandi [mailto:muthu1086@gmail.com<ma...@gmail.com>]
Sent: Thursday, November 27, 2014 9:08 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Can't integrate Elasticsearch with Hive
Hi Atul
Check your typo "tblproperties('es.resources' = 'employee/address') " need to be "es.resource" as it rightly indicated in the Error.
Regards
Muthupandi.K
[Image removed by sender.] Think before you print.
On Fri, Nov 28, 2014 at 3:43 AM, Edward Capriolo <ed...@gmail.com>> wrote:
Because the elastic search storage handler is not part of hive you might be able to find more help asking the author of the storage handler. From the errors you are reporting it likely does not work and not tested against this version of hive.
Re: Can't integrate Elasticsearch with Hive
Posted by Muthu Pandi <mu...@gmail.com>.
How did you get elasticsearch jar??
Try to build the jar against your version and use that jar.
*RegardsMuthupandi.K*
Think before you print.
On Fri, Nov 28, 2014 at 12:23 PM, Atul Paldhikar (apaldhik) <
apaldhik@cisco.com> wrote:
> Thanks Muthu for pointing out the issue.
>
>
>
> However unfortunately I am back to the same old issue while accessing the
> external table. I am getting the following exception again, there wasn’t
> any change in the environment.
>
>
>
> I tried all the cleanup but still no luck :-( Here are the steps I tried
>
>
>
> 1. Removed the other jar for es-hadoop 2.1.x completely from the server
>
>
>
> 2. Updated the hive-site.xml as below, but didn't work
>
>
>
> <property>
>
> <name>hive.aux.jars.path</name>
>
>
> <value>/apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar</value>
>
> <description>A comma separated list (with no spaces) of the jar
> files</description>
>
> </property>
>
>
>
> 3. Added the jar file in the hiveconf as below, still the same issue
>
>
>
> hive --hiveconf
> hive.aux.jars.path=/apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar
>
>
>
> 4. Tried adding the jar file in the hive session, still didn't work
>
>
>
> add jar
> /apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar;
>
>
>
> Thanks
>
> - Atul
>
>
>
> ERROR:
>
>
>
> 2014-11-26 23:09:22,069 ERROR [main]: exec.DDLTask
> (DDLTask.java:execute(478)) - java.lang.IllegalAccessError: tried to access
> class org.elasticsearch.hadoop.hive.HiveUtils from class
> org.elasticsearch.hadoop.hive.EsSerDe
>
> at
> org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:81)
>
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:339)
>
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:288)
>
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:281)
>
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:631)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:593)
>
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4189)
>
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281)
>
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
>
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
>
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503)
>
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270)
>
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
>
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
>
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
>
>
> 2014-11-26 23:09:22,069 ERROR [main]: ql.Driver
> (SessionState.java:printError(545)) - FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. tried to access class
> org.elasticsearch.hadoop.hive.HiveUtils from class
> org.elasticsearch.hadoop.hive.EsSerDe
>
>
>
> *From:* Muthu Pandi [mailto:muthu1086@gmail.com]
> *Sent:* Thursday, November 27, 2014 9:08 PM
> *To:* user@hive.apache.org
> *Subject:* Re: Can't integrate Elasticsearch with Hive
>
>
>
> Hi Atul
>
>
>
> Check your typo "tblproperties('es.resources' = 'employee/address')
> " need to be "es.resource" as it rightly indicated in the Error.
>
>
>
> *Regards Muthupandi.K*
>
> *[image: Image removed by sender.]* Think before you print.
>
>
>
>
>
> On Fri, Nov 28, 2014 at 3:43 AM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
> Because the elastic search storage handler is not part of hive you might
> be able to find more help asking the author of the storage handler. From
> the errors you are reporting it likely does not work and not tested against
> this version of hive.
>
>
>
RE: Can't integrate Elasticsearch with Hive
Posted by "Atul Paldhikar (apaldhik)" <ap...@cisco.com>.
Thanks Muthu for pointing out the issue.
However unfortunately I am back to the same old issue while accessing the external table. I am getting the following exception again, there wasn’t any change in the environment.
I tried all the cleanup but still no luck :-( Here are the steps I tried
1. Removed the other jar for es-hadoop 2.1.x completely from the server
2. Updated the hive-site.xml as below, but didn't work
<property>
<name>hive.aux.jars.path</name>
<value>/apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar</value>
<description>A comma separated list (with no spaces) of the jar files</description>
</property>
3. Added the jar file in the hiveconf as below, still the same issue
hive --hiveconf hive.aux.jars.path=/apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar
4. Tried adding the jar file in the hive session, still didn't work
add jar /apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar;
Thanks
- Atul
ERROR:
2014-11-26 23:09:22,069 ERROR [main]: exec.DDLTask (DDLTask.java:execute(478)) - java.lang.IllegalAccessError: tried to access class org.elasticsearch.hadoop.hive.HiveUtils from class org.elasticsearch.hadoop.hive.EsSerDe
at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:81)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:339)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:288)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:281)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:631)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:593)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4189)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
2014-11-26 23:09:22,069 ERROR [main]: ql.Driver (SessionState.java:printError(545)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. tried to access class org.elasticsearch.hadoop.hive.HiveUtils from class org.elasticsearch.hadoop.hive.EsSerDe
From: Muthu Pandi [mailto:muthu1086@gmail.com]
Sent: Thursday, November 27, 2014 9:08 PM
To: user@hive.apache.org
Subject: Re: Can't integrate Elasticsearch with Hive
Hi Atul
Check your typo "tblproperties('es.resources' = 'employee/address') " need to be "es.resource" as it rightly indicated in the Error.
Regards
Muthupandi.K
[Image removed by sender.] Think before you print.
On Fri, Nov 28, 2014 at 3:43 AM, Edward Capriolo <ed...@gmail.com>> wrote:
Because the elastic search storage handler is not part of hive you might be able to find more help asking the author of the storage handler. From the errors you are reporting it likely does not work and not tested against this version of hive.
Re: Can't integrate Elasticsearch with Hive
Posted by Muthu Pandi <mu...@gmail.com>.
Hi Atul
Check your typo "tblproperties('es.resources' = 'employee/address') " need
to be "es.resource" as it rightly indicated in the Error.
*RegardsMuthupandi.K*
Think before you print.
On Fri, Nov 28, 2014 at 3:43 AM, Edward Capriolo <ed...@gmail.com>
wrote:
> Because the elastic search storage handler is not part of hive you might
> be able to find more help asking the author of the storage handler. From
> the errors you are reporting it likely does not work and not tested against
> this version of hive.
>
Re: Can't integrate Elasticsearch with Hive
Posted by Edward Capriolo <ed...@gmail.com>.
Because the elastic search storage handler is not part of hive you might be
able to find more help asking the author of the storage handler. From the
errors you are reporting it likely does not work and not tested against
this version of hive.
RE: Can't integrate Elasticsearch with Hive
Posted by "Atul Paldhikar (apaldhik)" <ap...@cisco.com>.
I was able to create the External table. But now I am getting an issue in loading the data in the table. I did specify the “es.resources” in the external table so I am not sure why this error is coming.
External Table:
create external table ex_address (name String, st_no INT, st_name string, city string, state string, zip INT) stored by 'org.elasticsearch.hadoop.hive.EsStorageHandler' tblproperties('es.resources' = 'employee/address'
Insert Statement:
insert overwrite table ex_address select name, st_no, st_name, city, state, zip from employee.address;
Hive Logs:
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: No resource ['es.resource'] (index/query/location) specified
at org.elasticsearch.hadoop.util.Assert.hasText(Assert.java:30)
at org.elasticsearch.hadoop.mr.EsOutputFormat.init(EsOutputFormat.java:336)
at org.elasticsearch.hadoop.mr.EsOutputFormat.checkOutputSpecs(EsOutputFormat.java:330)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.checkOutputSpecs(FileSinkOperator.java:951)
at org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:67)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:460)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Job Submission failed with exception 'org.elasticsearch.hadoop.EsHadoopIllegalArgumentException(No resource ['es.resource'] (index/query/location) specified)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
hive>
From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
Sent: Thursday, November 27, 2014 7:43 AM
To: user@hive.apache.org
Subject: Re: Can't integrate Elasticsearch with Hive
https://docs.oracle.com/javase/7/docs/api/java/lang/IllegalAccessError.html
Likley this means the storage handler was built against a different version of hive or one of the dependencies of the two has changed in an incompatible way. E.G. (ES) antlr 2.5 vs (hive) antlr 2.6 or some other dependency.
On Thu, Nov 27, 2014 at 2:58 AM, Atul Paldhikar (apaldhik) <ap...@cisco.com>> wrote:
Hi All,
I am using Hive 0.13.1 and Hadoop 2.5.1 and trying to create an external table so data can me loaded from Hive to Elasticsearch. However I keep getting the following error. I have tried with following jars but same error. I will really appreciate for any pointers.
Thanks
- Atul
<property>
<name>hive.aux.jars.path</name>
<!--
<value>/apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar</value>
-->
<value>/apps/sas/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.Beta3.jar</value>
<description>A comma separated list (with no spaces) of the jar files</description>
</property>
ERROR :
2014-11-26 23:09:22,069 ERROR [main]: exec.DDLTask (DDLTask.java:execute(478)) - java.lang.IllegalAccessError: tried to access class org.elasticsearch.hadoop.hive.HiveUtils from class org.elasticsearch.hadoop.hive.EsSerDe
at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:81)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:339)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:288)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:281)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:631)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:593)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4189)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
2014-11-26 23:09:22,069 ERROR [main]: ql.Driver (SessionState.java:printError(545)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. tried to access class org.elasticsearch.hadoop.hive.HiveUtils from class org.elasticsearch.hadoop.hive.EsSerDe
Re: Can't integrate Elasticsearch with Hive
Posted by Edward Capriolo <ed...@gmail.com>.
https://docs.oracle.com/javase/7/docs/api/java/lang/IllegalAccessError.html
Likley this means the storage handler was built against a different version
of hive or one of the dependencies of the two has changed in an
incompatible way. E.G. (ES) antlr 2.5 vs (hive) antlr 2.6 or some other
dependency.
On Thu, Nov 27, 2014 at 2:58 AM, Atul Paldhikar (apaldhik) <
apaldhik@cisco.com> wrote:
> Hi All,
>
>
>
> I am using Hive 0.13.1 and Hadoop 2.5.1 and trying to create an external
> table so data can me loaded from Hive to Elasticsearch. However I keep
> getting the following error. I have tried with following jars but same
> error. I will really appreciate for any pointers.
>
>
>
> Thanks
>
> - Atul
>
>
>
> <property>
>
> <name>hive.aux.jars.path</name>
>
> <!--
>
>
> <value>/apps/sas/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-2.0.2.jar</value>
>
> -->
>
>
> <value>/apps/sas/elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.Beta3.jar</value>
>
> <description>A comma separated list (with no spaces) of the jar
> files</description>
>
> </property>
>
>
>
> ERROR :
>
>
>
> 2014-11-26 23:09:22,069 ERROR [main]: exec.DDLTask
> (DDLTask.java:execute(478)) - java.lang.IllegalAccessError: tried to access
> class org.elasticsearch.hadoop.hive.HiveUtils from class
> org.elasticsearch.hadoop.hive.EsSerDe
>
> at
> org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:81)
>
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:339)
>
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:288)
>
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:281)
>
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:631)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:593)
>
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4189)
>
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281)
>
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
>
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
>
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503)
>
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270)
>
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
>
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
>
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
>
>
> 2014-11-26 23:09:22,069 ERROR [main]: ql.Driver
> (SessionState.java:printError(545)) - FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. tried to access class
> org.elasticsearch.hadoop.hive.HiveUtils from class
> org.elasticsearch.hadoop.hive.EsSerDe
>
>
>