You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Tarush Grover (JIRA)" <ji...@apache.org> on 2015/11/21 15:37:11 UTC
[jira] [Reopened] (HIVE-9970) Hive on spark
[ https://issues.apache.org/jira/browse/HIVE-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tarush Grover reopened HIVE-9970:
---------------------------------
Assignee: Tarush Grover
This issue still persists my environment is :
hadoop = 2.6
hive = 1.1.1
spark = 1.5.1
below are logs in the hive.log :
15/11/21 19:56:27 [HiveServer2-Background-Pool: Thread-39]: ERROR exec.Task: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1638)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1397)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:145)
at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:70)
at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:209)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot run program "/home/adt/server/spark1.5/spark-1.5.1set mapreduce.input.fileinputformat.split.maxsize=750000000set hive.vectorized.execution.enabled=trueset hive.cbo.enable=trueset hive.optimize.reducededuplication.min.reducer=4set hive.optimize.reducededuplication=trueset hive.orc.splits.include.file.footer=falseset hive.merge.mapfiles=trueset hive.merge.sparkfiles=falseset hive.merge.smallfiles.avgsize=16000000set hive.merge.size.per.task=256000000set hive.merge.orcfile.stripe.level=trueset hive.auto.convert.join=trueset hive.auto.convert.join.noconditionaltask=trueset hive.auto.convert.join.noconditionaltask.size=894435328set hive.optimize.bucketmapjoin.sortedmerge=falseset hive.map.aggr.hash.percentmemory=0.5set hive.map.aggr=trueset hive.optimize.sort.dynamic.partition=falseset hive.stats.autogather=trueset hive.stats.fetch.column.stats=trueset hive.vectorized.execution.reduce.enabled=falseset hive.vectorized.groupby.checkinterval=4096set hive.vectorized.groupby.flush.percent=0.1set hive.compute.query.using.stats=trueset hive.limit.pushdown.memory.usage=0.4set hive.optimize.index.filter=trueset hive.exec.reducers.bytes.per.reducer=67108864set hive.smbjoin.cache.rows=10000set hive.exec.orc.default.stripe.size=67108864set hive.fetch.task.conversion=moreset hive.fetch.task.conversion.threshold=1073741824set hive.fetch.task.aggr=falseset mapreduce.input.fileinputformat.list-status.num-threads=5set spark.kryo.referenceTracking=false#set spark.kryo.classesToRegister=org.apache.hadoop.hive.ql.io.HiveKey,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch/bin/spark-submit": error=36, File name too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.hive.spark.client.SparkClientImpl.startDriver(SparkClientImpl.java:376)
at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:89)
at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:88)
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:58)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
... 22 more
Caused by: java.io.IOException: error=36, File name too long
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 28 more
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
> Hive on spark
> -------------
>
> Key: HIVE-9970
> URL: https://issues.apache.org/jira/browse/HIVE-9970
> Project: Hive
> Issue Type: Bug
> Reporter: Amithsha
> Assignee: Tarush Grover
>
> Hi all,
> Recently i have configured Spark 1.2.0 and my environment is hadoop
> 2.6.0 hive 1.1.0 Here i have tried hive on Spark while executing
> insert into i am getting the following g error.
> Query ID = hadoop2_20150313162828_8764adad-a8e4-49da-9ef5-35e4ebd6bc63
> Total jobs = 1
> Launching Job 1 out of 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=<number>
> Failed to execute spark task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create
> spark client.)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> Have added the spark-assembly jar in hive lib
> And also in hive console using the command add jar followed by the steps
> set spark.home=/opt/spark-1.2.1/;
> add jar /opt/spark-1.2.1/assembly/target/scala-2.10/spark-assembly-1.2.1-hadoop2.4.0.jar;
> set hive.execution.engine=spark;
> set spark.master=spark://xxxxxxx:7077;
> set spark.eventLog.enabled=true;
> set spark.executor.memory=512m;
> set spark.serializer=org.apache.spark.serializer.KryoSerializer;
> Can anyone suggest!!!!
> Thanks & Regards
> Amithsha
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)