You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Archit Thakur <ar...@gmail.com> on 2014/05/08 13:41:31 UTC
Tez HIve problems.
Hi,
I am facing few problems while using Tez on Hive.
I am able to run Tez through YARN submitting independent MR procesing job.
Also, when I run SELECT query on hive shell by setting
*set hive.execution.engine=mr;*
but setting
*mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez*
This launches tez job, I checked by seeing AM's UI of machine.
But when I
*set hive.execution.engine=tez;*
*It gives me *Query ID =
admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
Total jobs = 1
Launching Job 1 out of 1
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.tez.TezTask
*I checked the hive logs:*
exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Am I doing something wrong? The Tez I am running by setting*
hive.execution.engine=mr;* but setting
*mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez. is it
correct? Does it launches Tez job or is it MR?*
Thanks and Regards,
Archit Thakur.
Re: Tez HIve problems.
Posted by Hitesh Shah <hi...@apache.org>.
@Archit, if the hive shell was shut down, can you obtain all the tez logs using:
$yarn logs -applicationId <appId>
Or, for a running query, go to the RM UI, find the application id and use the logs link to view/download the logs.
thanks
— Hitesh
On May 12, 2014, at 11:57 PM, Archit Thakur <ar...@gmail.com> wrote:
> @Vikram: I have attached the complete hive.log. PLease go in the last to see that it keeps on going and going.
> Applications logs: There are no tez logs generated.
>
> So, No exception or stack trace.
>
> @Bikas: Yes, it does in the release notes of https://issues.apache.org/jira/browse/HIVE-6098.
> There should be enough resources left because I am able to run it when I set set hive.execution.engine=mr;
> Thanks and Regards,
> Archit Thakur.
>
>
> On Mon, May 12, 2014 at 10:27 PM, Bikas Saha <bi...@hortonworks.com> wrote:
> Do the hive on tez instruction not specify that the hive-exec jar needs to be uploaded first?
>
>
> About the stuck job, you can check the resources in the cluster via the RM web UI. Please confirm that there are enough resources left in the cluster to run tasks. It may be that the cluster only has enough resources to run the App Master (controller process) and not the tasks.
>
>
> From: Archit Thakur [mailto:archit279thakur@gmail.com]
> Sent: Sunday, May 11, 2014 9:59 PM
> To: user@tez.incubator.apache.org
> Cc: user@hive.apache.org
> Subject: Re: Tez HIve problems.
>
>
> Hi Bikas,
>
> I was able to resolve this issue, by copying hive-exec.jar to hdfs://user/admin/, It was not able to localize it while running Tez.
>
> Though, I am facing some other problem now. My Tez job is not progressing,
>
> My Tez job doesn't progress, it just keeps printing:
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
>
> In hive logs also, it keeps on printing the same.
>
> Any idea, where should I look into or how should I debug the problem. Or
> pointers on problem itself would help.
>
> Thanks Archit.
>
>
> On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com> wrote:
>
> Setting execution engine to mr and framework name to yarn = Hive compiles to MR and runs on MR.
>
> Setting execution engine to mr and framework name to yarn-tez = Hive compiles to MR and runs on Tez.
>
> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>
>
> Did you change the execution engine in the same shell that was earlier running with execution engine set to mr? If yes, then try and set the execution engine in hive-site.xml and use a new hive shell.
>
> Have you set all required hive configurations?
>
>
> If you can still reproduce the issue, then you should create a jira in the Apache hive project to track the bug.
>
>
> Bikas
>
>
> From: Archit Thakur [mailto:archit279thakur@gmail.com]
> Sent: Thursday, May 08, 2014 4:42 AM
> To: user@hive.apache.org; user@tez.incubator.apache.org
> Subject: Tez HIve problems.
>
>
> Hi,
>
> I am facing few problems while using Tez on Hive.
>
> I am able to run Tez through YARN submitting independent MR procesing job.
>
> Also, when I run SELECT query on hive shell by setting
>
> set hive.execution.engine=mr;
>
> but setting mapreduce.framework.name=yarn-tez
>
> This launches tez job, I checked by seeing AM's UI of machine.
>
> But when I set hive.execution.engine=tez;
>
> It gives me
>
> Query ID = admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
> Total jobs = 1
> Launching Job 1 out of 1
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
>
> I checked the hive logs:
>
> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
> at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
> at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Am I doing something wrong? The Tez I am running by setting hive.execution.engine=mr; but setting mapreduce.framework.name=yarn-tez. is it correct? Does it launches Tez job or is it MR?
>
> Thanks and Regards,
>
> Archit Thakur.
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
> <hive.log>
Re: Tez HIve problems.
Posted by Hitesh Shah <hi...@apache.org>.
@Archit, if the hive shell was shut down, can you obtain all the tez logs using:
$yarn logs -applicationId <appId>
Or, for a running query, go to the RM UI, find the application id and use the logs link to view/download the logs.
thanks
— Hitesh
On May 12, 2014, at 11:57 PM, Archit Thakur <ar...@gmail.com> wrote:
> @Vikram: I have attached the complete hive.log. PLease go in the last to see that it keeps on going and going.
> Applications logs: There are no tez logs generated.
>
> So, No exception or stack trace.
>
> @Bikas: Yes, it does in the release notes of https://issues.apache.org/jira/browse/HIVE-6098.
> There should be enough resources left because I am able to run it when I set set hive.execution.engine=mr;
> Thanks and Regards,
> Archit Thakur.
>
>
> On Mon, May 12, 2014 at 10:27 PM, Bikas Saha <bi...@hortonworks.com> wrote:
> Do the hive on tez instruction not specify that the hive-exec jar needs to be uploaded first?
>
>
> About the stuck job, you can check the resources in the cluster via the RM web UI. Please confirm that there are enough resources left in the cluster to run tasks. It may be that the cluster only has enough resources to run the App Master (controller process) and not the tasks.
>
>
> From: Archit Thakur [mailto:archit279thakur@gmail.com]
> Sent: Sunday, May 11, 2014 9:59 PM
> To: user@tez.incubator.apache.org
> Cc: user@hive.apache.org
> Subject: Re: Tez HIve problems.
>
>
> Hi Bikas,
>
> I was able to resolve this issue, by copying hive-exec.jar to hdfs://user/admin/, It was not able to localize it while running Tez.
>
> Though, I am facing some other problem now. My Tez job is not progressing,
>
> My Tez job doesn't progress, it just keeps printing:
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
>
> In hive logs also, it keeps on printing the same.
>
> Any idea, where should I look into or how should I debug the problem. Or
> pointers on problem itself would help.
>
> Thanks Archit.
>
>
> On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com> wrote:
>
> Setting execution engine to mr and framework name to yarn = Hive compiles to MR and runs on MR.
>
> Setting execution engine to mr and framework name to yarn-tez = Hive compiles to MR and runs on Tez.
>
> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>
>
> Did you change the execution engine in the same shell that was earlier running with execution engine set to mr? If yes, then try and set the execution engine in hive-site.xml and use a new hive shell.
>
> Have you set all required hive configurations?
>
>
> If you can still reproduce the issue, then you should create a jira in the Apache hive project to track the bug.
>
>
> Bikas
>
>
> From: Archit Thakur [mailto:archit279thakur@gmail.com]
> Sent: Thursday, May 08, 2014 4:42 AM
> To: user@hive.apache.org; user@tez.incubator.apache.org
> Subject: Tez HIve problems.
>
>
> Hi,
>
> I am facing few problems while using Tez on Hive.
>
> I am able to run Tez through YARN submitting independent MR procesing job.
>
> Also, when I run SELECT query on hive shell by setting
>
> set hive.execution.engine=mr;
>
> but setting mapreduce.framework.name=yarn-tez
>
> This launches tez job, I checked by seeing AM's UI of machine.
>
> But when I set hive.execution.engine=tez;
>
> It gives me
>
> Query ID = admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
> Total jobs = 1
> Launching Job 1 out of 1
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
>
> I checked the hive logs:
>
> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
> at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
> at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Am I doing something wrong? The Tez I am running by setting hive.execution.engine=mr; but setting mapreduce.framework.name=yarn-tez. is it correct? Does it launches Tez job or is it MR?
>
> Thanks and Regards,
>
> Archit Thakur.
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
> <hive.log>
Re: Tez HIve problems.
Posted by Archit Thakur <ar...@gmail.com>.
@Vikram: I have attached the complete hive.log. PLease go in the last to
see that it keeps on going and going.
Applications logs: There are no tez logs generated.
So, No exception or stack trace.
@Bikas: Yes, it does in the release notes of
https://issues.apache.org/jira/browse/HIVE-6098.
There should be enough resources left because I am able to run it when I
set set hive.execution.engine=mr;
Thanks and Regards,
Archit Thakur.
On Mon, May 12, 2014 at 10:27 PM, Bikas Saha <bi...@hortonworks.com> wrote:
> Do the hive on tez instruction not specify that the hive-exec jar needs to
> be uploaded first?
>
>
>
> About the stuck job, you can check the resources in the cluster via the RM
> web UI. Please confirm that there are enough resources left in the cluster
> to run tasks. It may be that the cluster only has enough resources to run
> the App Master (controller process) and not the tasks.
>
>
>
> *From:* Archit Thakur [mailto:archit279thakur@gmail.com]
> *Sent:* Sunday, May 11, 2014 9:59 PM
> *To:* user@tez.incubator.apache.org
> *Cc:* user@hive.apache.org
> *Subject:* Re: Tez HIve problems.
>
>
>
> Hi Bikas,
>
> I was able to resolve this issue, by copying hive-exec.jar to
> hdfs://user/admin/, It was not able to localize it while running Tez.
>
> Though, I am facing some other problem now. My Tez job is not progressing,
>
> My Tez job doesn't progress, it just keeps printing:
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
>
> In hive logs also, it keeps on printing the same.
>
> Any idea, where should I look into or how should I debug the problem. Or
> pointers on problem itself would help.
>
> Thanks Archit.
>
>
>
> On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com> wrote:
>
> Setting execution engine to mr and framework name to yarn = Hive compiles
> to MR and runs on MR.
>
> Setting execution engine to mr and framework name to yarn-tez = Hive
> compiles to MR and runs on Tez.
>
> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>
>
>
> Did you change the execution engine in the same shell that was earlier
> running with execution engine set to mr? If yes, then try and set the
> execution engine in hive-site.xml and use a new hive shell.
>
> Have you set all required hive configurations?
>
>
>
> If you can still reproduce the issue, then you should create a jira in the
> Apache hive project to track the bug.
>
>
>
> Bikas
>
>
>
> *From:* Archit Thakur [mailto:archit279thakur@gmail.com]
> *Sent:* Thursday, May 08, 2014 4:42 AM
> *To:* user@hive.apache.org; user@tez.incubator.apache.org
> *Subject:* Tez HIve problems.
>
>
>
> Hi,
>
> I am facing few problems while using Tez on Hive.
>
> I am able to run Tez through YARN submitting independent MR procesing job.
>
> Also, when I run SELECT query on hive shell by setting
>
> *set hive.execution.engine=mr;*
>
> but setting *mapreduce.framework.name
> <http://mapreduce.framework.name>=yarn-tez*
>
> This launches tez job, I checked by seeing AM's UI of machine.
>
> But when I *set hive.execution.engine=tez;*
>
>
>
> *It gives me *Query ID =
> admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
> Total jobs = 1
> Launching Job 1 out of 1
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
>
> *I checked the hive logs:*
>
> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Am I doing something wrong? The Tez I am running by setting*
> hive.execution.engine=mr;* but setting *mapreduce.framework.name
> <http://mapreduce.framework.name>=yarn-tez. is it correct? Does it launches
> Tez job or is it MR?*
>
> Thanks and Regards,
>
> Archit Thakur.
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
RE: Tez HIve problems.
Posted by Bikas Saha <bi...@hortonworks.com>.
Do the hive on tez instruction not specify that the hive-exec jar needs to
be uploaded first?
About the stuck job, you can check the resources in the cluster via the RM
web UI. Please confirm that there are enough resources left in the cluster
to run tasks. It may be that the cluster only has enough resources to run
the App Master (controller process) and not the tasks.
*From:* Archit Thakur [mailto:archit279thakur@gmail.com]
*Sent:* Sunday, May 11, 2014 9:59 PM
*To:* user@tez.incubator.apache.org
*Cc:* user@hive.apache.org
*Subject:* Re: Tez HIve problems.
Hi Bikas,
I was able to resolve this issue, by copying hive-exec.jar to
hdfs://user/admin/, It was not able to localize it while running Tez.
Though, I am facing some other problem now. My Tez job is not progressing,
My Tez job doesn't progress, it just keeps printing:
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
In hive logs also, it keeps on printing the same.
Any idea, where should I look into or how should I debug the problem. Or
pointers on problem itself would help.
Thanks Archit.
On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com> wrote:
Setting execution engine to mr and framework name to yarn = Hive compiles
to MR and runs on MR.
Setting execution engine to mr and framework name to yarn-tez = Hive
compiles to MR and runs on Tez.
Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
Did you change the execution engine in the same shell that was earlier
running with execution engine set to mr? If yes, then try and set the
execution engine in hive-site.xml and use a new hive shell.
Have you set all required hive configurations?
If you can still reproduce the issue, then you should create a jira in the
Apache hive project to track the bug.
Bikas
*From:* Archit Thakur [mailto:archit279thakur@gmail.com]
*Sent:* Thursday, May 08, 2014 4:42 AM
*To:* user@hive.apache.org; user@tez.incubator.apache.org
*Subject:* Tez HIve problems.
Hi,
I am facing few problems while using Tez on Hive.
I am able to run Tez through YARN submitting independent MR procesing job.
Also, when I run SELECT query on hive shell by setting
*set hive.execution.engine=mr;*
but setting *mapreduce.framework.name
<http://mapreduce.framework.name>=yarn-tez*
This launches tez job, I checked by seeing AM's UI of machine.
But when I *set hive.execution.engine=tez;*
*It gives me *Query ID =
admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
Total jobs = 1
Launching Job 1 out of 1
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.tez.TezTask
*I checked the hive logs:*
exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Am I doing something wrong? The Tez I am running by setting*
hive.execution.engine=mr;* but setting *mapreduce.framework.name
<http://mapreduce.framework.name>=yarn-tez. is it correct? Does it launches
Tez job or is it MR?*
Thanks and Regards,
Archit Thakur.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Tez HIve problems.
Posted by Hitesh Shah <hi...@apache.org>.
Hi Archit,
I don’t believe apache mailing lists allow attachments. Could you possibly use something like pastebin or any other uploader site and provide a link to the logs?
thanks
— HItesh
On May 14, 2014, at 2:45 AM, Archit Thakur <ar...@gmail.com> wrote:
> forwarding to tez-user.
>
>
> On Wed, May 14, 2014 at 3:14 PM, Archit Thakur <ar...@gmail.com> wrote:
> Hitesh: I have attached application logs.
>
>
>
> On Tue, May 13, 2014 at 8:43 PM, Carter Shanklin <ca...@hortonworks.com> wrote:
> Archit,
>
> I have seen this problem on small sandbox environments with very little memory given to YARN. In those cases, the app master would consume enough memory that there was no memory left to launch containers. Can you confirm that you have enough memory to launch a few containers?
>
>
> On Sun, May 11, 2014 at 9:59 PM, Archit Thakur <ar...@gmail.com> wrote:
> Hi Bikas,
>
> I was able to resolve this issue, by copying hive-exec.jar to hdfs://user/admin/, It was not able to localize it while running Tez.
>
> Though, I am facing some other problem now. My Tez job is not progressing,
>
> My Tez job doesn't progress, it just keeps printing:
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1
>
> In hive logs also, it keeps on printing the same.
>
> Any idea, where should I look into or how should I debug the problem. Or
> pointers on problem itself would help.
> Thanks Archit.
>
>
> On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com> wrote:
> Setting execution engine to mr and framework name to yarn = Hive compiles to MR and runs on MR.
>
> Setting execution engine to mr and framework name to yarn-tez = Hive compiles to MR and runs on Tez.
>
> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>
>
> Did you change the execution engine in the same shell that was earlier running with execution engine set to mr? If yes, then try and set the execution engine in hive-site.xml and use a new hive shell.
>
> Have you set all required hive configurations?
>
>
> If you can still reproduce the issue, then you should create a jira in the Apache hive project to track the bug.
>
>
> Bikas
>
>
> From: Archit Thakur [mailto:archit279thakur@gmail.com]
> Sent: Thursday, May 08, 2014 4:42 AM
> To: user@hive.apache.org; user@tez.incubator.apache.org
> Subject: Tez HIve problems.
>
>
> Hi,
>
> I am facing few problems while using Tez on Hive.
>
> I am able to run Tez through YARN submitting independent MR procesing job.
>
> Also, when I run SELECT query on hive shell by setting
>
> set hive.execution.engine=mr;
>
> but setting mapreduce.framework.name=yarn-tez
>
> This launches tez job, I checked by seeing AM's UI of machine.
>
> But when I set hive.execution.engine=tez;
>
> It gives me
>
> Query ID = admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
> Total jobs = 1
> Launching Job 1 out of 1
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
>
> I checked the hive logs:
>
> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
> at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
> at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
>
> Am I doing something wrong? The Tez I am running by setting hive.execution.engine=mr; but setting mapreduce.framework.name=yarn-tez. is it correct? Does it launches Tez job or is it MR?
>
>
> Thanks and Regards,
>
> Archit Thakur.
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
>
>
>
> --
> Carter Shanklin
> Director, Product Management
> Hortonworks
> (M): +1.650.644.8795 (T): @cshanklin
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
>
RE: Tez HIve problems.
Posted by Bikas Saha <bi...@hortonworks.com>.
Please open a jira on https://issues.apache.org/jira/browse/TEZ and attach
the logs there. Could also please attach a screenshot of the RM web UI page
that shows the containers allocated, containers free etc.
Bikas
*From:* Archit Thakur [mailto:archit279thakur@gmail.com]
*Sent:* Wednesday, May 14, 2014 2:46 AM
*To:* user@tez.incubator.apache.org
*Subject:* Re: Tez HIve problems.
forwarding to tez-user.
On Wed, May 14, 2014 at 3:14 PM, Archit Thakur <ar...@gmail.com>
wrote:
Hitesh: I have attached application logs.
On Tue, May 13, 2014 at 8:43 PM, Carter Shanklin <ca...@hortonworks.com>
wrote:
Archit,
I have seen this problem on small sandbox environments with very little
memory given to YARN. In those cases, the app master would consume enough
memory that there was no memory left to launch containers. Can you confirm
that you have enough memory to launch a few containers?
On Sun, May 11, 2014 at 9:59 PM, Archit Thakur <ar...@gmail.com>
wrote:
Hi Bikas,
I was able to resolve this issue, by copying hive-exec.jar to
hdfs://user/admin/, It was not able to localize it while running Tez.
Though, I am facing some other problem now. My Tez job is not progressing,
My Tez job doesn't progress, it just keeps printing:
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
In hive logs also, it keeps on printing the same.
Any idea, where should I look into or how should I debug the problem. Or
pointers on problem itself would help.
Thanks Archit.
On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com> wrote:
Setting execution engine to mr and framework name to yarn = Hive compiles
to MR and runs on MR.
Setting execution engine to mr and framework name to yarn-tez = Hive
compiles to MR and runs on Tez.
Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
Did you change the execution engine in the same shell that was earlier
running with execution engine set to mr? If yes, then try and set the
execution engine in hive-site.xml and use a new hive shell.
Have you set all required hive configurations?
If you can still reproduce the issue, then you should create a jira in the
Apache hive project to track the bug.
Bikas
*From:* Archit Thakur [mailto:archit279thakur@gmail.com]
*Sent:* Thursday, May 08, 2014 4:42 AM
*To:* user@hive.apache.org; user@tez.incubator.apache.org
*Subject:* Tez HIve problems.
Hi,
I am facing few problems while using Tez on Hive.
I am able to run Tez through YARN submitting independent MR procesing job.
Also, when I run SELECT query on hive shell by setting
*set hive.execution.engine=mr;*
but setting *mapreduce.framework.name
<http://mapreduce.framework.name>=yarn-tez*
This launches tez job, I checked by seeing AM's UI of machine.
But when I *set hive.execution.engine=tez;*
*It gives me *Query ID =
admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
Total jobs = 1
Launching Job 1 out of 1
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.tez.TezTask
*I checked the hive logs:*
exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Am I doing something wrong? The Tez I am running by setting*
hive.execution.engine=mr;* but setting *mapreduce.framework.name
<http://mapreduce.framework.name>=yarn-tez. is it correct? Does it launches
Tez job or is it MR?*
Thanks and Regards,
Archit Thakur.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
--
Carter Shanklin
Director, Product Management
Hortonworks
(M): +1.650.644.8795 (T): @cshanklin <http://twitter.com/cshanklin>
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Tez HIve problems.
Posted by Archit Thakur <ar...@gmail.com>.
forwarding to tez-user.
On Wed, May 14, 2014 at 3:14 PM, Archit Thakur <ar...@gmail.com>wrote:
> Hitesh: I have attached application logs.
>
>
>
> On Tue, May 13, 2014 at 8:43 PM, Carter Shanklin <ca...@hortonworks.com>wrote:
>
>> Archit,
>>
>> I have seen this problem on small sandbox environments with very little
>> memory given to YARN. In those cases, the app master would consume enough
>> memory that there was no memory left to launch containers. Can you confirm
>> that you have enough memory to launch a few containers?
>>
>>
>> On Sun, May 11, 2014 at 9:59 PM, Archit Thakur <archit279thakur@gmail.com
>> > wrote:
>>
>>> Hi Bikas,
>>>
>>> I was able to resolve this issue, by copying hive-exec.jar to
>>> hdfs://user/admin/, It was not able to localize it while running Tez.
>>>
>>> Though, I am facing some other problem now. My Tez job is not
>>> progressing,
>>>
>>> My Tez job doesn't progress, it just keeps printing:
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>>> 4: 0/1
>>>
>>> In hive logs also, it keeps on printing the same.
>>>
>>> Any idea, where should I look into or how should I debug the problem. Or
>>> pointers on problem itself would help.
>>> Thanks Archit.
>>>
>>>
>>> On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com>wrote:
>>>
>>>> Setting execution engine to mr and framework name to yarn = Hive
>>>> compiles to MR and runs on MR.
>>>>
>>>> Setting execution engine to mr and framework name to yarn-tez = Hive
>>>> compiles to MR and runs on Tez.
>>>>
>>>> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>>>>
>>>>
>>>>
>>>> Did you change the execution engine in the same shell that was earlier
>>>> running with execution engine set to mr? If yes, then try and set the
>>>> execution engine in hive-site.xml and use a new hive shell.
>>>>
>>>> Have you set all required hive configurations?
>>>>
>>>>
>>>>
>>>> If you can still reproduce the issue, then you should create a jira in
>>>> the Apache hive project to track the bug.
>>>>
>>>>
>>>>
>>>> Bikas
>>>>
>>>>
>>>>
>>>> *From:* Archit Thakur [mailto:archit279thakur@gmail.com]
>>>> *Sent:* Thursday, May 08, 2014 4:42 AM
>>>> *To:* user@hive.apache.org; user@tez.incubator.apache.org
>>>> *Subject:* Tez HIve problems.
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>> I am facing few problems while using Tez on Hive.
>>>>
>>>> I am able to run Tez through YARN submitting independent MR procesing
>>>> job.
>>>>
>>>> Also, when I run SELECT query on hive shell by setting
>>>>
>>>> *set hive.execution.engine=mr;*
>>>>
>>>> but setting *mapreduce.framework.name
>>>> <http://mapreduce.framework.name>=yarn-tez*
>>>>
>>>> This launches tez job, I checked by seeing AM's UI of machine.
>>>>
>>>> But when I *set hive.execution.engine=tez;*
>>>>
>>>>
>>>>
>>>> *It gives me *Query ID =
>>>> admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
>>>> Total jobs = 1
>>>> Launching Job 1 out of 1
>>>> FAILED: Execution Error, return code 1 from
>>>> org.apache.hadoop.hive.ql.exec.tez.TezTask
>>>>
>>>> *I checked the hive logs:*
>>>>
>>>> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
>>>> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
>>>> at
>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
>>>> at
>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
>>>> at
>>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>>>> at
>>>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
>>>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
>>>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
>>>> at
>>>> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
>>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
>>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
>>>> at
>>>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>>>> at
>>>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>>>> at
>>>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
>>>> at
>>>> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
>>>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
>>>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>>>> Source)
>>>> at java.lang.reflect.Method.invoke(Unknown Source)
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>
>>>> Am I doing something wrong? The Tez I am running by setting*
>>>> hive.execution.engine=mr;* but setting
>>>>
>>>> *mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez.
>>>> is it correct? Does it launches Tez job or is it MR? *
>>>>
>>>> Thanks and Regards,
>>>>
>>>> Archit Thakur.
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>
>>
>> --
>> Carter Shanklin
>> Director, Product Management
>> Hortonworks
>> (M): +1.650.644.8795 (T): @cshanklin <http://twitter.com/cshanklin>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
Re: Tez HIve problems.
Posted by Archit Thakur <ar...@gmail.com>.
Hitesh: I have attached application logs.
On Tue, May 13, 2014 at 8:43 PM, Carter Shanklin <ca...@hortonworks.com>wrote:
> Archit,
>
> I have seen this problem on small sandbox environments with very little
> memory given to YARN. In those cases, the app master would consume enough
> memory that there was no memory left to launch containers. Can you confirm
> that you have enough memory to launch a few containers?
>
>
> On Sun, May 11, 2014 at 9:59 PM, Archit Thakur <ar...@gmail.com>wrote:
>
>> Hi Bikas,
>>
>> I was able to resolve this issue, by copying hive-exec.jar to
>> hdfs://user/admin/, It was not able to localize it while running Tez.
>>
>> Though, I am facing some other problem now. My Tez job is not progressing,
>>
>> My Tez job doesn't progress, it just keeps printing:
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer
>> 4: 0/1
>>
>> In hive logs also, it keeps on printing the same.
>>
>> Any idea, where should I look into or how should I debug the problem. Or
>> pointers on problem itself would help.
>> Thanks Archit.
>>
>>
>> On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com>wrote:
>>
>>> Setting execution engine to mr and framework name to yarn = Hive
>>> compiles to MR and runs on MR.
>>>
>>> Setting execution engine to mr and framework name to yarn-tez = Hive
>>> compiles to MR and runs on Tez.
>>>
>>> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>>>
>>>
>>>
>>> Did you change the execution engine in the same shell that was earlier
>>> running with execution engine set to mr? If yes, then try and set the
>>> execution engine in hive-site.xml and use a new hive shell.
>>>
>>> Have you set all required hive configurations?
>>>
>>>
>>>
>>> If you can still reproduce the issue, then you should create a jira in
>>> the Apache hive project to track the bug.
>>>
>>>
>>>
>>> Bikas
>>>
>>>
>>>
>>> *From:* Archit Thakur [mailto:archit279thakur@gmail.com]
>>> *Sent:* Thursday, May 08, 2014 4:42 AM
>>> *To:* user@hive.apache.org; user@tez.incubator.apache.org
>>> *Subject:* Tez HIve problems.
>>>
>>>
>>>
>>> Hi,
>>>
>>> I am facing few problems while using Tez on Hive.
>>>
>>> I am able to run Tez through YARN submitting independent MR procesing
>>> job.
>>>
>>> Also, when I run SELECT query on hive shell by setting
>>>
>>> *set hive.execution.engine=mr;*
>>>
>>> but setting *mapreduce.framework.name
>>> <http://mapreduce.framework.name>=yarn-tez*
>>>
>>> This launches tez job, I checked by seeing AM's UI of machine.
>>>
>>> But when I *set hive.execution.engine=tez;*
>>>
>>>
>>>
>>> *It gives me *Query ID =
>>> admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
>>> Total jobs = 1
>>> Launching Job 1 out of 1
>>> FAILED: Execution Error, return code 1 from
>>> org.apache.hadoop.hive.ql.exec.tez.TezTask
>>>
>>> *I checked the hive logs:*
>>>
>>> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
>>> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
>>> at
>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
>>> at
>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
>>> at
>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>>> at
>>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
>>> at
>>> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
>>> at
>>> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
>>> at
>>> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
>>> at
>>> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
>>> at
>>> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
>>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
>>> at
>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
>>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
>>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
>>> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
>>> at
>>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>>> at
>>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>>> at
>>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
>>> at
>>> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
>>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
>>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>>> Source)
>>> at java.lang.reflect.Method.invoke(Unknown Source)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>
>>> Am I doing something wrong? The Tez I am running by setting*
>>> hive.execution.engine=mr;* but setting
>>>
>>> *mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez. is
>>> it correct? Does it launches Tez job or is it MR? *
>>>
>>> Thanks and Regards,
>>>
>>> Archit Thakur.
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Carter Shanklin
> Director, Product Management
> Hortonworks
> (M): +1.650.644.8795 (T): @cshanklin <http://twitter.com/cshanklin>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Tez HIve problems.
Posted by Carter Shanklin <ca...@hortonworks.com>.
Archit,
I have seen this problem on small sandbox environments with very little
memory given to YARN. In those cases, the app master would consume enough
memory that there was no memory left to launch containers. Can you confirm
that you have enough memory to launch a few containers?
On Sun, May 11, 2014 at 9:59 PM, Archit Thakur <ar...@gmail.com>wrote:
> Hi Bikas,
>
> I was able to resolve this issue, by copying hive-exec.jar to
> hdfs://user/admin/, It was not able to localize it while running Tez.
>
> Though, I am facing some other problem now. My Tez job is not progressing,
>
> My Tez job doesn't progress, it just keeps printing:
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
>
> In hive logs also, it keeps on printing the same.
>
> Any idea, where should I look into or how should I debug the problem. Or
> pointers on problem itself would help.
> Thanks Archit.
>
>
> On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com> wrote:
>
>> Setting execution engine to mr and framework name to yarn = Hive compiles
>> to MR and runs on MR.
>>
>> Setting execution engine to mr and framework name to yarn-tez = Hive
>> compiles to MR and runs on Tez.
>>
>> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>>
>>
>>
>> Did you change the execution engine in the same shell that was earlier
>> running with execution engine set to mr? If yes, then try and set the
>> execution engine in hive-site.xml and use a new hive shell.
>>
>> Have you set all required hive configurations?
>>
>>
>>
>> If you can still reproduce the issue, then you should create a jira in
>> the Apache hive project to track the bug.
>>
>>
>>
>> Bikas
>>
>>
>>
>> *From:* Archit Thakur [mailto:archit279thakur@gmail.com]
>> *Sent:* Thursday, May 08, 2014 4:42 AM
>> *To:* user@hive.apache.org; user@tez.incubator.apache.org
>> *Subject:* Tez HIve problems.
>>
>>
>>
>> Hi,
>>
>> I am facing few problems while using Tez on Hive.
>>
>> I am able to run Tez through YARN submitting independent MR procesing job.
>>
>> Also, when I run SELECT query on hive shell by setting
>>
>> *set hive.execution.engine=mr;*
>>
>> but setting *mapreduce.framework.name
>> <http://mapreduce.framework.name>=yarn-tez*
>>
>> This launches tez job, I checked by seeing AM's UI of machine.
>>
>> But when I *set hive.execution.engine=tez;*
>>
>>
>>
>> *It gives me *Query ID =
>> admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
>> Total jobs = 1
>> Launching Job 1 out of 1
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.tez.TezTask
>>
>> *I checked the hive logs:*
>>
>> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
>> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
>> at
>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
>> at
>> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
>> at
>> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
>> at
>> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
>> at
>> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
>> at
>> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
>> at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
>> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>> at java.lang.reflect.Method.invoke(Unknown Source)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>
>> Am I doing something wrong? The Tez I am running by setting*
>> hive.execution.engine=mr;* but setting
>>
>> *mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez. is
>> it correct? Does it launches Tez job or is it MR? *
>>
>> Thanks and Regards,
>>
>> Archit Thakur.
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Carter Shanklin
Director, Product Management
Hortonworks
(M): +1.650.644.8795 (T): @cshanklin <http://twitter.com/cshanklin>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Fwd: Tez HIve problems.
Posted by Archit Thakur <ar...@gmail.com>.
Forwarding to dev.
Hi Bikas,
I was able to resolve this issue, by copying hive-exec.jar to
hdfs://user/admin/, It was not able to localize it while running Tez.
Though, I am facing some other problem now. My Tez job is not progressing,
My Tez job doesn't progress, it just keeps printing:
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
In hive logs also, it keeps on printing the same.
Any idea, where should I look into or how should I debug the problem. Or
pointers on problem itself would help.
Thanks Archit.
On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com> wrote:
> Setting execution engine to mr and framework name to yarn = Hive compiles
> to MR and runs on MR.
>
> Setting execution engine to mr and framework name to yarn-tez = Hive
> compiles to MR and runs on Tez.
>
> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>
>
>
> Did you change the execution engine in the same shell that was earlier
> running with execution engine set to mr? If yes, then try and set the
> execution engine in hive-site.xml and use a new hive shell.
>
> Have you set all required hive configurations?
>
>
>
> If you can still reproduce the issue, then you should create a jira in the
> Apache hive project to track the bug.
>
>
>
> Bikas
>
>
>
> *From:* Archit Thakur [mailto:archit279thakur@gmail.com]
> *Sent:* Thursday, May 08, 2014 4:42 AM
> *To:* user@hive.apache.org; user@tez.incubator.apache.org
> *Subject:* Tez HIve problems.
>
>
>
> Hi,
>
> I am facing few problems while using Tez on Hive.
>
> I am able to run Tez through YARN submitting independent MR procesing job.
>
> Also, when I run SELECT query on hive shell by setting
>
> *set hive.execution.engine=mr;*
>
> but setting *mapreduce.framework.name
> <http://mapreduce.framework.name>=yarn-tez*
>
> This launches tez job, I checked by seeing AM's UI of machine.
>
> But when I *set hive.execution.engine=tez;*
>
>
>
> *It gives me *Query ID =
> admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
> Total jobs = 1
> Launching Job 1 out of 1
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
>
> *I checked the hive logs:*
>
> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Am I doing something wrong? The Tez I am running by setting*
> hive.execution.engine=mr;* but setting
>
> *mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez. is
> it correct? Does it launches Tez job or is it MR? *
>
> Thanks and Regards,
>
> Archit Thakur.
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Tez HIve problems.
Posted by Vikram Dixit <vi...@hortonworks.com>.
Hi Archit,
You can take a look at the application logs to see if there are any
failures/stack traces in there. I can also take a look if you could post
the hive.log and application logs to this email or a jira.
Thanks
Vikram.
On Sun, May 11, 2014 at 9:59 PM, Archit Thakur <ar...@gmail.com>wrote:
> Hi Bikas,
>
> I was able to resolve this issue, by copying hive-exec.jar to
> hdfs://user/admin/, It was not able to localize it while running Tez.
>
> Though, I am facing some other problem now. My Tez job is not progressing,
>
> My Tez job doesn't progress, it just keeps printing:
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
> 0/1
>
> In hive logs also, it keeps on printing the same.
>
> Any idea, where should I look into or how should I debug the problem. Or
> pointers on problem itself would help.
> Thanks Archit.
>
>
> On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com> wrote:
>
>> Setting execution engine to mr and framework name to yarn = Hive compiles
>> to MR and runs on MR.
>>
>> Setting execution engine to mr and framework name to yarn-tez = Hive
>> compiles to MR and runs on Tez.
>>
>> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>>
>>
>>
>> Did you change the execution engine in the same shell that was earlier
>> running with execution engine set to mr? If yes, then try and set the
>> execution engine in hive-site.xml and use a new hive shell.
>>
>> Have you set all required hive configurations?
>>
>>
>>
>> If you can still reproduce the issue, then you should create a jira in
>> the Apache hive project to track the bug.
>>
>>
>>
>> Bikas
>>
>>
>>
>> *From:* Archit Thakur [mailto:archit279thakur@gmail.com]
>> *Sent:* Thursday, May 08, 2014 4:42 AM
>> *To:* user@hive.apache.org; user@tez.incubator.apache.org
>> *Subject:* Tez HIve problems.
>>
>>
>>
>> Hi,
>>
>> I am facing few problems while using Tez on Hive.
>>
>> I am able to run Tez through YARN submitting independent MR procesing job.
>>
>> Also, when I run SELECT query on hive shell by setting
>>
>> *set hive.execution.engine=mr;*
>>
>> but setting *mapreduce.framework.name
>> <http://mapreduce.framework.name>=yarn-tez*
>>
>> This launches tez job, I checked by seeing AM's UI of machine.
>>
>> But when I *set hive.execution.engine=tez;*
>>
>>
>>
>> *It gives me *Query ID =
>> admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
>> Total jobs = 1
>> Launching Job 1 out of 1
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.tez.TezTask
>>
>> *I checked the hive logs:*
>>
>> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
>> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
>> at
>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
>> at
>> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
>> at
>> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
>> at
>> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
>> at
>> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
>> at
>> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
>> at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
>> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>> at java.lang.reflect.Method.invoke(Unknown Source)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>
>> Am I doing something wrong? The Tez I am running by setting*
>> hive.execution.engine=mr;* but setting
>>
>> *mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez. is
>> it correct? Does it launches Tez job or is it MR? *
>>
>> Thanks and Regards,
>>
>> Archit Thakur.
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Tez HIve problems.
Posted by Archit Thakur <ar...@gmail.com>.
Hi Bikas,
I was able to resolve this issue, by copying hive-exec.jar to
hdfs://user/admin/, It was not able to localize it while running Tez.
Though, I am facing some other problem now. My Tez job is not progressing,
My Tez job doesn't progress, it just keeps printing:
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
In hive logs also, it keeps on printing the same.
Any idea, where should I look into or how should I debug the problem. Or
pointers on problem itself would help.
Thanks Archit.
On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com> wrote:
> Setting execution engine to mr and framework name to yarn = Hive compiles
> to MR and runs on MR.
>
> Setting execution engine to mr and framework name to yarn-tez = Hive
> compiles to MR and runs on Tez.
>
> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>
>
>
> Did you change the execution engine in the same shell that was earlier
> running with execution engine set to mr? If yes, then try and set the
> execution engine in hive-site.xml and use a new hive shell.
>
> Have you set all required hive configurations?
>
>
>
> If you can still reproduce the issue, then you should create a jira in the
> Apache hive project to track the bug.
>
>
>
> Bikas
>
>
>
> *From:* Archit Thakur [mailto:archit279thakur@gmail.com]
> *Sent:* Thursday, May 08, 2014 4:42 AM
> *To:* user@hive.apache.org; user@tez.incubator.apache.org
> *Subject:* Tez HIve problems.
>
>
>
> Hi,
>
> I am facing few problems while using Tez on Hive.
>
> I am able to run Tez through YARN submitting independent MR procesing job.
>
> Also, when I run SELECT query on hive shell by setting
>
> *set hive.execution.engine=mr;*
>
> but setting *mapreduce.framework.name
> <http://mapreduce.framework.name>=yarn-tez*
>
> This launches tez job, I checked by seeing AM's UI of machine.
>
> But when I *set hive.execution.engine=tez;*
>
>
>
> *It gives me *Query ID =
> admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
> Total jobs = 1
> Launching Job 1 out of 1
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
>
> *I checked the hive logs:*
>
> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Am I doing something wrong? The Tez I am running by setting*
> hive.execution.engine=mr;* but setting
>
> *mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez. is
> it correct? Does it launches Tez job or is it MR? *
>
> Thanks and Regards,
>
> Archit Thakur.
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Tez HIve problems.
Posted by Archit Thakur <ar...@gmail.com>.
Hi Bikas,
I was able to resolve this issue, by copying hive-exec.jar to
hdfs://user/admin/, It was not able to localize it while running Tez.
Though, I am facing some other problem now. My Tez job is not progressing,
My Tez job doesn't progress, it just keeps printing:
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4:
0/1
In hive logs also, it keeps on printing the same.
Any idea, where should I look into or how should I debug the problem. Or
pointers on problem itself would help.
Thanks Archit.
On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bi...@hortonworks.com> wrote:
> Setting execution engine to mr and framework name to yarn = Hive compiles
> to MR and runs on MR.
>
> Setting execution engine to mr and framework name to yarn-tez = Hive
> compiles to MR and runs on Tez.
>
> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>
>
>
> Did you change the execution engine in the same shell that was earlier
> running with execution engine set to mr? If yes, then try and set the
> execution engine in hive-site.xml and use a new hive shell.
>
> Have you set all required hive configurations?
>
>
>
> If you can still reproduce the issue, then you should create a jira in the
> Apache hive project to track the bug.
>
>
>
> Bikas
>
>
>
> *From:* Archit Thakur [mailto:archit279thakur@gmail.com]
> *Sent:* Thursday, May 08, 2014 4:42 AM
> *To:* user@hive.apache.org; user@tez.incubator.apache.org
> *Subject:* Tez HIve problems.
>
>
>
> Hi,
>
> I am facing few problems while using Tez on Hive.
>
> I am able to run Tez through YARN submitting independent MR procesing job.
>
> Also, when I run SELECT query on hive shell by setting
>
> *set hive.execution.engine=mr;*
>
> but setting *mapreduce.framework.name
> <http://mapreduce.framework.name>=yarn-tez*
>
> This launches tez job, I checked by seeing AM's UI of machine.
>
> But when I *set hive.execution.engine=tez;*
>
>
>
> *It gives me *Query ID =
> admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
> Total jobs = 1
> Launching Job 1 out of 1
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
>
> *I checked the hive logs:*
>
> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Am I doing something wrong? The Tez I am running by setting*
> hive.execution.engine=mr;* but setting
>
> *mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez. is
> it correct? Does it launches Tez job or is it MR? *
>
> Thanks and Regards,
>
> Archit Thakur.
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
RE: Tez HIve problems.
Posted by Bikas Saha <bi...@hortonworks.com>.
Setting execution engine to mr and framework name to yarn = Hive compiles
to MR and runs on MR.
Setting execution engine to mr and framework name to yarn-tez = Hive
compiles to MR and runs on Tez.
Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
Did you change the execution engine in the same shell that was earlier
running with execution engine set to mr? If yes, then try and set the
execution engine in hive-site.xml and use a new hive shell.
Have you set all required hive configurations?
If you can still reproduce the issue, then you should create a jira in the
Apache hive project to track the bug.
Bikas
*From:* Archit Thakur [mailto:archit279thakur@gmail.com]
*Sent:* Thursday, May 08, 2014 4:42 AM
*To:* user@hive.apache.org; user@tez.incubator.apache.org
*Subject:* Tez HIve problems.
Hi,
I am facing few problems while using Tez on Hive.
I am able to run Tez through YARN submitting independent MR procesing job.
Also, when I run SELECT query on hive shell by setting
*set hive.execution.engine=mr;*
but setting *mapreduce.framework.name
<http://mapreduce.framework.name>=yarn-tez*
This launches tez job, I checked by seeing AM's UI of machine.
But when I *set hive.execution.engine=tez;*
*It gives me *Query ID =
admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
Total jobs = 1
Launching Job 1 out of 1
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.tez.TezTask
*I checked the hive logs:*
exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Am I doing something wrong? The Tez I am running by setting*
hive.execution.engine=mr;* but setting
*mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez. is it
correct? Does it launches Tez job or is it MR?*
Thanks and Regards,
Archit Thakur.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.