You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Yekesa Kosuru <vi...@yahoo.com> on 2010/03/18 04:46:06 UTC
hive errors
Hi,
I am a newbie to hiveql. I love what I see but I am having issues with simple hive queries. Below is the transcript. Can someone tell me why reducers going backwards (goes to 67% and then drops to 57%) and everytime I see this, the query finally errors out.
hive> from
> (
> select S.v k, sum (0.5 * e.val * S.r) Val from es e join journal_ratio S
> on (e.k=S.k)
> where e.ca = 1 group by S.v
> ) t
> insert overwrite table temp select t.*;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Defaulting to jobconf value of: 7
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201003132100_0003, Tracking URL = http://dacn0:50030/jobdetails.jsp?jobid=job_201003132100_0003
Kill Command = /usr/local/hadoop/0.20.1/bin/../bin/hadoop job -Dmapred.job.tracker=dacn0:54311 -kill job_201003132100_0003
2010-03-18 05:21:38,618 Stage-1 map = 0%, reduce = 0%
2010-03-18 05:21:56,709 Stage-1 map = 6%, reduce = 0%
2010-03-18 05:21:57,717 Stage-1 map = 9%, reduce = 0%
2010-03-18 05:21:59,731 Stage-1 map = 13%, reduce = 0%
2010-03-18 05:22:00,737 Stage-1 map = 15%, reduce = 0%
2010-03-18 05:22:02,748 Stage-1 map = 19%, reduce = 0%
2010-03-18 05:22:03,755 Stage-1 map = 21%, reduce = 0%
2010-03-18 05:22:05,768 Stage-1 map = 23%, reduce = 0%
2010-03-18 05:22:06,777 Stage-1 map = 28%, reduce = 0%
2010-03-18 05:22:07,785 Stage-1 map = 32%, reduce = 0%
2010-03-18 05:22:09,798 Stage-1 map = 34%, reduce = 0%
2010-03-18 05:22:10,805 Stage-1 map = 37%, reduce = 0%
2010-03-18 05:22:12,820 Stage-1 map = 39%, reduce = 0%
2010-03-18 05:22:13,842 Stage-1 map = 41%, reduce = 0%
2010-03-18 05:22:14,850 Stage-1 map = 43%, reduce = 0%
2010-03-18 05:22:15,858 Stage-1 map = 45%, reduce = 1%
2010-03-18 05:22:17,872 Stage-1 map = 51%, reduce = 2%
2010-03-18 05:22:18,879 Stage-1 map = 53%, reduce = 5%
2010-03-18 05:22:20,893 Stage-1 map = 57%, reduce = 6%
2010-03-18 05:22:21,900 Stage-1 map = 60%, reduce = 8%
2010-03-18 05:22:22,907 Stage-1 map = 60%, reduce = 10%
2010-03-18 05:22:23,915 Stage-1 map = 64%, reduce = 11%
2010-03-18 05:22:24,922 Stage-1 map = 68%, reduce = 12%
2010-03-18 05:22:25,930 Stage-1 map = 70%, reduce = 12%
2010-03-18 05:22:26,937 Stage-1 map = 71%, reduce = 13%
2010-03-18 05:22:27,944 Stage-1 map = 74%, reduce = 13%
2010-03-18 05:22:28,952 Stage-1 map = 77%, reduce = 13%
2010-03-18 05:22:29,960 Stage-1 map = 79%, reduce = 14%
2010-03-18 05:22:30,968 Stage-1 map = 81%, reduce = 14%
2010-03-18 05:22:31,975 Stage-1 map = 83%, reduce = 14%
2010-03-18 05:22:32,983 Stage-1 map = 88%, reduce = 14%
2010-03-18 05:22:33,991 Stage-1 map = 91%, reduce = 17%
2010-03-18 05:22:35,001 Stage-1 map = 92%, reduce = 18%
2010-03-18 05:22:36,009 Stage-1 map = 96%, reduce = 19%
2010-03-18 05:22:37,017 Stage-1 map = 97%, reduce = 19%
2010-03-18 05:22:39,029 Stage-1 map = 99%, reduce = 19%
2010-03-18 05:22:40,037 Stage-1 map = 100%, reduce = 21%
2010-03-18 05:22:42,050 Stage-1 map = 100%, reduce = 23%
2010-03-18 05:22:43,058 Stage-1 map = 100%, reduce = 26%
2010-03-18 05:22:46,076 Stage-1 map = 100%, reduce = 28%
2010-03-18 05:22:48,091 Stage-1 map = 100%, reduce = 30%
2010-03-18 05:22:49,098 Stage-1 map = 100%, reduce = 36%
2010-03-18 05:22:52,117 Stage-1 map = 100%, reduce = 46%
2010-03-18 05:22:54,129 Stage-1 map = 100%, reduce = 52%
2010-03-18 05:22:55,136 Stage-1 map = 100%, reduce = 62%
2010-03-18 05:22:57,148 Stage-1 map = 100%, reduce = 67%
2010-03-18 05:23:51,425 Stage-1 map = 100%, reduce = 57%
after this, it continues
2010-03-18 05:24:03,487 Stage-1 map = 100%, reduce = 61%
2010-03-18 05:24:09,520 Stage-1 map = 100%, reduce = 62%
2010-03-18 05:24:12,539 Stage-1 map = 100%, reduce = 67%
2010-03-18 05:24:21,592 Stage-1 map = 100%, reduce = 68%
2010-03-18 05:25:08,823 Stage-1 map = 100%, reduce = 58%
2010-03-18 05:26:09,126 Stage-1 map = 100%, reduce = 58%
jumps around until it errors out as follows:
2010-03-18 05:38:23,638 Stage-1 map = 100%, reduce = 50%
2010-03-18 05:38:43,786 Stage-1 map = 100%, reduce = 40%
2010-03-18 05:38:46,802 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201003132100_0003 with errors
Failed tasks with most(4) failures :
Task URL: http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver
Hive version = 0.20.1, Java = 1.6
Is this a user error? as in should I add group by, or sort by or cluster by clauses to the query ? Query was run on the master name node, and there are 4 data nodes. With other queries I have seen the map % progresses and drops and eventually errors out.
Any help is appreciated.
Thanks
Vissu
Re: hive errors
Posted by Mafish Liu <ma...@gmail.com>.
You should access the URL with browser, instead of vi.
2010/3/18 Yekesa Kosuru <vi...@yahoo.com>:
> I have this in my error log:
>
>
> vi taskdetails.jsp?jobid=job_201003132100_0003
> <head>
> <link rel="stylesheet" type="text/css" href="/static/hadoop.css">
> <title>Hadoop Task Details</title>
> </head>
> <body>
> <h1>Job <a href="/jobdetails.jsp?jobid=job_201003132100_0003">job_201003132100_0003</a></h1>
>
> <hr>
>
> <h2>All Task Attempts</h2>
> <center>
>
> <h3>No Task Attempts found</h3>
>
>
> <hr>
> <a href="jobdetails.jsp?jobid=job_201003132100_0003">Go back to the job</a><br>
> <a href="jobtracker.jsp">Go back to JobTracker</a><br>
> <hr />
> <a href='http://hadoop.apache.org/core'>Hadoop</a>, 2010.
> </body></html>
>
>
>
> --- On Thu, 3/18/10, Mafish Liu <ma...@gmail.com> wrote:
>
>> From: Mafish Liu <ma...@gmail.com>
>> Subject: Re: hive errors
>> To: hive-user@hadoop.apache.org
>> Date: Thursday, March 18, 2010, 12:01 AM
>> Yekesa:
>>
>> You'r reducer meet some errors while procesing.
>>
>> You should check
>> http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003
>> for more details of the failed reduce task.
>>
>> Please post the error log here for further analyzing.
>>
>> Mafish
>>
>> 2010/3/18 Yekesa Kosuru <vi...@yahoo.com>:
>> > Hi,
>> >
>> > I am a newbie to hiveql. I love what I see but I am
>> having issues with simple hive queries. Below is the
>> transcript. Can someone tell me why reducers going backwards
>> (goes to 67% and then drops to 57%) and everytime I see
>> this, the query finally errors out.
>> >
>> >
>> > hive> from
>> > > (
>> > > select S.v k, sum (0.5 * e.val * S.r) Val
>> from es e join journal_ratio S
>> > > on (e.k=S.k)
>> > > where e.ca = 1 group by S.v
>> > > ) t
>> > > insert overwrite table temp select t.*;
>> > Total MapReduce jobs = 2
>> > Launching Job 1 out of 2
>> > Number of reduce tasks not specified. Defaulting to
>> jobconf value of: 7
>> > In order to change the average load for a reducer (in
>> bytes):
>> > set
>> hive.exec.reducers.bytes.per.reducer=<number>
>> > In order to limit the maximum number of reducers:
>> > set hive.exec.reducers.max=<number>
>> > In order to set a constant number of reducers:
>> > set mapred.reduce.tasks=<number>
>> > Starting Job = job_201003132100_0003, Tracking URL =
>> http://dacn0:50030/jobdetails.jsp?jobid=job_201003132100_0003
>> > Kill Command =
>> /usr/local/hadoop/0.20.1/bin/../bin/hadoop job
>> -Dmapred.job.tracker=dacn0:54311 -kill
>> job_201003132100_0003
>> > 2010-03-18 05:21:38,618 Stage-1 map = 0%, reduce =
>> 0%
>> > 2010-03-18 05:21:56,709 Stage-1 map = 6%, reduce =
>> 0%
>> > 2010-03-18 05:21:57,717 Stage-1 map = 9%, reduce =
>> 0%
>> > 2010-03-18 05:21:59,731 Stage-1 map = 13%, reduce =
>> 0%
>> > 2010-03-18 05:22:00,737 Stage-1 map = 15%, reduce =
>> 0%
>> > 2010-03-18 05:22:02,748 Stage-1 map = 19%, reduce =
>> 0%
>> > 2010-03-18 05:22:03,755 Stage-1 map = 21%, reduce =
>> 0%
>> > 2010-03-18 05:22:05,768 Stage-1 map = 23%, reduce =
>> 0%
>> > 2010-03-18 05:22:06,777 Stage-1 map = 28%, reduce =
>> 0%
>> > 2010-03-18 05:22:07,785 Stage-1 map = 32%, reduce =
>> 0%
>> > 2010-03-18 05:22:09,798 Stage-1 map = 34%, reduce =
>> 0%
>> > 2010-03-18 05:22:10,805 Stage-1 map = 37%, reduce =
>> 0%
>> > 2010-03-18 05:22:12,820 Stage-1 map = 39%, reduce =
>> 0%
>> > 2010-03-18 05:22:13,842 Stage-1 map = 41%, reduce =
>> 0%
>> > 2010-03-18 05:22:14,850 Stage-1 map = 43%, reduce =
>> 0%
>> > 2010-03-18 05:22:15,858 Stage-1 map = 45%, reduce =
>> 1%
>> > 2010-03-18 05:22:17,872 Stage-1 map = 51%, reduce =
>> 2%
>> > 2010-03-18 05:22:18,879 Stage-1 map = 53%, reduce =
>> 5%
>> > 2010-03-18 05:22:20,893 Stage-1 map = 57%, reduce =
>> 6%
>> > 2010-03-18 05:22:21,900 Stage-1 map = 60%, reduce =
>> 8%
>> > 2010-03-18 05:22:22,907 Stage-1 map = 60%, reduce =
>> 10%
>> > 2010-03-18 05:22:23,915 Stage-1 map = 64%, reduce =
>> 11%
>> > 2010-03-18 05:22:24,922 Stage-1 map = 68%, reduce =
>> 12%
>> > 2010-03-18 05:22:25,930 Stage-1 map = 70%, reduce =
>> 12%
>> > 2010-03-18 05:22:26,937 Stage-1 map = 71%, reduce =
>> 13%
>> > 2010-03-18 05:22:27,944 Stage-1 map = 74%, reduce =
>> 13%
>> > 2010-03-18 05:22:28,952 Stage-1 map = 77%, reduce =
>> 13%
>> > 2010-03-18 05:22:29,960 Stage-1 map = 79%, reduce =
>> 14%
>> > 2010-03-18 05:22:30,968 Stage-1 map = 81%, reduce =
>> 14%
>> > 2010-03-18 05:22:31,975 Stage-1 map = 83%, reduce =
>> 14%
>> > 2010-03-18 05:22:32,983 Stage-1 map = 88%, reduce =
>> 14%
>> > 2010-03-18 05:22:33,991 Stage-1 map = 91%, reduce =
>> 17%
>> > 2010-03-18 05:22:35,001 Stage-1 map = 92%, reduce =
>> 18%
>> > 2010-03-18 05:22:36,009 Stage-1 map = 96%, reduce =
>> 19%
>> > 2010-03-18 05:22:37,017 Stage-1 map = 97%, reduce =
>> 19%
>> > 2010-03-18 05:22:39,029 Stage-1 map = 99%, reduce =
>> 19%
>> > 2010-03-18 05:22:40,037 Stage-1 map = 100%, reduce =
>> 21%
>> > 2010-03-18 05:22:42,050 Stage-1 map = 100%, reduce =
>> 23%
>> > 2010-03-18 05:22:43,058 Stage-1 map = 100%, reduce =
>> 26%
>> > 2010-03-18 05:22:46,076 Stage-1 map = 100%, reduce =
>> 28%
>> > 2010-03-18 05:22:48,091 Stage-1 map = 100%, reduce =
>> 30%
>> > 2010-03-18 05:22:49,098 Stage-1 map = 100%, reduce =
>> 36%
>> > 2010-03-18 05:22:52,117 Stage-1 map = 100%, reduce =
>> 46%
>> > 2010-03-18 05:22:54,129 Stage-1 map = 100%, reduce =
>> 52%
>> > 2010-03-18 05:22:55,136 Stage-1 map = 100%, reduce =
>> 62%
>> > 2010-03-18 05:22:57,148 Stage-1 map = 100%, reduce =
>> 67%
>> > 2010-03-18 05:23:51,425 Stage-1 map = 100%, reduce =
>> 57%
>> >
>> > after this, it continues
>> >
>> > 2010-03-18 05:24:03,487 Stage-1 map = 100%, reduce =
>> 61%
>> > 2010-03-18 05:24:09,520 Stage-1 map = 100%, reduce =
>> 62%
>> > 2010-03-18 05:24:12,539 Stage-1 map = 100%, reduce =
>> 67%
>> > 2010-03-18 05:24:21,592 Stage-1 map = 100%, reduce =
>> 68%
>> > 2010-03-18 05:25:08,823 Stage-1 map = 100%, reduce =
>> 58%
>> > 2010-03-18 05:26:09,126 Stage-1 map = 100%, reduce =
>> 58%
>> >
>> > jumps around until it errors out as follows:
>> >
>> > 2010-03-18 05:38:23,638 Stage-1 map = 100%, reduce =
>> 50%
>> > 2010-03-18 05:38:43,786 Stage-1 map = 100%, reduce =
>> 40%
>> > 2010-03-18 05:38:46,802 Stage-1 map = 100%, reduce =
>> 100%
>> > Ended Job = job_201003132100_0003 with errors
>> >
>> > Failed tasks with most(4) failures :
>> > Task URL: http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003
>> >
>> > FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.ExecDriver
>> >
>> >
>> > Hive version = 0.20.1, Java = 1.6
>> >
>> > Is this a user error? as in should I add group by, or
>> sort by or cluster by clauses to the query ? Query was run
>> on the master name node, and there are 4 data nodes. With
>> other queries I have seen the map % progresses and drops and
>> eventually errors out.
>> >
>> > Any help is appreciated.
>> >
>> > Thanks
>> >
>> > Vissu
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Mafish@gmail.com
>>
>
>
>
>
--
Mafish@gmail.com
Re: hive errors
Posted by Yekesa Kosuru <vi...@yahoo.com>.
I have this in my error log:
vi taskdetails.jsp?jobid=job_201003132100_0003
<head>
<link rel="stylesheet" type="text/css" href="/static/hadoop.css">
<title>Hadoop Task Details</title>
</head>
<body>
<h1>Job <a href="/jobdetails.jsp?jobid=job_201003132100_0003">job_201003132100_0003</a></h1>
<hr>
<h2>All Task Attempts</h2>
<center>
<h3>No Task Attempts found</h3>
<hr>
<a href="jobdetails.jsp?jobid=job_201003132100_0003">Go back to the job</a><br>
<a href="jobtracker.jsp">Go back to JobTracker</a><br>
<hr />
<a href='http://hadoop.apache.org/core'>Hadoop</a>, 2010.
</body></html>
--- On Thu, 3/18/10, Mafish Liu <ma...@gmail.com> wrote:
> From: Mafish Liu <ma...@gmail.com>
> Subject: Re: hive errors
> To: hive-user@hadoop.apache.org
> Date: Thursday, March 18, 2010, 12:01 AM
> Yekesa:
>
> You'r reducer meet some errors while procesing.
>
> You should check
> http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003
> for more details of the failed reduce task.
>
> Please post the error log here for further analyzing.
>
> Mafish
>
> 2010/3/18 Yekesa Kosuru <vi...@yahoo.com>:
> > Hi,
> >
> > I am a newbie to hiveql. I love what I see but I am
> having issues with simple hive queries. Below is the
> transcript. Can someone tell me why reducers going backwards
> (goes to 67% and then drops to 57%) and everytime I see
> this, the query finally errors out.
> >
> >
> > hive> from
> > > (
> > > select S.v k, sum (0.5 * e.val * S.r) Val
> from es e join journal_ratio S
> > > on (e.k=S.k)
> > > where e.ca = 1 group by S.v
> > > ) t
> > > insert overwrite table temp select t.*;
> > Total MapReduce jobs = 2
> > Launching Job 1 out of 2
> > Number of reduce tasks not specified. Defaulting to
> jobconf value of: 7
> > In order to change the average load for a reducer (in
> bytes):
> > set
> hive.exec.reducers.bytes.per.reducer=<number>
> > In order to limit the maximum number of reducers:
> > set hive.exec.reducers.max=<number>
> > In order to set a constant number of reducers:
> > set mapred.reduce.tasks=<number>
> > Starting Job = job_201003132100_0003, Tracking URL =
> http://dacn0:50030/jobdetails.jsp?jobid=job_201003132100_0003
> > Kill Command =
> /usr/local/hadoop/0.20.1/bin/../bin/hadoop job
> -Dmapred.job.tracker=dacn0:54311 -kill
> job_201003132100_0003
> > 2010-03-18 05:21:38,618 Stage-1 map = 0%, reduce =
> 0%
> > 2010-03-18 05:21:56,709 Stage-1 map = 6%, reduce =
> 0%
> > 2010-03-18 05:21:57,717 Stage-1 map = 9%, reduce =
> 0%
> > 2010-03-18 05:21:59,731 Stage-1 map = 13%, reduce =
> 0%
> > 2010-03-18 05:22:00,737 Stage-1 map = 15%, reduce =
> 0%
> > 2010-03-18 05:22:02,748 Stage-1 map = 19%, reduce =
> 0%
> > 2010-03-18 05:22:03,755 Stage-1 map = 21%, reduce =
> 0%
> > 2010-03-18 05:22:05,768 Stage-1 map = 23%, reduce =
> 0%
> > 2010-03-18 05:22:06,777 Stage-1 map = 28%, reduce =
> 0%
> > 2010-03-18 05:22:07,785 Stage-1 map = 32%, reduce =
> 0%
> > 2010-03-18 05:22:09,798 Stage-1 map = 34%, reduce =
> 0%
> > 2010-03-18 05:22:10,805 Stage-1 map = 37%, reduce =
> 0%
> > 2010-03-18 05:22:12,820 Stage-1 map = 39%, reduce =
> 0%
> > 2010-03-18 05:22:13,842 Stage-1 map = 41%, reduce =
> 0%
> > 2010-03-18 05:22:14,850 Stage-1 map = 43%, reduce =
> 0%
> > 2010-03-18 05:22:15,858 Stage-1 map = 45%, reduce =
> 1%
> > 2010-03-18 05:22:17,872 Stage-1 map = 51%, reduce =
> 2%
> > 2010-03-18 05:22:18,879 Stage-1 map = 53%, reduce =
> 5%
> > 2010-03-18 05:22:20,893 Stage-1 map = 57%, reduce =
> 6%
> > 2010-03-18 05:22:21,900 Stage-1 map = 60%, reduce =
> 8%
> > 2010-03-18 05:22:22,907 Stage-1 map = 60%, reduce =
> 10%
> > 2010-03-18 05:22:23,915 Stage-1 map = 64%, reduce =
> 11%
> > 2010-03-18 05:22:24,922 Stage-1 map = 68%, reduce =
> 12%
> > 2010-03-18 05:22:25,930 Stage-1 map = 70%, reduce =
> 12%
> > 2010-03-18 05:22:26,937 Stage-1 map = 71%, reduce =
> 13%
> > 2010-03-18 05:22:27,944 Stage-1 map = 74%, reduce =
> 13%
> > 2010-03-18 05:22:28,952 Stage-1 map = 77%, reduce =
> 13%
> > 2010-03-18 05:22:29,960 Stage-1 map = 79%, reduce =
> 14%
> > 2010-03-18 05:22:30,968 Stage-1 map = 81%, reduce =
> 14%
> > 2010-03-18 05:22:31,975 Stage-1 map = 83%, reduce =
> 14%
> > 2010-03-18 05:22:32,983 Stage-1 map = 88%, reduce =
> 14%
> > 2010-03-18 05:22:33,991 Stage-1 map = 91%, reduce =
> 17%
> > 2010-03-18 05:22:35,001 Stage-1 map = 92%, reduce =
> 18%
> > 2010-03-18 05:22:36,009 Stage-1 map = 96%, reduce =
> 19%
> > 2010-03-18 05:22:37,017 Stage-1 map = 97%, reduce =
> 19%
> > 2010-03-18 05:22:39,029 Stage-1 map = 99%, reduce =
> 19%
> > 2010-03-18 05:22:40,037 Stage-1 map = 100%, reduce =
> 21%
> > 2010-03-18 05:22:42,050 Stage-1 map = 100%, reduce =
> 23%
> > 2010-03-18 05:22:43,058 Stage-1 map = 100%, reduce =
> 26%
> > 2010-03-18 05:22:46,076 Stage-1 map = 100%, reduce =
> 28%
> > 2010-03-18 05:22:48,091 Stage-1 map = 100%, reduce =
> 30%
> > 2010-03-18 05:22:49,098 Stage-1 map = 100%, reduce =
> 36%
> > 2010-03-18 05:22:52,117 Stage-1 map = 100%, reduce =
> 46%
> > 2010-03-18 05:22:54,129 Stage-1 map = 100%, reduce =
> 52%
> > 2010-03-18 05:22:55,136 Stage-1 map = 100%, reduce =
> 62%
> > 2010-03-18 05:22:57,148 Stage-1 map = 100%, reduce =
> 67%
> > 2010-03-18 05:23:51,425 Stage-1 map = 100%, reduce =
> 57%
> >
> > after this, it continues
> >
> > 2010-03-18 05:24:03,487 Stage-1 map = 100%, reduce =
> 61%
> > 2010-03-18 05:24:09,520 Stage-1 map = 100%, reduce =
> 62%
> > 2010-03-18 05:24:12,539 Stage-1 map = 100%, reduce =
> 67%
> > 2010-03-18 05:24:21,592 Stage-1 map = 100%, reduce =
> 68%
> > 2010-03-18 05:25:08,823 Stage-1 map = 100%, reduce =
> 58%
> > 2010-03-18 05:26:09,126 Stage-1 map = 100%, reduce =
> 58%
> >
> > jumps around until it errors out as follows:
> >
> > 2010-03-18 05:38:23,638 Stage-1 map = 100%, reduce =
> 50%
> > 2010-03-18 05:38:43,786 Stage-1 map = 100%, reduce =
> 40%
> > 2010-03-18 05:38:46,802 Stage-1 map = 100%, reduce =
> 100%
> > Ended Job = job_201003132100_0003 with errors
> >
> > Failed tasks with most(4) failures :
> > Task URL: http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003
> >
> > FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.ExecDriver
> >
> >
> > Hive version = 0.20.1, Java = 1.6
> >
> > Is this a user error? as in should I add group by, or
> sort by or cluster by clauses to the query ? Query was run
> on the master name node, and there are 4 data nodes. With
> other queries I have seen the map % progresses and drops and
> eventually errors out.
> >
> > Any help is appreciated.
> >
> > Thanks
> >
> > Vissu
> >
> >
> >
> >
>
>
>
> --
> Mafish@gmail.com
>
Re: hive errors
Posted by Mafish Liu <ma...@gmail.com>.
Yekesa:
You'r reducer meet some errors while procesing.
You should check
http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003
for more details of the failed reduce task.
Please post the error log here for further analyzing.
Mafish
2010/3/18 Yekesa Kosuru <vi...@yahoo.com>:
> Hi,
>
> I am a newbie to hiveql. I love what I see but I am having issues with simple hive queries. Below is the transcript. Can someone tell me why reducers going backwards (goes to 67% and then drops to 57%) and everytime I see this, the query finally errors out.
>
>
> hive> from
> > (
> > select S.v k, sum (0.5 * e.val * S.r) Val from es e join journal_ratio S
> > on (e.k=S.k)
> > where e.ca = 1 group by S.v
> > ) t
> > insert overwrite table temp select t.*;
> Total MapReduce jobs = 2
> Launching Job 1 out of 2
> Number of reduce tasks not specified. Defaulting to jobconf value of: 7
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=<number>
> Starting Job = job_201003132100_0003, Tracking URL = http://dacn0:50030/jobdetails.jsp?jobid=job_201003132100_0003
> Kill Command = /usr/local/hadoop/0.20.1/bin/../bin/hadoop job -Dmapred.job.tracker=dacn0:54311 -kill job_201003132100_0003
> 2010-03-18 05:21:38,618 Stage-1 map = 0%, reduce = 0%
> 2010-03-18 05:21:56,709 Stage-1 map = 6%, reduce = 0%
> 2010-03-18 05:21:57,717 Stage-1 map = 9%, reduce = 0%
> 2010-03-18 05:21:59,731 Stage-1 map = 13%, reduce = 0%
> 2010-03-18 05:22:00,737 Stage-1 map = 15%, reduce = 0%
> 2010-03-18 05:22:02,748 Stage-1 map = 19%, reduce = 0%
> 2010-03-18 05:22:03,755 Stage-1 map = 21%, reduce = 0%
> 2010-03-18 05:22:05,768 Stage-1 map = 23%, reduce = 0%
> 2010-03-18 05:22:06,777 Stage-1 map = 28%, reduce = 0%
> 2010-03-18 05:22:07,785 Stage-1 map = 32%, reduce = 0%
> 2010-03-18 05:22:09,798 Stage-1 map = 34%, reduce = 0%
> 2010-03-18 05:22:10,805 Stage-1 map = 37%, reduce = 0%
> 2010-03-18 05:22:12,820 Stage-1 map = 39%, reduce = 0%
> 2010-03-18 05:22:13,842 Stage-1 map = 41%, reduce = 0%
> 2010-03-18 05:22:14,850 Stage-1 map = 43%, reduce = 0%
> 2010-03-18 05:22:15,858 Stage-1 map = 45%, reduce = 1%
> 2010-03-18 05:22:17,872 Stage-1 map = 51%, reduce = 2%
> 2010-03-18 05:22:18,879 Stage-1 map = 53%, reduce = 5%
> 2010-03-18 05:22:20,893 Stage-1 map = 57%, reduce = 6%
> 2010-03-18 05:22:21,900 Stage-1 map = 60%, reduce = 8%
> 2010-03-18 05:22:22,907 Stage-1 map = 60%, reduce = 10%
> 2010-03-18 05:22:23,915 Stage-1 map = 64%, reduce = 11%
> 2010-03-18 05:22:24,922 Stage-1 map = 68%, reduce = 12%
> 2010-03-18 05:22:25,930 Stage-1 map = 70%, reduce = 12%
> 2010-03-18 05:22:26,937 Stage-1 map = 71%, reduce = 13%
> 2010-03-18 05:22:27,944 Stage-1 map = 74%, reduce = 13%
> 2010-03-18 05:22:28,952 Stage-1 map = 77%, reduce = 13%
> 2010-03-18 05:22:29,960 Stage-1 map = 79%, reduce = 14%
> 2010-03-18 05:22:30,968 Stage-1 map = 81%, reduce = 14%
> 2010-03-18 05:22:31,975 Stage-1 map = 83%, reduce = 14%
> 2010-03-18 05:22:32,983 Stage-1 map = 88%, reduce = 14%
> 2010-03-18 05:22:33,991 Stage-1 map = 91%, reduce = 17%
> 2010-03-18 05:22:35,001 Stage-1 map = 92%, reduce = 18%
> 2010-03-18 05:22:36,009 Stage-1 map = 96%, reduce = 19%
> 2010-03-18 05:22:37,017 Stage-1 map = 97%, reduce = 19%
> 2010-03-18 05:22:39,029 Stage-1 map = 99%, reduce = 19%
> 2010-03-18 05:22:40,037 Stage-1 map = 100%, reduce = 21%
> 2010-03-18 05:22:42,050 Stage-1 map = 100%, reduce = 23%
> 2010-03-18 05:22:43,058 Stage-1 map = 100%, reduce = 26%
> 2010-03-18 05:22:46,076 Stage-1 map = 100%, reduce = 28%
> 2010-03-18 05:22:48,091 Stage-1 map = 100%, reduce = 30%
> 2010-03-18 05:22:49,098 Stage-1 map = 100%, reduce = 36%
> 2010-03-18 05:22:52,117 Stage-1 map = 100%, reduce = 46%
> 2010-03-18 05:22:54,129 Stage-1 map = 100%, reduce = 52%
> 2010-03-18 05:22:55,136 Stage-1 map = 100%, reduce = 62%
> 2010-03-18 05:22:57,148 Stage-1 map = 100%, reduce = 67%
> 2010-03-18 05:23:51,425 Stage-1 map = 100%, reduce = 57%
>
> after this, it continues
>
> 2010-03-18 05:24:03,487 Stage-1 map = 100%, reduce = 61%
> 2010-03-18 05:24:09,520 Stage-1 map = 100%, reduce = 62%
> 2010-03-18 05:24:12,539 Stage-1 map = 100%, reduce = 67%
> 2010-03-18 05:24:21,592 Stage-1 map = 100%, reduce = 68%
> 2010-03-18 05:25:08,823 Stage-1 map = 100%, reduce = 58%
> 2010-03-18 05:26:09,126 Stage-1 map = 100%, reduce = 58%
>
> jumps around until it errors out as follows:
>
> 2010-03-18 05:38:23,638 Stage-1 map = 100%, reduce = 50%
> 2010-03-18 05:38:43,786 Stage-1 map = 100%, reduce = 40%
> 2010-03-18 05:38:46,802 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_201003132100_0003 with errors
>
> Failed tasks with most(4) failures :
> Task URL: http://dacn0:50030/taskdetails.jsp?jobid=job_201003132100_0003&tipid=task_201003132100_0003_r_000003
>
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver
>
>
> Hive version = 0.20.1, Java = 1.6
>
> Is this a user error? as in should I add group by, or sort by or cluster by clauses to the query ? Query was run on the master name node, and there are 4 data nodes. With other queries I have seen the map % progresses and drops and eventually errors out.
>
> Any help is appreciated.
>
> Thanks
>
> Vissu
>
>
>
>
--
Mafish@gmail.com