You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by hadoop n00b <ne...@gmail.com> on 2011/08/11 18:32:26 UTC
Reducer Issue in New Setup
Hello,
We have just setup Hive on a new Hadoop cluster.
When I run a select * on a table, it works fine but when I run any query
which needs a reducer, like count(1) or a where condition, the query just
sits there doing nothing (map 0%). I see some message like no reducers to
run. How do I fix this? The configuration et al is exactly like another
environment that is working fine.
We are running CDH2 with Hadoop 0.20.1 and Hive 0.4.1
Cheers!
Re: Reducer Issue in New Setup
Posted by hadoop n00b <ne...@gmail.com>.
Hi All,
It turned out to be a DNS issue as Loren pointed out. It's working fine now.
Not going for CDH3 right now as this was only a hardware move.
Thanks for your help!
On Thu, Aug 11, 2011 at 10:18 PM, Loren Siebert <lo...@siebert.org> wrote:
> Can you run normal MR jobs, like the example Pi calculation? Sometimes a
> no-reducer problem stems from DNS issues— reducers use node names, not IP
> addresses, so you need to have each machine knows how to resolve the names
> of all the other machines in the cluster.
>
> If it's a new cluster, you may want to look at CDH3, as it is a pretty big
> leap forward.
>
> On Aug 11, 2011, at 9:35 AM, Travis Powell wrote:
>
> Have you checked your logs? These are often the best places to start.****
> Look at the running job and click on the running count, the current task,
> then the task logs.****
> ** **
> Sometimes they’re helpful, sometimes they’re not.****
> ** **
> http://hadoop-master:50030/jobtracker.jsp****
> ** **
> Travis Powell / tpowell@tealeaf.com****
> ** **
> *From:* hadoop n00b [mailto:new2hive@gmail.com]
> *Sent:* Thursday, August 11, 2011 9:32 AM
> *To:* user@hive.apache.org
> *Subject:* Reducer Issue in New Setup****
> ** **
> Hello,****
> ****
> We have just setup Hive on a new Hadoop cluster.****
> ****
> When I run a select * on a table, it works fine but when I run any query
> which needs a reducer, like count(1) or a where condition, the query just
> sits there doing nothing (map 0%). I see some message like no reducers to
> run. How do I fix this? The configuration et al is exactly like another
> environment that is working fine.****
> ****
> We are running CDH2 with Hadoop 0.20.1 and Hive 0.4.1****
> ****
> Cheers!****
>
>
>
Re: Reducer Issue in New Setup
Posted by Loren Siebert <lo...@siebert.org>.
Can you run normal MR jobs, like the example Pi calculation? Sometimes a no-reducer problem stems from DNS issues— reducers use node names, not IP addresses, so you need to have each machine knows how to resolve the names of all the other machines in the cluster.
If it's a new cluster, you may want to look at CDH3, as it is a pretty big leap forward.
On Aug 11, 2011, at 9:35 AM, Travis Powell wrote:
> Have you checked your logs? These are often the best places to start.
> Look at the running job and click on the running count, the current task, then the task logs.
>
> Sometimes they’re helpful, sometimes they’re not.
>
> http://hadoop-master:50030/jobtracker.jsp
>
> Travis Powell / tpowell@tealeaf.com
>
> From: hadoop n00b [mailto:new2hive@gmail.com]
> Sent: Thursday, August 11, 2011 9:32 AM
> To: user@hive.apache.org
> Subject: Reducer Issue in New Setup
>
> Hello,
>
> We have just setup Hive on a new Hadoop cluster.
>
> When I run a select * on a table, it works fine but when I run any query which needs a reducer, like count(1) or a where condition, the query just sits there doing nothing (map 0%). I see some message like no reducers to run. How do I fix this? The configuration et al is exactly like another environment that is working fine.
>
> We are running CDH2 with Hadoop 0.20.1 and Hive 0.4.1
>
> Cheers!
RE: Reducer Issue in New Setup
Posted by Travis Powell <tp...@tealeaf.com>.
Have you checked your logs? These are often the best places to start.
Look at the running job and click on the running count, the current
task, then the task logs.
Sometimes they're helpful, sometimes they're not.
http://hadoop-master:50030/jobtracker.jsp
Travis Powell / tpowell@tealeaf.com
From: hadoop n00b [mailto:new2hive@gmail.com]
Sent: Thursday, August 11, 2011 9:32 AM
To: user@hive.apache.org
Subject: Reducer Issue in New Setup
Hello,
We have just setup Hive on a new Hadoop cluster.
When I run a select * on a table, it works fine but when I run any query
which needs a reducer, like count(1) or a where condition, the query
just sits there doing nothing (map 0%). I see some message like no
reducers to run. How do I fix this? The configuration et al is exactly
like another environment that is working fine.
We are running CDH2 with Hadoop 0.20.1 and Hive 0.4.1
Cheers!
RE: Reducer Issue in New Setup
Posted by "Aggarwal, Vaibhav" <va...@amazon.com>.
Are you using a custom scheduler?
I have seen issues with jobs having 0 mappers and 1 reducer with Fair scheduler.
From: hadoop n00b [mailto:new2hive@gmail.com]
Sent: Thursday, August 11, 2011 9:32 AM
To: user@hive.apache.org
Subject: Reducer Issue in New Setup
Hello,
We have just setup Hive on a new Hadoop cluster.
When I run a select * on a table, it works fine but when I run any query which needs a reducer, like count(1) or a where condition, the query just sits there doing nothing (map 0%). I see some message like no reducers to run. How do I fix this? The configuration et al is exactly like another environment that is working fine.
We are running CDH2 with Hadoop 0.20.1 and Hive 0.4.1
Cheers!