why hive ignore my setting about reduce task number?

[luocanrao <lu...@sohu.com>]

In hive-default.xml I have set reduce task num is 20. But the hive ignore my
setting in second job .

Here is hive log, 

 

Total MapReduce jobs = 2

Launching Job 1 out of 2

Number of reduce tasks not specified. Defaulting to jobconf value of: 20

Launching Job 2 out of 2

Number of reduce tasks determined at compile time: 1

 

The num of reduce tasks in first job is 20.

But the second job ignore my setting, determined by compiler.

How can set all the reduce task to number 20?

Re: why hive ignore my setting about reduce task number?

[Ted Xu <te...@gmail.com>]

Hi luocanrao,

Hive takes reduce number by: first compile time determined number, then user
setting, then default setting.

If the compiler set reduce number to 1, the processing query probably
contains some logic that can not be done by more than 1 reducer, for
example, *sort by* or *cartessian join*.

2010/5/12 luocanrao <lu...@sohu.com>

>  In hive-default.xml I have set reduce task num is 20. But the hive ignore
> my setting in second job .
>
> Here is hive log,
>
>
>
> Total MapReduce jobs = 2
>
> Launching Job 1 out of 2
>
> Number of reduce tasks not specified. Defaulting to jobconf value of: 20
>
> Launching Job 2 out of 2
>
> Number of reduce tasks determined at compile time: 1
>
>
>
> The num of reduce tasks in first job is 20.
>
> But the second job ignore my setting, determined by compiler.
>
> How can set all the reduce task to number 20?
>



-- 
Best Regards,
Ted Xu