TEZ reducer skew

[Kiran Kolli <Ki...@microsoft.com>]

In HIVE TEZ job 15 task ran for reducer. Due to data skew 2 tasks are running for very long compared to others. 

Looking at the counters REDUCE_INPUT_GROUPS are almost approximately same across reducer tasks. But REDUCE_INPUT_RECORDS of the skewed tasks are like 180 times more than others. How to avoid skew to reducers. 

Thanks,
Kiran 

Re: TEZ reducer skew

[Gopal Vijayaraghavan <go...@apache.org>]

On 1/5/16, 7:51 PM, "Kiran Kolli" <Ki...@microsoft.com> wrote:

>In this case its data-skew. Of all the keys there will be few keys which
>have more records. What extra information do you need?

Zipf distributions have different workarounds for joins, group-bys &
windowing functions.

The query along with something like "explain formatted <query>;" JSON.

More specifically, enough information to plot it out like

http://people.apache.org/~gopalv/q27-plan.svg


Please also collect the runtime info for the automated Tez analyzers
(TEZ-2690).

If you need to run the import tool on an older release, Rajesh has a
backport standalone.

https://github.com/rajeshbalamohan/tez-ats-import


Cheers,
Gopal

RE: TEZ reducer skew

[Kiran Kolli <Ki...@microsoft.com>]

In this case its data-skew. Of all the keys there will be few keys which have more records. What extra information do you need? 


-----Original Message-----
From: Gopal Vijayaraghavan [mailto:gopal@hortonworks.com] On Behalf Of Gopal Vijayaraghavan
Sent: Tuesday, January 5, 2016 5:36 PM
To: user@tez.apache.org
Cc: Kiran Kolli <Ki...@microsoft.com>
Subject: Re: TEZ reducer skew


>Looking at the counters REDUCE_INPUT_GROUPS are almost approximately 
>same across reducer tasks. But REDUCE_INPUT_RECORDS of the skewed tasks 
>are like 180 times more than others. How to avoid skew to reducers.

That really depends. Is the skew a representation of the input or is it an artificially introduced skew due to a query plan.

If your input is skewed (as in, user='' is the same as user=null), then occasionally you can write query fragments which remove such skews before shuffling.

Occasionally, a user will write an incorrect query which produces this as well. For instance,

select sum(sales) from txns tx, (select a.type from accounts where account_date = '2015-12-25') t where tx.type = t.type;

is a skewed query accidentally written by a user.

If the skew is really in the input, Tez (or map-reduce/spark) cannot actually redistribute a skewed key arbitrarily without knowing the semantics of redistribution in the higher level planner.

This problem has many many workarounds, but none of them apply to any other scenario - so please elaborate.

Cheers,
Gopal

Re: TEZ reducer skew

[Gopal Vijayaraghavan <go...@apache.org>]

>Looking at the counters REDUCE_INPUT_GROUPS are almost approximately same
>across reducer tasks. But REDUCE_INPUT_RECORDS of the skewed tasks are
>like 180 times more than others. How to avoid skew to reducers.

That really depends. Is the skew a representation of the input or is it an
artificially introduced skew due to a query plan.

If your input is skewed (as in, user='' is the same as user=null), then
occasionally you can write query fragments which remove such skews before
shuffling.

Occasionally, a user will write an incorrect query which produces this as
well. For instance,

select sum(sales) from txns tx, (select a.type from accounts where
account_date = '2015-12-25') t where tx.type = t.type;

is a skewed query accidentally written by a user.

If the skew is really in the input, Tez (or map-reduce/spark) cannot
actually redistribute a skewed key arbitrarily without knowing the
semantics of redistribution in the higher level planner.

This problem has many many workarounds, but none of them apply to any
other scenario - so please elaborate.

Cheers,
Gopal