回复: Re: hive on tez not convert map join to broadcast join

["r7raul1984@163.com" <r7...@163.com>]

set hive.mapjoin.smalltable.filesize   is no effect fot   hive tez?



r7raul1984@163.com
 
发件人： Vikram Dixit
发送时间： 2015-05-13 09:26
收件人： user
主题： Re: hive on tez not convert map join to broadcast join
From the explain output, it looks like the small table is of size 57 MB whereas the no conditional task size is set to 10MB. If you bump up the no conditional task size to be larger than 57MB, tez would convert the join to a map join.

Thanks
Vikram.

From: "r7raul1984@163.com" <r7...@163.com>
Reply-To: user <us...@tez.apache.org>
Date: Tuesday, May 12, 2015 at 6:21 PM
To: user <us...@tez.apache.org>
Subject: Re: Re: hive on tez not convert map join to broadcast join

I usehive 1.1.0 + tez0.53

set hive.mapjoin.smalltable.filesize = 60000000; 
set hive.auto.convert.join=true; 
set hive.auto.convert.join.noconditionaltask=true 
set hive.auto.convert.join.noconditionaltask.size=10000000




r7raul1984@163.com
 
From: Vikram Dixit
Date: 2015-05-13 08:47
To: user
Subject: Re: hive on tez not convert map join to broadcast join
Hi,

Can you share your configuration values for the following when you run in tez:

hive.auto.convert.join
hive.auto.convert.join.noconditionaltask
hive.auto.convert.join.noconditionaltask.size

What are the sizes of the tables? Stored in which format? Size of the tables and size of the tables on disc?

Thanks
Vikram.

From: "r7raul1984@163.com" <r7...@163.com>
Reply-To: user <us...@tez.apache.org>
Date: Tuesday, May 12, 2015 at 5:42 PM
To: user <us...@tez.apache.org>
Subject: hive on tez not convert map join to broadcast join

In MR query plan is 

Map Join Operator 
condition map: 
Left Outer Join0 to 1 
keys: 
0 ordr_code (type: string), cart_prod_id (type: bigint) 
1 parnt_ordr_code (type: string), comb_prod_id (type: bigint) 
outputColumnNames: _col1, _col2, _col3, _col5, _col10, _col11, _col15, _col16, 

But in tez 

 Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE)     No  broadcast edge 
Reducer 3 <- Map 5 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)

Merge Join Operator 
condition map: 
Left Outer Join0 to 1 
keys: 
0 ordr_code (type: string), cart_prod_id (type: bigint) 
1 parnt_ordr_code (type: string), comb_prod_id (type: bigint)



r7raul1984@163.com