You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Lefty Leverenz (JIRA)" <ji...@apache.org> on 2017/09/18 05:20:00 UTC
[jira] [Comment Edited] (HIVE-12492) MapJoin: 4 million unique
integers seems to be a probe plateau
[ https://issues.apache.org/jira/browse/HIVE-12492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887496#comment-15887496 ]
Lefty Leverenz edited comment on HIVE-12492 at 9/18/17 5:19 AM:
----------------------------------------------------------------
Doc note: This adds *hive.auto.convert.join.hashtable.max.entries* to HiveConf.java, so it needs to be documented in the wiki.
* [Configuration Properties -- Query and DDL Execution | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution]
Added a TODOC2.2 label.
Edit (7/Mar/17): HIVE-16137 changes the default value to 40,000,000 so that's what should be documented in the wiki.
Typo alert: In the parameter description, "does not take affect" should be "does not take effect." This can be corrected in the wiki.
Edit (18/Sep/17): HIVE-17276 fixes the typo in the parameter description and HIVE-17521 changes the default value to 21,000,000 (both in release 3.0.0).
was (Author: lefty@hortonworks.com):
Doc note: This adds *hive.auto.convert.join.hashtable.max.entries* to HiveConf.java, so it needs to be documented in the wiki.
* [Configuration Properties -- Query and DDL Execution | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution]
Added a TODOC2.2 label.
Edit (7/Mar/17): HIVE-16137 changes the default value to 40,000,000 so that's what should be documented in the wiki.
Typo alert: In the parameter description, "does not take affect" should be "does not take effect." This can be corrected in the wiki.
> MapJoin: 4 million unique integers seems to be a probe plateau
> --------------------------------------------------------------
>
> Key: HIVE-12492
> URL: https://issues.apache.org/jira/browse/HIVE-12492
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
> Affects Versions: 1.3.0, 1.2.1, 2.0.0
> Reporter: Gopal V
> Assignee: Jesus Camacho Rodriguez
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-12492.01.patch, HIVE-12492.02.patch, HIVE-12492.patch
>
>
> After 4 million keys, the map-join implementation seems to suffer from a performance degradation.
> The hashtable build & probe time makes this very inefficient, even if the data is very compact (i.e 2 ints).
> Falling back onto the shuffle join or bucket map-join is useful after 2^22 items.
> (Note: this fixes a statsutil issue - due to the extra clone() in the column stats path)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)