You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2018/08/01 17:25:01 UTC
[jira] [Commented] (IMPALA-7234) Non-deterministic majority format
for a table with equal partition instances
[ https://issues.apache.org/jira/browse/IMPALA-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16565670#comment-16565670 ]
ASF subversion and git services commented on IMPALA-7234:
---------------------------------------------------------
Commit 672a271fd0966bd77f38eda9b6f1e768415bac04 in impala's branch refs/heads/master from poojanilangekar
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=672a271 ]
IMPALA-7234: Improve memory estimates produced by the Planner
Previously, the planner used the getMajorityFormat to estimate
the memory requirements of its partitions. Additionally, before
IMPALA-6625 was merged, the majority format for a multi-format
table with no numerical majority was calculated using a HashMap,
thus producing non deterministic results. This change ensures that
the memory estimate is deterministic and always based on partition
that has the maximum memory requirement.
Testing: Ran all PlannerTests. Also, modified plans of scans with
multiple partitions to ensure that the memory estimate produced
corresponds to the partition with the maximum requirement.
Change-Id: I0666ae3d45fbd8615d3fa9a8626ebd29cf94fb4b
Reviewed-on: http://gerrit.cloudera.org:8080/11001
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Non-deterministic majority format for a table with equal partition instances
> -----------------------------------------------------------------------------
>
> Key: IMPALA-7234
> URL: https://issues.apache.org/jira/browse/IMPALA-7234
> Project: IMPALA
> Issue Type: Bug
> Reporter: Pooja Nilangekar
> Assignee: Pooja Nilangekar
> Priority: Major
>
> The getMajorityFormat method of the FeCatalogUtils currently returns non-deterministic results when its argument is a list of partitions where there is no numerical majority in terms of the number of instances. The result is determined by the order in which the partitions are added to the HashMap. We need more deterministic results which also considers the memory requirement among different types of partitions. Ideally, this function should return the format with higher memory requirements in case of a tie.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org