You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/03/24 13:38:00 UTC
[jira] [Work logged] (HIVE-24930) Operator.setDone() short-circuit
from child op is not used in vectorized codepath (childSize == 1)
[ https://issues.apache.org/jira/browse/HIVE-24930?focusedWorklogId=571154&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-571154 ]
ASF GitHub Bot logged work on HIVE-24930:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Mar/21 13:37
Start Date: 24/Mar/21 13:37
Worklog Time Spent: 10m
Work Description: abstractdog opened a new pull request #2113:
URL: https://github.com/apache/hive/pull/2113
### What changes were proposed in this pull request?
Taking child operator's done field into consideration on vectorForward.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 571154)
Remaining Estimate: 0h
Time Spent: 10m
> Operator.setDone() short-circuit from child op is not used in vectorized codepath (childSize == 1)
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-24930
> URL: https://issues.apache.org/jira/browse/HIVE-24930
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This looks like a possible performance regression in case of limit, considering the following query:
> {code}
> explain vectorization detail select
> ws_item_sk item_sk, d_date,
> sum(ws_sales_price) over (partition by ws_item_sk order by d_date range between 10 preceding and current row) cume_sales,
> last_value(ws_sales_price) over (partition by ws_item_sk order by d_date range between 10 preceding and current row) last_price
> from web_sales
> ,date_dim
> where ws_sold_date_sk=d_date_sk
> and d_month_seq between 1214 and 1214+11
> and ws_item_sk is not NULL
> group by ws_item_sk, d_date, ws_sales_price
> limit 100;
> {code}
> in case of vectorized ptf (note: the issue is independent of ptf operator though), the whole pipeline process all the rows, which leads to serious performance regression
> non-vectorized:
> {code}
> set hive.vectorized.execution.ptf.enabled=false;
> ...
> | Select Operator |
> | Statistics: Num rows: 1415172503/1439591782 Data size: 248969569264 Basic stats: COMPLETE Column stats: COMPLETE |
> | PTF Operator |
> | Statistics: Num rows: 1415172503/449131 Data size: 248969569264 Basic stats: COMPLETE Column stats: COMPLETE |
> | Select Operator |
> | Statistics: Num rows: 1415172503/11526 Data size: 565867418560 Basic stats: COMPLETE Column stats: COMPLETE |
> {code}
> vectorized:
> {code}
> set hive.vectorized.execution.ptf.enabled=true;
> ...
> | Select Operator |
> | Statistics: Num rows: 1415172503/1439591782 Data size: 248969569264 Basic stats: COMPLETE Column stats: COMPLETE |
> | PTF Operator |
> | Statistics: Num rows: 1415172503/1439591782 Data size: 248969569264 Basic stats: COMPLETE Column stats: COMPLETE |
> | Select Operator |
> | Statistics: Num rows: 1415172503/1439591782 Data size: 565867418560 Basic stats: COMPLETE Column stats: COMPLETE |
> | File Output Operator |
> | Statistics: Num rows: 100/11300 Data size: 40000 Basic stats: COMPLETE Column stats: COMPLETE |
> {code}
> this is because this short-circuit is missing if childSize==1 (from vectorForward):
> {code}
> // if all children are done, this operator is also done
> if (childrenDone != 0 && childrenDone == childOperatorsArray.length) {
> setDone(true);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)