You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/06/19 13:26:21 UTC
[GitHub] [hive] pgaref opened a new pull request #1152: HIVE-23730: TezCompiler tracking the original TS key columnName
pgaref opened a new pull request #1152:
URL: https://github.com/apache/hive/pull/1152
TezCompiler tracking the original TS key columnName used for MJ probedecode
Change-Id: Ia1f42f36bb6ea7108628d99f306d3c74398dca6b
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HIVE-XXXXX: Fix a typo in YYY)
For more details, please see https://cwiki.apache.org/confluence/display/Hive/HowToContribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pgaref commented on a change in pull request #1152: HIVE-23730: TezCompiler tracking the original TS key columnName
Posted by GitBox <gi...@apache.org>.
pgaref commented on a change in pull request #1152:
URL: https://github.com/apache/hive/pull/1152#discussion_r443479460
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
##########
@@ -1566,13 +1569,38 @@ private void removeSemijoinsParallelToMapJoin(OptimizeTezProcContext procCtx)
List<ExprNodeDesc> keyDesc = selectedMJOp.getConf().getKeys().get(posBigTable);
ExprNodeColumnDesc keyCol = (ExprNodeColumnDesc) keyDesc.get(0);
-
- tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
- keyCol.getColumn(), selectedMJOpRatio);
+ String realTSColName = getOriginalTSColName(selectedMJOp, keyCol.getColumn());
+ if (realTSColName != null) {
+ tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
+ realTSColName, selectedMJOpRatio);
+ } else {
+ LOG.warn("ProbeDecode could not find TSColName for ColKey {} with MJ Schema {} ", keyCol, selectedMJOp.getSchema());
+ }
}
return tsProbeDecodeCtx;
}
+ private static String getOriginalTSColName(MapJoinOperator mjOp, String internalCoName) {
Review comment:
Utility method is now moved to operatorUtils
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] jcamachor merged pull request #1152: HIVE-23730: TezCompiler tracking the original TS key columnName
Posted by GitBox <gi...@apache.org>.
jcamachor merged pull request #1152:
URL: https://github.com/apache/hive/pull/1152
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] jcamachor commented on a change in pull request #1152: HIVE-23730: TezCompiler tracking the original TS key columnName
Posted by GitBox <gi...@apache.org>.
jcamachor commented on a change in pull request #1152:
URL: https://github.com/apache/hive/pull/1152#discussion_r443970247
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
##########
@@ -1566,13 +1569,38 @@ private void removeSemijoinsParallelToMapJoin(OptimizeTezProcContext procCtx)
List<ExprNodeDesc> keyDesc = selectedMJOp.getConf().getKeys().get(posBigTable);
ExprNodeColumnDesc keyCol = (ExprNodeColumnDesc) keyDesc.get(0);
-
- tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
- keyCol.getColumn(), selectedMJOpRatio);
+ String realTSColName = getOriginalTSColName(selectedMJOp, keyCol.getColumn());
+ if (realTSColName != null) {
+ tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
+ realTSColName, selectedMJOpRatio);
+ } else {
+ LOG.warn("ProbeDecode could not find TSColName for ColKey {} with MJ Schema {} ", keyCol, selectedMJOp.getSchema());
Review comment:
Cool! I think we should enable it by default indeed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pgaref commented on a change in pull request #1152: HIVE-23730: TezCompiler tracking the original TS key columnName
Posted by GitBox <gi...@apache.org>.
pgaref commented on a change in pull request #1152:
URL: https://github.com/apache/hive/pull/1152#discussion_r443617251
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
##########
@@ -1566,13 +1569,38 @@ private void removeSemijoinsParallelToMapJoin(OptimizeTezProcContext procCtx)
List<ExprNodeDesc> keyDesc = selectedMJOp.getConf().getKeys().get(posBigTable);
ExprNodeColumnDesc keyCol = (ExprNodeColumnDesc) keyDesc.get(0);
-
- tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
- keyCol.getColumn(), selectedMJOpRatio);
+ String realTSColName = getOriginalTSColName(selectedMJOp, keyCol.getColumn());
+ if (realTSColName != null) {
+ tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
+ realTSColName, selectedMJOpRatio);
+ } else {
+ LOG.warn("ProbeDecode could not find TSColName for ColKey {} with MJ Schema {} ", keyCol, selectedMJOp.getSchema());
Review comment:
Qtest results here:
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-1152/4/tests/
Seems that for for existing MJ ops the probedecode optimisation works fine (properly finds original TS col alias as well). Not sure if we want to enable probe by default however. Thoughts? cc @ashutoshc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] jcamachor commented on a change in pull request #1152: HIVE-23730: TezCompiler tracking the original TS key columnName
Posted by GitBox <gi...@apache.org>.
jcamachor commented on a change in pull request #1152:
URL: https://github.com/apache/hive/pull/1152#discussion_r443271395
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
##########
@@ -1566,13 +1569,38 @@ private void removeSemijoinsParallelToMapJoin(OptimizeTezProcContext procCtx)
List<ExprNodeDesc> keyDesc = selectedMJOp.getConf().getKeys().get(posBigTable);
ExprNodeColumnDesc keyCol = (ExprNodeColumnDesc) keyDesc.get(0);
-
- tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
- keyCol.getColumn(), selectedMJOpRatio);
+ String realTSColName = getOriginalTSColName(selectedMJOp, keyCol.getColumn());
+ if (realTSColName != null) {
+ tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
+ realTSColName, selectedMJOpRatio);
+ } else {
+ LOG.warn("ProbeDecode could not find TSColName for ColKey {} with MJ Schema {} ", keyCol, selectedMJOp.getSchema());
+ }
}
return tsProbeDecodeCtx;
}
+ private static String getOriginalTSColName(MapJoinOperator mjOp, String internalCoName) {
Review comment:
Can you check if this can already be done by any of the utility methods in `OperatorUtils`? If it is not, can we move it to that class with rest of utility methods in case it is useful in the future?
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
##########
@@ -1566,13 +1569,38 @@ private void removeSemijoinsParallelToMapJoin(OptimizeTezProcContext procCtx)
List<ExprNodeDesc> keyDesc = selectedMJOp.getConf().getKeys().get(posBigTable);
ExprNodeColumnDesc keyCol = (ExprNodeColumnDesc) keyDesc.get(0);
-
- tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
- keyCol.getColumn(), selectedMJOpRatio);
+ String realTSColName = getOriginalTSColName(selectedMJOp, keyCol.getColumn());
+ if (realTSColName != null) {
+ tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
+ realTSColName, selectedMJOpRatio);
+ } else {
+ LOG.warn("ProbeDecode could not find TSColName for ColKey {} with MJ Schema {} ", keyCol, selectedMJOp.getSchema());
Review comment:
You could throw an error only if you are running in test mode:
```
if (conf.getBoolVar(ConfVars.HIVE_IN_TEST)) {
...
```
While I am not a huge fan of interleaving a check like this within the production code, it may help you identify any issues, gaps, or regressions in the future.
If you do not want to do that, an alternative is uploading a patch throwing an error instead of printing the warning to get a complete test run, then create follow-up issues if there are any. It will not help you with regressions in the future, but you will be able to identify any existing issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pgaref commented on a change in pull request #1152: HIVE-23730: TezCompiler tracking the original TS key columnName
Posted by GitBox <gi...@apache.org>.
pgaref commented on a change in pull request #1152:
URL: https://github.com/apache/hive/pull/1152#discussion_r443482036
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
##########
@@ -1566,13 +1569,38 @@ private void removeSemijoinsParallelToMapJoin(OptimizeTezProcContext procCtx)
List<ExprNodeDesc> keyDesc = selectedMJOp.getConf().getKeys().get(posBigTable);
ExprNodeColumnDesc keyCol = (ExprNodeColumnDesc) keyDesc.get(0);
-
- tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
- keyCol.getColumn(), selectedMJOpRatio);
+ String realTSColName = getOriginalTSColName(selectedMJOp, keyCol.getColumn());
+ if (realTSColName != null) {
+ tsProbeDecodeCtx = new TableScanOperator.ProbeDecodeContext(mjCacheKey, mjSmallTablePos,
+ realTSColName, selectedMJOpRatio);
+ } else {
+ LOG.warn("ProbeDecode could not find TSColName for ColKey {} with MJ Schema {} ", keyCol, selectedMJOp.getSchema());
Review comment:
Hey @jcamachor , thanks for the comments!
The HIVE_IN_TEST trick could work as long we enable probedecode optimisation by default right? (currently if off)
Just enabled the optimisation for this PR (throwing an exception instead of warn) to identify any existing issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pgaref commented on pull request #1152: HIVE-23730: TezCompiler tracking the original TS key columnName
Posted by GitBox <gi...@apache.org>.
pgaref commented on pull request #1152:
URL: https://github.com/apache/hive/pull/1152#issuecomment-649787045
Hey @jcamachor latest run is green after enabling probedecode by default and updating MJ query outputs -- could you please take another look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org