You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "ramesh0201 (via GitHub)" <gi...@apache.org> on 2024/09/23 20:34:40 UTC
[PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
ramesh0201 opened a new pull request, #5462:
URL: https://github.com/apache/hive/pull/5462
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Map join container cache key should be unique for each map join hash table if they are of different join types. This will prevent hash table from getting shared between different join types.
### Why are the changes needed?
Sharing same hash table between different join types will not work because they expect different hash table type based on the join type. Otherwise we might run into ClassCastException and also potential wrong results in certain cases.
### Does this PR introduce _any_ user-facing change?
No
### Is the change a dependency upgrade?
No
### How was this patch tested?
cd ql
mvn test -Dtest=TestSharedWorkOptimizer#testMapJoinCacheReuse
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5462:
URL: https://github.com/apache/hive/pull/5462#issuecomment-2549560447
@kasakrisz, do you know if there is anything left here? could this be merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5462:
URL: https://github.com/apache/hive/pull/5462#issuecomment-2414168636
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5462) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5462&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5462&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5462&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5462&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5462&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5462)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on code in PR #5462:
URL: https://github.com/apache/hive/pull/5462#discussion_r1900558346
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java:
##########
@@ -1007,6 +1007,16 @@ private boolean canShareBroadcastInputs(
return false;
}
+ // Map Joins when vectorized can have different formats for the hash tables built on the small table.
+ // Reusing hash tables between different join type can lead to ClassCastException or even wrong results.
+ if (mapJoinOp1.getConf().getConds() != null && mapJoinOp1.getConf().getConds().length > 0
Review Comment:
Thank you! Modified this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5462:
URL: https://github.com/apache/hive/pull/5462#issuecomment-2369447922
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5462) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5462&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5462&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5462&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5462&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5462&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5462)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5462:
URL: https://github.com/apache/hive/pull/5462#issuecomment-2529753016
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5462) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5462&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5462&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5462&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5462&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5462&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5462)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on code in PR #5462:
URL: https://github.com/apache/hive/pull/5462#discussion_r1801807544
##########
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestSharedWorkOptimizer.java:
##########
@@ -323,39 +314,72 @@ public void testMapJoinCacheReuse() {
Operator<?> rsC2 = OperatorFactory.getAndMakeChild(getReduceSinkDesc(), smallTableC);
// case 1. MapJoin1: (big, A, B), MapJoin2: (big, A, B)
- List<Operator<?>> joinSource1 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource2 = Arrays.asList(ts2, rsA2, rsB2);
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), true);
- MapJoinOperator mapJoin1 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource1);
- MapJoinOperator mapJoin2 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource2);
+ // case 2. MapJoin1: (big, A, B), MapJoin2: (big, A, C)
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsC2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
Review Comment:
Addressed this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5462:
URL: https://github.com/apache/hive/pull/5462#discussion_r1826016878
##########
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestSharedWorkOptimizer.java:
##########
@@ -27,18 +27,9 @@
import org.apache.hadoop.hive.ql.metadata.Table;
import org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.SharedWorkOptimizerCache;
import org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.TSComparator;
+import org.apache.hadoop.hive.ql.parse.JoinType;
import org.apache.hadoop.hive.ql.parse.SemanticException;
-import org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc;
-import org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc;
-import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
-import org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc;
-import org.apache.hadoop.hive.ql.plan.FilterDesc;
-import org.apache.hadoop.hive.ql.plan.MapJoinDesc;
-import org.apache.hadoop.hive.ql.plan.OperatorDesc;
-import org.apache.hadoop.hive.ql.plan.ReduceSinkDesc;
-import org.apache.hadoop.hive.ql.plan.Statistics;
-import org.apache.hadoop.hive.ql.plan.TableDesc;
-import org.apache.hadoop.hive.ql.plan.TableScanDesc;
+import org.apache.hadoop.hive.ql.plan.*;
Review Comment:
star import
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 merged PR #5462:
URL: https://github.com/apache/hive/pull/5462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5462:
URL: https://github.com/apache/hive/pull/5462#issuecomment-2567329840
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5462) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5462&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5462&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5462&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5462&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5462&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5462)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on code in PR #5462:
URL: https://github.com/apache/hive/pull/5462#discussion_r1774934271
##########
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestSharedWorkOptimizer.java:
##########
@@ -323,39 +314,72 @@ public void testMapJoinCacheReuse() {
Operator<?> rsC2 = OperatorFactory.getAndMakeChild(getReduceSinkDesc(), smallTableC);
// case 1. MapJoin1: (big, A, B), MapJoin2: (big, A, B)
- List<Operator<?>> joinSource1 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource2 = Arrays.asList(ts2, rsA2, rsB2);
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), true);
- MapJoinOperator mapJoin1 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource1);
- MapJoinOperator mapJoin2 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource2);
+ // case 2. MapJoin1: (big, A, B), MapJoin2: (big, A, C)
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsC2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
- runMapJoinCacheReuseOptimization(mapJoin1, mapJoin2);
- assertEquals(mapJoin1.getConf().getCacheKey(), mapJoin2.getConf().getCacheKey());
+ // case 3. MapJoin1: (big, A, B), MapJoin2: (A, big, B)
+ setupJoinOperatorsAndTest(ts1, rsA2, rsA1, ts2, rsB1, rsB2, 0, 1,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
- // case 2. MapJoin3: (big, A, B), MapJoin4: (big, A, C)
- List<Operator<?>> joinSource3 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource4 = Arrays.asList(ts2, rsA2, rsC2);
+ // case 4. MapJoin1: (big, A, B), MapJoin2: (big, A, B) with inner join and left outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.LEFTOUTER.ordinal(), false);
- MapJoinOperator mapJoin3 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource3);
- MapJoinOperator mapJoin4 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource4);
+ // case 5. same as case4 with inner join and right outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.RIGHTOUTER.ordinal(), false);
- runMapJoinCacheReuseOptimization(mapJoin3, mapJoin4);
- assertNotEquals(mapJoin3.getConf().getCacheKey(), mapJoin4.getConf().getCacheKey());
+ // case 6. same as case4 with inner join and left semi join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.LEFTSEMI.ordinal(), false);
- // case 3. MapJoin5: (big, A, B), MapJoin6: (A, big, B)
- List<Operator<?>> joinSource5 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource6 = Arrays.asList(rsA2, ts2, rsB2);
+ // case 7. same as case4 with inner join and anti join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.ANTI.ordinal(), false);
- MapJoinOperator mapJoin5 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource5);
- MapJoinOperator mapJoin6 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(1), joinSource6);
+ // case 8. same as case4 with right outer join and left semi join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.RIGHTOUTER.ordinal(), JoinType.LEFTSEMI.ordinal(), false);
+
+ // case 9. same as case4 with right outer join and left outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.RIGHTOUTER.ordinal(), JoinType.LEFTOUTER.ordinal(), true);
+
+ // case 10. same as case4 with full outer join and inner join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.FULLOUTER.ordinal(), JoinType.INNER.ordinal(), false);
+ }
- runMapJoinCacheReuseOptimization(mapJoin5, mapJoin6);
- assertNotEquals(mapJoin5.getConf().getCacheKey(), mapJoin6.getConf().getCacheKey());
+ private void setupJoinOperatorsAndTest(Operator<?> join1Op1, Operator<?> join2Op1, Operator<?> join1Op2,
+ Operator<?> join2Op2, Operator<?> join1Op3, Operator<?> join2Op3,
+ int join1PosBigTable, int join2PosBigTable, int join1Type, int join2Type,
+ boolean positive) {
+ List<Operator<?>> joinSource1 = Arrays.asList(join1Op1, join1Op2, join1Op3);
+ List<Operator<?>> joinSource2 = Arrays.asList(join2Op1, join2Op2, join2Op3);
+
+ MapJoinOperator mapJoin1 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
+ cCtx, getMapJoinDesc(join1PosBigTable), joinSource1);
+ MapJoinOperator mapJoin2 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
+ cCtx, getMapJoinDesc(join2PosBigTable), joinSource2);
+ JoinCondDesc cond1 = new JoinCondDesc(0,0, join1Type);
+ JoinCondDesc cond2 = new JoinCondDesc(0,0, join2Type);
+ mapJoin1.getConf().setConds(new JoinCondDesc[]{cond1});
+ mapJoin2.getConf().setConds(new JoinCondDesc[]{cond2});
+ boolean isNoOuterJoin1 = !((join1Type == JoinType.RIGHTOUTER.ordinal()) || (join1Type == JoinType.LEFTOUTER.ordinal())
+ || (join1Type == JoinType.FULLOUTER.ordinal()));
Review Comment:
nit.: extra brackets can be removed:
```
!(join1Type == JoinType.RIGHTOUTER.ordinal() ||
join1Type == JoinType.LEFTOUTER.ordinal() ||
join1Type == JoinType.FULLOUTER.ordinal());
```
##########
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestSharedWorkOptimizer.java:
##########
@@ -323,39 +314,72 @@ public void testMapJoinCacheReuse() {
Operator<?> rsC2 = OperatorFactory.getAndMakeChild(getReduceSinkDesc(), smallTableC);
// case 1. MapJoin1: (big, A, B), MapJoin2: (big, A, B)
- List<Operator<?>> joinSource1 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource2 = Arrays.asList(ts2, rsA2, rsB2);
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), true);
- MapJoinOperator mapJoin1 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource1);
- MapJoinOperator mapJoin2 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource2);
+ // case 2. MapJoin1: (big, A, B), MapJoin2: (big, A, C)
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsC2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
- runMapJoinCacheReuseOptimization(mapJoin1, mapJoin2);
- assertEquals(mapJoin1.getConf().getCacheKey(), mapJoin2.getConf().getCacheKey());
+ // case 3. MapJoin1: (big, A, B), MapJoin2: (A, big, B)
+ setupJoinOperatorsAndTest(ts1, rsA2, rsA1, ts2, rsB1, rsB2, 0, 1,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
- // case 2. MapJoin3: (big, A, B), MapJoin4: (big, A, C)
- List<Operator<?>> joinSource3 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource4 = Arrays.asList(ts2, rsA2, rsC2);
+ // case 4. MapJoin1: (big, A, B), MapJoin2: (big, A, B) with inner join and left outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.LEFTOUTER.ordinal(), false);
- MapJoinOperator mapJoin3 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource3);
- MapJoinOperator mapJoin4 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource4);
+ // case 5. same as case4 with inner join and right outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.RIGHTOUTER.ordinal(), false);
- runMapJoinCacheReuseOptimization(mapJoin3, mapJoin4);
- assertNotEquals(mapJoin3.getConf().getCacheKey(), mapJoin4.getConf().getCacheKey());
+ // case 6. same as case4 with inner join and left semi join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.LEFTSEMI.ordinal(), false);
- // case 3. MapJoin5: (big, A, B), MapJoin6: (A, big, B)
- List<Operator<?>> joinSource5 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource6 = Arrays.asList(rsA2, ts2, rsB2);
+ // case 7. same as case4 with inner join and anti join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.ANTI.ordinal(), false);
- MapJoinOperator mapJoin5 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource5);
- MapJoinOperator mapJoin6 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(1), joinSource6);
+ // case 8. same as case4 with right outer join and left semi join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.RIGHTOUTER.ordinal(), JoinType.LEFTSEMI.ordinal(), false);
+
+ // case 9. same as case4 with right outer join and left outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.RIGHTOUTER.ordinal(), JoinType.LEFTOUTER.ordinal(), true);
+
+ // case 10. same as case4 with full outer join and inner join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.FULLOUTER.ordinal(), JoinType.INNER.ordinal(), false);
+ }
- runMapJoinCacheReuseOptimization(mapJoin5, mapJoin6);
- assertNotEquals(mapJoin5.getConf().getCacheKey(), mapJoin6.getConf().getCacheKey());
+ private void setupJoinOperatorsAndTest(Operator<?> join1Op1, Operator<?> join2Op1, Operator<?> join1Op2,
+ Operator<?> join2Op2, Operator<?> join1Op3, Operator<?> join2Op3,
Review Comment:
I think the original code was more straightforward. What are `join1Op1, join2Op1, ... join2Op3` stands for? How about replacing these with `List<Operator<?>> joinSource1, List<Operator<?>> joinSource2` ?
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java:
##########
@@ -1006,6 +1006,16 @@ private boolean canShareBroadcastInputs(
return false;
}
+ // Map Joins when vectorized can have different formats for the hash tables built on the small table.
+ // Reusing hash tables between different join type can lead to ClassCastException or even wrong results.
+ if (mapJoinOp1.getConf().getConds() != null && mapJoinOp1.getConf().getConds().length > 0
+ && mapJoinOp2.getConf().getConds() != null && mapJoinOp2.getConf().getConds().length > 0
+ && mapJoinOp1.getConf().getConds()[0].getType() != mapJoinOp2.getConf().getConds()[0].getType()
+ && (mapJoinOp1.getConf().isNoOuterJoin() || mapJoinOp2.getConf().isNoOuterJoin())) {
Review Comment:
Does it mean that hashtable reuse is allowed if both mapjoins are outer joins but the join type is different?
##########
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestSharedWorkOptimizer.java:
##########
@@ -323,39 +314,72 @@ public void testMapJoinCacheReuse() {
Operator<?> rsC2 = OperatorFactory.getAndMakeChild(getReduceSinkDesc(), smallTableC);
// case 1. MapJoin1: (big, A, B), MapJoin2: (big, A, B)
- List<Operator<?>> joinSource1 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource2 = Arrays.asList(ts2, rsA2, rsB2);
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), true);
- MapJoinOperator mapJoin1 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource1);
- MapJoinOperator mapJoin2 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource2);
+ // case 2. MapJoin1: (big, A, B), MapJoin2: (big, A, C)
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsC2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
- runMapJoinCacheReuseOptimization(mapJoin1, mapJoin2);
- assertEquals(mapJoin1.getConf().getCacheKey(), mapJoin2.getConf().getCacheKey());
+ // case 3. MapJoin1: (big, A, B), MapJoin2: (A, big, B)
+ setupJoinOperatorsAndTest(ts1, rsA2, rsA1, ts2, rsB1, rsB2, 0, 1,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
- // case 2. MapJoin3: (big, A, B), MapJoin4: (big, A, C)
- List<Operator<?>> joinSource3 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource4 = Arrays.asList(ts2, rsA2, rsC2);
+ // case 4. MapJoin1: (big, A, B), MapJoin2: (big, A, B) with inner join and left outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.LEFTOUTER.ordinal(), false);
- MapJoinOperator mapJoin3 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource3);
- MapJoinOperator mapJoin4 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource4);
+ // case 5. same as case4 with inner join and right outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.RIGHTOUTER.ordinal(), false);
- runMapJoinCacheReuseOptimization(mapJoin3, mapJoin4);
- assertNotEquals(mapJoin3.getConf().getCacheKey(), mapJoin4.getConf().getCacheKey());
+ // case 6. same as case4 with inner join and left semi join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.LEFTSEMI.ordinal(), false);
- // case 3. MapJoin5: (big, A, B), MapJoin6: (A, big, B)
- List<Operator<?>> joinSource5 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource6 = Arrays.asList(rsA2, ts2, rsB2);
+ // case 7. same as case4 with inner join and anti join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.ANTI.ordinal(), false);
- MapJoinOperator mapJoin5 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource5);
- MapJoinOperator mapJoin6 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(1), joinSource6);
+ // case 8. same as case4 with right outer join and left semi join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.RIGHTOUTER.ordinal(), JoinType.LEFTSEMI.ordinal(), false);
+
+ // case 9. same as case4 with right outer join and left outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.RIGHTOUTER.ordinal(), JoinType.LEFTOUTER.ordinal(), true);
+
+ // case 10. same as case4 with full outer join and inner join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.FULLOUTER.ordinal(), JoinType.INNER.ordinal(), false);
+ }
- runMapJoinCacheReuseOptimization(mapJoin5, mapJoin6);
- assertNotEquals(mapJoin5.getConf().getCacheKey(), mapJoin6.getConf().getCacheKey());
+ private void setupJoinOperatorsAndTest(Operator<?> join1Op1, Operator<?> join2Op1, Operator<?> join1Op2,
Review Comment:
This method can be simplified further if the `MapJoinOperator` creation is extracted to another method like
```
MapJoinOperator setupMapJoin(List<Operator<?>> joinSource, int joinType, int join1PosBigTable) {
...
return mapJoinOperator;
}
```
Then it can be called twice to construct the two map join operators
##########
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestSharedWorkOptimizer.java:
##########
@@ -323,39 +314,72 @@ public void testMapJoinCacheReuse() {
Operator<?> rsC2 = OperatorFactory.getAndMakeChild(getReduceSinkDesc(), smallTableC);
// case 1. MapJoin1: (big, A, B), MapJoin2: (big, A, B)
- List<Operator<?>> joinSource1 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource2 = Arrays.asList(ts2, rsA2, rsB2);
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), true);
- MapJoinOperator mapJoin1 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource1);
- MapJoinOperator mapJoin2 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource2);
+ // case 2. MapJoin1: (big, A, B), MapJoin2: (big, A, C)
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsC2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
Review Comment:
It would be better to move each case to its test method because in the current form the context (ts1, smallTableA, rsA1, etc) of the tests is reused. The code or any test may makes changes on these objects and causes interference between test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5462:
URL: https://github.com/apache/hive/pull/5462#issuecomment-2369868920
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5462) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5462&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5462&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5462&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5462&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5462&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5462)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on code in PR #5462:
URL: https://github.com/apache/hive/pull/5462#discussion_r1801806837
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java:
##########
@@ -1006,6 +1006,16 @@ private boolean canShareBroadcastInputs(
return false;
}
+ // Map Joins when vectorized can have different formats for the hash tables built on the small table.
+ // Reusing hash tables between different join type can lead to ClassCastException or even wrong results.
+ if (mapJoinOp1.getConf().getConds() != null && mapJoinOp1.getConf().getConds().length > 0
+ && mapJoinOp2.getConf().getConds() != null && mapJoinOp2.getConf().getConds().length > 0
+ && mapJoinOp1.getConf().getConds()[0].getType() != mapJoinOp2.getConf().getConds()[0].getType()
+ && (mapJoinOp1.getConf().isNoOuterJoin() || mapJoinOp2.getConf().isNoOuterJoin())) {
Review Comment:
Yes, if they are both outer joins, we still generate the same hash table type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5462:
URL: https://github.com/apache/hive/pull/5462#discussion_r1889184757
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java:
##########
@@ -1007,6 +1007,16 @@ private boolean canShareBroadcastInputs(
return false;
}
+ // Map Joins when vectorized can have different formats for the hash tables built on the small table.
+ // Reusing hash tables between different join type can lead to ClassCastException or even wrong results.
+ if (mapJoinOp1.getConf().getConds() != null && mapJoinOp1.getConf().getConds().length > 0
Review Comment:
minor: could be simplified with util method:
````
ArrayUtils.isNotEmpty(mapJoinOp1.getConf().getConds())
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on code in PR #5462:
URL: https://github.com/apache/hive/pull/5462#discussion_r1801807286
##########
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestSharedWorkOptimizer.java:
##########
@@ -323,39 +314,72 @@ public void testMapJoinCacheReuse() {
Operator<?> rsC2 = OperatorFactory.getAndMakeChild(getReduceSinkDesc(), smallTableC);
// case 1. MapJoin1: (big, A, B), MapJoin2: (big, A, B)
- List<Operator<?>> joinSource1 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource2 = Arrays.asList(ts2, rsA2, rsB2);
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), true);
- MapJoinOperator mapJoin1 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource1);
- MapJoinOperator mapJoin2 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource2);
+ // case 2. MapJoin1: (big, A, B), MapJoin2: (big, A, C)
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsC2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
- runMapJoinCacheReuseOptimization(mapJoin1, mapJoin2);
- assertEquals(mapJoin1.getConf().getCacheKey(), mapJoin2.getConf().getCacheKey());
+ // case 3. MapJoin1: (big, A, B), MapJoin2: (A, big, B)
+ setupJoinOperatorsAndTest(ts1, rsA2, rsA1, ts2, rsB1, rsB2, 0, 1,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
- // case 2. MapJoin3: (big, A, B), MapJoin4: (big, A, C)
- List<Operator<?>> joinSource3 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource4 = Arrays.asList(ts2, rsA2, rsC2);
+ // case 4. MapJoin1: (big, A, B), MapJoin2: (big, A, B) with inner join and left outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.LEFTOUTER.ordinal(), false);
- MapJoinOperator mapJoin3 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource3);
- MapJoinOperator mapJoin4 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource4);
+ // case 5. same as case4 with inner join and right outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.RIGHTOUTER.ordinal(), false);
- runMapJoinCacheReuseOptimization(mapJoin3, mapJoin4);
- assertNotEquals(mapJoin3.getConf().getCacheKey(), mapJoin4.getConf().getCacheKey());
+ // case 6. same as case4 with inner join and left semi join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.LEFTSEMI.ordinal(), false);
- // case 3. MapJoin5: (big, A, B), MapJoin6: (A, big, B)
- List<Operator<?>> joinSource5 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource6 = Arrays.asList(rsA2, ts2, rsB2);
+ // case 7. same as case4 with inner join and anti join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.ANTI.ordinal(), false);
- MapJoinOperator mapJoin5 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource5);
- MapJoinOperator mapJoin6 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(1), joinSource6);
+ // case 8. same as case4 with right outer join and left semi join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.RIGHTOUTER.ordinal(), JoinType.LEFTSEMI.ordinal(), false);
+
+ // case 9. same as case4 with right outer join and left outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.RIGHTOUTER.ordinal(), JoinType.LEFTOUTER.ordinal(), true);
+
+ // case 10. same as case4 with full outer join and inner join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.FULLOUTER.ordinal(), JoinType.INNER.ordinal(), false);
+ }
- runMapJoinCacheReuseOptimization(mapJoin5, mapJoin6);
- assertNotEquals(mapJoin5.getConf().getCacheKey(), mapJoin6.getConf().getCacheKey());
+ private void setupJoinOperatorsAndTest(Operator<?> join1Op1, Operator<?> join2Op1, Operator<?> join1Op2,
+ Operator<?> join2Op2, Operator<?> join1Op3, Operator<?> join2Op3,
Review Comment:
Modified this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28532 Map Join Reuse cache allows to share hashtables for different join types [hive]
Posted by "ramesh0201 (via GitHub)" <gi...@apache.org>.
ramesh0201 commented on code in PR #5462:
URL: https://github.com/apache/hive/pull/5462#discussion_r1801807862
##########
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestSharedWorkOptimizer.java:
##########
@@ -323,39 +314,72 @@ public void testMapJoinCacheReuse() {
Operator<?> rsC2 = OperatorFactory.getAndMakeChild(getReduceSinkDesc(), smallTableC);
// case 1. MapJoin1: (big, A, B), MapJoin2: (big, A, B)
- List<Operator<?>> joinSource1 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource2 = Arrays.asList(ts2, rsA2, rsB2);
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), true);
- MapJoinOperator mapJoin1 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource1);
- MapJoinOperator mapJoin2 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource2);
+ // case 2. MapJoin1: (big, A, B), MapJoin2: (big, A, C)
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsC2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
- runMapJoinCacheReuseOptimization(mapJoin1, mapJoin2);
- assertEquals(mapJoin1.getConf().getCacheKey(), mapJoin2.getConf().getCacheKey());
+ // case 3. MapJoin1: (big, A, B), MapJoin2: (A, big, B)
+ setupJoinOperatorsAndTest(ts1, rsA2, rsA1, ts2, rsB1, rsB2, 0, 1,
+ JoinType.INNER.ordinal(), JoinType.INNER.ordinal(), false);
- // case 2. MapJoin3: (big, A, B), MapJoin4: (big, A, C)
- List<Operator<?>> joinSource3 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource4 = Arrays.asList(ts2, rsA2, rsC2);
+ // case 4. MapJoin1: (big, A, B), MapJoin2: (big, A, B) with inner join and left outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.LEFTOUTER.ordinal(), false);
- MapJoinOperator mapJoin3 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource3);
- MapJoinOperator mapJoin4 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource4);
+ // case 5. same as case4 with inner join and right outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.RIGHTOUTER.ordinal(), false);
- runMapJoinCacheReuseOptimization(mapJoin3, mapJoin4);
- assertNotEquals(mapJoin3.getConf().getCacheKey(), mapJoin4.getConf().getCacheKey());
+ // case 6. same as case4 with inner join and left semi join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.LEFTSEMI.ordinal(), false);
- // case 3. MapJoin5: (big, A, B), MapJoin6: (A, big, B)
- List<Operator<?>> joinSource5 = Arrays.asList(ts1, rsA1, rsB1);
- List<Operator<?>> joinSource6 = Arrays.asList(rsA2, ts2, rsB2);
+ // case 7. same as case4 with inner join and anti join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.INNER.ordinal(), JoinType.ANTI.ordinal(), false);
- MapJoinOperator mapJoin5 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(0), joinSource5);
- MapJoinOperator mapJoin6 = (MapJoinOperator) OperatorFactory.getAndMakeChild(
- cCtx, getMapJoinDesc(1), joinSource6);
+ // case 8. same as case4 with right outer join and left semi join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.RIGHTOUTER.ordinal(), JoinType.LEFTSEMI.ordinal(), false);
+
+ // case 9. same as case4 with right outer join and left outer join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.RIGHTOUTER.ordinal(), JoinType.LEFTOUTER.ordinal(), true);
+
+ // case 10. same as case4 with full outer join and inner join
+ setupJoinOperatorsAndTest(ts1, ts2, rsA1, rsA2, rsB1, rsB2, 0, 0,
+ JoinType.FULLOUTER.ordinal(), JoinType.INNER.ordinal(), false);
+ }
- runMapJoinCacheReuseOptimization(mapJoin5, mapJoin6);
- assertNotEquals(mapJoin5.getConf().getCacheKey(), mapJoin6.getConf().getCacheKey());
+ private void setupJoinOperatorsAndTest(Operator<?> join1Op1, Operator<?> join2Op1, Operator<?> join1Op2,
Review Comment:
Addressed this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org