You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "okumin (via GitHub)" <gi...@apache.org> on 2024/01/25 08:36:31 UTC

[PR] [WIP] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

okumin opened a new pull request, #5037:
URL: https://github.com/apache/hive/pull/5037

   Don't review this yet
   
   
   
   
   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
     2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
     3. Ensure you have added or run the appropriate tests for your PR: 
     4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX:  Your PR title ...'.
     5. Be sure to keep the PR description updated to reflect all changes.
     6. Please write your PR title to summarize what this PR proposes.
     7. If possible, provide a concise example to reproduce the issue for a faster review.
   
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   ### Is the change a dependency upgrade?
   <!--
   If yes, please attach a file with output from mvn dependency:tree to validate a complete upgrade of dependency.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-1924046957

   > I believe the link should be allowed - but in this the merge should be allowed due to the fact that these things represent the same thing.
   
   
   
   I see now why the signature become an obstacle - I believe the SWO was able to match the same plan-parts to only scan the table once; if you don't have link calls there to explain the situation -> that could cause equiv violations here...could you please check that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-2042611991

   Let me close this one because we will likely go with #5077 or another approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [WIP] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-1911974645

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-1926644977

   > each CTE is compiled separately
   got it now - I was  thinking its the other way around
   
   I was thinking about this a bit in the last couple days; here is a summary of my thoughts:
   
   * to have this feature cause trouble the following is needed:
      * a failing query
      * an incorrectly taken stat entry
      * the bad stat entry causes further issues
      * ....not very likely :D
   * keeping multiple equiv groups which should be merged is not fortunate - but may not cause trouble
   * at the end of the day all a `Map<Signature, RuntimeStat>` will be created from all these details
     * if there is are 2 groups which have the same `Signature` that will mean there will be multiple candidates for the key in the map => which means those should have been in a single group...that's why I didn't really like to relax the check on the signature
       * I believe lookup-by-signatue will either return one a representative group ; even when there are more
   
   from the listed approaches in the doc:
   * I liked the best the `RelTreeSignature` related; I think that would be the best if possioble
      * this might be a way to achieve a merge between the 2 groups
      * have you tried somehow relax the alias name of `HiveTableScan` in the `RelTreeSignature` ? 
         * that class creates the signature of the relnodes 1-by-1 ; so you could access/customize/etc based on the type
            * possibly use the `@Signature` annotation
            * use something to filter out poisonous stuff `@IgnoreSig` or something
            * by-hand stuff
   * on the other hand after thinking it thru what might be the outcomes - I find that the idea to not pull the signatures during linking might actually work correctly...
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [WIP] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-1914780935

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [WIP] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5037:
URL: https://github.com/apache/hive/pull/5037#discussion_r1470541604


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,30 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
-    }
-    if (mGroups.size() > 1) {
+    final EquivGroup group1 = objectMap.get(o1);
+    final EquivGroup group2 = objectMap.get(o2);
+    if (group1 != null && group2 != null && group1 != group2) {

Review Comment:
   I made some refactoring and will resolve the current conversations once.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin closed pull request #5037: HIVE-24167: TPC-DS query 14 fails while generating plan for the filter
URL: https://github.com/apache/hive/pull/5037


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [WIP] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-1911395609

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5037:
URL: https://github.com/apache/hive/pull/5037#discussion_r1476160515


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,46 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
+    // Caches signatures on the first access. A signature of an Operator could change as optimizers could mutate it,
+    // keeping its semantics. The current implementation caches the signature before optimizations so that we can
+    // link Operators with their signatures at consistent timing.
+    registerSignature(o1);
+    registerSignature(o2);

Review Comment:
   > Assuming that we do not allow implicit linking via signatures I don't know what's the point of registering the signature at this stage
   
   That's because some optimization can mutate Operators, changing their true signatures. [One example I remember is TableScanPPD](https://github.com/apache/hive/blob/rel/release-4.0.0-beta-1/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java#L414). I'm not 100% sure which signatures should be used in that case, but it sounds more reasonable and predictable for me to use ones before optimization than ones in the middle of optimization. I would say Operators should not be mutable for the signing purpose, but it is a too fundamental change.
   
   I will make a reply to @kgyrtkirk in another thread as I guess those comments were written before reading my gist.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #5037:
URL: https://github.com/apache/hive/pull/5037#discussion_r1475951785


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,46 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
+    // Caches signatures on the first access. A signature of an Operator could change as optimizers could mutate it,
+    // keeping its semantics. The current implementation caches the signature before optimizations so that we can
+    // link Operators with their signatures at consistent timing.
+    registerSignature(o1);
+    registerSignature(o2);
+    final EquivGroup linkedGroup1 = objectMap.get(o1);
+    final EquivGroup linkedGroup2 = objectMap.get(o2);
+
+    if (linkedGroup1 == null && linkedGroup2 == null) {
+      final EquivGroup group = new EquivGroup();
+      group.add(o1);
+      group.add(o2);
+      groups.add(group);
+      return;
     }
-    if (mGroups.size() > 1) {
-      if (!mayMerge) {
-        throw new RuntimeException("equivalence mapping violation");
-      }
-      EquivGroup newGrp = new EquivGroup();
-      newGrp.add(o1);
-      newGrp.add(o2);
-      for (EquivGroup g : mGroups) {
-        for (Object o : g.members) {
-          newGrp.add(o);
-        }
-      }
-      groups.add(newGrp);
-      groups.removeAll(mGroups);
-    } else {
-      EquivGroup targetGroup = mGroups.isEmpty() ? new EquivGroup() : mGroups.iterator().next();
-      groups.add(targetGroup);
-      targetGroup.add(o1);
-      targetGroup.add(o2);
+    if (linkedGroup1 == null || linkedGroup2 == null || linkedGroup1 == linkedGroup2) {

Review Comment:
   what's all these changes do better than the old?



##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,46 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
+    // Caches signatures on the first access. A signature of an Operator could change as optimizers could mutate it,
+    // keeping its semantics. The current implementation caches the signature before optimizations so that we can
+    // link Operators with their signatures at consistent timing.

Review Comment:
   please remove this comment and don't pre-heat caches unless it gives real benefits
   the purpose of the cache is not something like this ; but to prevent `O(N^2)` computation ( and also to reduce storage size by making it more serialize friendly...)



##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,46 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
+    // Caches signatures on the first access. A signature of an Operator could change as optimizers could mutate it,
+    // keeping its semantics. The current implementation caches the signature before optimizations so that we can
+    // link Operators with their signatures at consistent timing.
+    registerSignature(o1);
+    registerSignature(o2);

Review Comment:
   > This change reverts that logic and takes signatures out of the equation for determining equivalent groups.
   
   haven't read all of it - but if thats true - please don't do that...
   looking at the volume of changes - it must be changing something fundamental...
   
   what's the issue - is it an equiv  violation? can you put the stacktrace somewhere?
   I think that should be fixed without altering this part - it might be a missing link; or a missing signature annotation somewhere - you should not change this at all... 
   
   all these things could enable to back-map runtime statistict up to the calcite join planning phase...but I think that feature was never completed/enabled



##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -286,8 +271,13 @@ public Iterator<EquivGroup> iterateGroups() {
   }
 
   public OpTreeSignature getSignatureOf(Operator<?> op) {
-    OpTreeSignature sig = signatureCache.getSignature(op);
-    return sig;
+    return signatureCache.getSignature(op);
+  }
+
+  private void registerSignature(Object o) {
+    if (o instanceof Operator) {
+      getSignatureOf((Operator<?>) o);
+    }
   }

Review Comment:
   whats the point of this method? the cache will compute it if needed...what the point of pre-registering - if that's needed its not a *cache* anymore!



##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,46 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
+    // Caches signatures on the first access. A signature of an Operator could change as optimizers could mutate it,
+    // keeping its semantics. The current implementation caches the signature before optimizations so that we can
+    // link Operators with their signatures at consistent timing.
+    registerSignature(o1);
+    registerSignature(o2);
+    final EquivGroup linkedGroup1 = objectMap.get(o1);
+    final EquivGroup linkedGroup2 = objectMap.get(o2);
+
+    if (linkedGroup1 == null && linkedGroup2 == null) {
+      final EquivGroup group = new EquivGroup();
+      group.add(o1);
+      group.add(o2);
+      groups.add(group);
+      return;
     }
-    if (mGroups.size() > 1) {
-      if (!mayMerge) {
-        throw new RuntimeException("equivalence mapping violation");
-      }
-      EquivGroup newGrp = new EquivGroup();
-      newGrp.add(o1);
-      newGrp.add(o2);
-      for (EquivGroup g : mGroups) {
-        for (Object o : g.members) {
-          newGrp.add(o);
-        }
-      }
-      groups.add(newGrp);
-      groups.removeAll(mGroups);
-    } else {
-      EquivGroup targetGroup = mGroups.isEmpty() ? new EquivGroup() : mGroups.iterator().next();
-      groups.add(targetGroup);
-      targetGroup.add(o1);
-      targetGroup.add(o2);
+    if (linkedGroup1 == null || linkedGroup2 == null || linkedGroup1 == linkedGroup2) {
+      final EquivGroup group = linkedGroup1 != null ? linkedGroup1 : linkedGroup2;
+      group.add(o1);
+      group.add(o2);
+      return;
     }
 
+    if (!mayMerge) {
+      throw new RuntimeException(String.format(
+          "Failed to link %s and %s. This error mostly means a bug of Hive",

Review Comment:
   I find this error message pretty cloudy on what the issue is... 
   
   I think the sentecte "Equivalence mapping violation" should be there - as that's what the problem is...we have 2 groups; not allowed to put them into the same equiv group -> big trouble...
   
   



##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,46 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
+    // Caches signatures on the first access. A signature of an Operator could change as optimizers could mutate it,
+    // keeping its semantics. The current implementation caches the signature before optimizations so that we can
+    // link Operators with their signatures at consistent timing.
+    registerSignature(o1);
+    registerSignature(o2);
+    final EquivGroup linkedGroup1 = objectMap.get(o1);
+    final EquivGroup linkedGroup2 = objectMap.get(o2);

Review Comment:
   these changes will weaken this stuff - so that 2 OPs which have the same signature will be allowed -> that's not good....
   its not causing much issues yet - because almost all existing operators have good signatures; but if that degrades...runtime stats will be applied incorrectly



##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,46 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
+    // Caches signatures on the first access. A signature of an Operator could change as optimizers could mutate it,
+    // keeping its semantics. The current implementation caches the signature before optimizations so that we can
+    // link Operators with their signatures at consistent timing.
+    registerSignature(o1);
+    registerSignature(o2);

Review Comment:
   to get a failing test for these changes you would need 2 ops which have conflicting signatures stored into the metastore ; loaded back and they might get applied incorrectly...
   possibly pretty hard to do...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [WIP] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-1914313579

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [WIP] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-1909840390

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5037:
URL: https://github.com/apache/hive/pull/5037#discussion_r1476260894


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,46 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
+    // Caches signatures on the first access. A signature of an Operator could change as optimizers could mutate it,
+    // keeping its semantics. The current implementation caches the signature before optimizations so that we can
+    // link Operators with their signatures at consistent timing.
+    registerSignature(o1);
+    registerSignature(o2);

Review Comment:
   I missed mentioning what are affected by such optimizations. Many test cases on the following revision failed with `equivalence mapping violation`.
   https://github.com/apache/hive/pull/5037/commits/217b26db7eee95a0d25725df985adc79705a80d6
   
   That's because it tries to LINK a pre-optimized operator with a post-optimized operator.
   1. Before optimization
       - Operator A has a signature SA and is grouped as GA
       - Operator B has a signature SB and is grouped as GB
   2. AuxSignatureLinker doesn't MERGE GA and GB as A and B have different signatures. Note that non aux signatures are not cached here
   3. After optimization
       - Operator B is optimized, and its signature becomes SA
   4. StatsRulePsrocFactory tries to LINK GA and GB as A and B share the same signature SA -> equivalence mapping violation
   
   Potentially, GA and GB are actually mergeable though the current implementation doesn't do so.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-1937743415

   @kgyrtkirk 
   I tried the option using `RelTreeSignature` in the following Pull Request. I still have some concerns, but it will likely work from my perspective.
   https://github.com/apache/hive/pull/5077
   
   Could you please take a look and see if it seems feasible? If we prefer #5077 , I will void this PR and try to get it merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #5037:
URL: https://github.com/apache/hive/pull/5037#discussion_r1475779718


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,46 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
+    // Caches signatures on the first access. A signature of an Operator could change as optimizers could mutate it,
+    // keeping its semantics. The current implementation caches the signature before optimizations so that we can
+    // link Operators with their signatures at consistent timing.
+    registerSignature(o1);
+    registerSignature(o2);
+    final EquivGroup linkedGroup1 = objectMap.get(o1);
+    final EquivGroup linkedGroup2 = objectMap.get(o2);
+
+    if (linkedGroup1 == null && linkedGroup2 == null) {
+      final EquivGroup group = new EquivGroup();
+      group.add(o1);
+      group.add(o2);
+      groups.add(group);
+      return;
     }
-    if (mGroups.size() > 1) {
-      if (!mayMerge) {
-        throw new RuntimeException("equivalence mapping violation");
-      }
-      EquivGroup newGrp = new EquivGroup();
-      newGrp.add(o1);
-      newGrp.add(o2);
-      for (EquivGroup g : mGroups) {
-        for (Object o : g.members) {
-          newGrp.add(o);
-        }
-      }
-      groups.add(newGrp);
-      groups.removeAll(mGroups);
-    } else {
-      EquivGroup targetGroup = mGroups.isEmpty() ? new EquivGroup() : mGroups.iterator().next();
-      groups.add(targetGroup);
-      targetGroup.add(o1);
-      targetGroup.add(o2);
+    if (linkedGroup1 == null || linkedGroup2 == null || linkedGroup1 == linkedGroup2) {
+      final EquivGroup group = linkedGroup1 != null ? linkedGroup1 : linkedGroup2;
+      group.add(o1);
+      group.add(o2);
+      return;
     }
 
+    if (!mayMerge) {
+      throw new RuntimeException(String.format(
+          "Failed to link %s and %s. This error mostly means a bug of Hive",

Review Comment:
   nit: An uncaught `RuntimeException` almost always denotes a bug of the application. I don't think we need to make the exception that verbose stating the obvious.
   
   Adding the operators which led to exception in the message is good idea, makes sense.



##########
ql/src/test/queries/clientpositive/cbo_cte_materialization.q:
##########
@@ -0,0 +1,27 @@
+--! qt:dataset:src
+
+set hive.optimize.cte.materialize.threshold=1;
+set hive.optimize.cte.materialize.full.aggregate.only=false;
+
+EXPLAIN CBO
+WITH materialized_cte AS (
+  SELECT key, value FROM src WHERE key != '100'
+),
+another_materialized_cte AS (
+  SELECT key, value FROM src WHERE key != '100'
+)
+SELECT a.key, a.value, b.key, b.value
+FROM materialized_cte a
+JOIN another_materialized_cte b ON a.key = b.key
+ORDER BY a.key;
+
+WITH materialized_cte AS (
+  SELECT key, value FROM src WHERE key != '100'
+),
+another_materialized_cte AS (
+  SELECT key, value FROM src WHERE key != '100'
+)
+SELECT a.key, a.value, b.key, b.value
+FROM materialized_cte a
+JOIN another_materialized_cte b ON a.key = b.key
+ORDER BY a.key;

Review Comment:
   The problem is in the compilation phase so I don't think we need to actually run the query. Consider dropping the execution. If you opt to keep it then I would suggest crafting a much simpler test (without 1K lines in the output). It would be nice to keep the execution time of our test suite as low as possible. Moreover, it is not easy to see what the `src` table contains so verifying that the result is indeed correct is cumbersome.



##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,46 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
+    // Caches signatures on the first access. A signature of an Operator could change as optimizers could mutate it,
+    // keeping its semantics. The current implementation caches the signature before optimizations so that we can
+    // link Operators with their signatures at consistent timing.
+    registerSignature(o1);
+    registerSignature(o2);

Review Comment:
   Linking operators implicitly (using signatures) was added by @kgyrtkirk in HIVE-18926. This change reverts that logic and takes signatures out of the equation for determining equivalent groups.
   
   My understanding is that if this change goes in operators can be linked with signatures **only** explicitly. I don't see test failures and .q.out changes due to this so it seems that the change does not have a big impact on existing use-cases. Moreover, it fixes some known compilation crashes so I consider this an improvement over the current situation.
   
   I don't see a significant risk in merging this change but I will let @kgyrtkirk comment in case I am missing something.
   
   Assuming that we do not allow implicit linking via signatures I don't know what's the point of registering the signature at this stage. My understanding is that the cache is meant to improve performance not a means to perform equivalence checks. If we don't make use of the signature here then I don't think we should compute it. @okumin although you added some comments justifying the registration I would appreciate some additional clarifications regarding these additions.



##########
ql/src/test/queries/clientpositive/cbo_cte_materialization.q:
##########
@@ -0,0 +1,27 @@
+--! qt:dataset:src
+
+set hive.optimize.cte.materialize.threshold=1;
+set hive.optimize.cte.materialize.full.aggregate.only=false;
+
+EXPLAIN CBO
+WITH materialized_cte AS (
+  SELECT key, value FROM src WHERE key != '100'
+),
+another_materialized_cte AS (
+  SELECT key, value FROM src WHERE key != '100'
+)
+SELECT a.key, a.value, b.key, b.value
+FROM materialized_cte a
+JOIN another_materialized_cte b ON a.key = b.key
+ORDER BY a.key;

Review Comment:
   Consider adding (or replacing this with) a traditional `EXPLAIN` where we can see the effect of `hive.optimize.cte.materialize.threshold` and the full plan for the materialized ctes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-1924100180

   @kgyrtkirk
   
   > I believe the link should be allowed - but in this the merge should be allowed due to the fact that these things represent the same thing.
   
   I guess this mentions the link on creating a FilterOperator?
   
   > I see now why the signature become an obstacle - I believe the SWO was able to match the same plan-parts to only scan the table once; if you don't have link calls there to explain the situation -> that could cause equiv violations here...could you please check that?
   
   I guess the SWO stands for SharedWorkOptimizer. With CTE materialization, each CTE is compiled separately. It means `materialized_cte`, `another_materialized_cte`, and the body part are compiled one by one in a dedicated manner. So, SharedWorkOptimizer has no chance as it's invoked in analyzeInternal.
   
   Just for a note. We use CTE because we need to support some cases which SharedWorkOptimizer doesn't support. An example is complex CTE + UNION ALL. We may replace CTE materialization with SharedWorkOptimizer when it becomes more mature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [WIP] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5037:
URL: https://github.com/apache/hive/pull/5037#discussion_r1467141935


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,30 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
-    }
-    if (mGroups.size() > 1) {
+    final EquivGroup group1 = objectMap.get(o1);
+    final EquivGroup group2 = objectMap.get(o2);
+    if (group1 != null && group2 != null && group1 != group2) {

Review Comment:
   Thanks for your review but you may not need to review this PR yet.
   I'm still investigating the issue and trying several approaches. I created this PR to trigger CI as I can't know which test cases could be affected because of the complexity of SemanticAnalyzer. It turned out that the current approach would break something else, and I am trying another approach.
   See also https://gist.github.com/okumin/b111fe0a911507bdf6a7204f49b9cb72 to trace the current findings and my activity logs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-1923666355

   just seen the details you linked to the PR.
   
   I believe the `link` should be allowed - but in this the merge should be allowed due to the fact that these things represent the same thing.
   I see now why the signature become an obstacle - I believe the SWO was able to match the same plan-parts to only scan the table once; if you don't have `link` calls there to explain the situation -> that could cause `equiv violations`  here...could you please check that?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [WIP] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "aturoczy (via GitHub)" <gi...@apache.org>.
aturoczy commented on code in PR #5037:
URL: https://github.com/apache/hive/pull/5037#discussion_r1466719874


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,30 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
-    }
-    if (mGroups.size() > 1) {
+    final EquivGroup group1 = objectMap.get(o1);
+    final EquivGroup group2 = objectMap.get(o2);
+    if (group1 != null && group2 != null && group1 != group2) {

Review Comment:
   I would change the logic a bit. In my mind, from Clean Code perspective if you check the true path it is easier to process and maintain the code.
   i.e:
   if (group1 == null || group2 == null || group1 == group2) 
   return;
   
   
   



##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,30 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
-    }
-    if (mGroups.size() > 1) {
+    final EquivGroup group1 = objectMap.get(o1);
+    final EquivGroup group2 = objectMap.get(o2);
+    if (group1 != null && group2 != null && group1 != group2) {
       if (!mayMerge) {
         throw new RuntimeException("equivalence mapping violation");
       }
       EquivGroup newGrp = new EquivGroup();
       newGrp.add(o1);
       newGrp.add(o2);
-      for (EquivGroup g : mGroups) {
-        for (Object o : g.members) {
-          newGrp.add(o);
-        }
-      }
+      group1.members.forEach(newGrp::add);
+      group2.members.forEach(newGrp::add);
       groups.add(newGrp);
-      groups.removeAll(mGroups);
+      groups.remove(group1);
+      groups.remove(group2);
     } else {
-      EquivGroup targetGroup = mGroups.isEmpty() ? new EquivGroup() : mGroups.iterator().next();
+      EquivGroup targetGroup = group1 != null ? group1 : (group2 != null ? group2 : new EquivGroup());

Review Comment:
   The logic is great, but my eyes hurted with this amount of ? : operator :) 



##########
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java:
##########
@@ -200,54 +199,30 @@ public void merge(Object o1, Object o2) {
   }
 
   private void link(Object o1, Object o2, boolean mayMerge) {
-
-    Set<Object> keySet = Collections.newSetFromMap(new IdentityHashMap<Object, Boolean>());
-    keySet.add(o1);
-    keySet.add(o2);
-    keySet.add(getKeyFor(o1));
-    keySet.add(getKeyFor(o2));
-
-    Set<EquivGroup> mGroups = Collections.newSetFromMap(new IdentityHashMap<EquivGroup, Boolean>());
-
-    for (Object object : keySet) {
-      EquivGroup group = objectMap.get(object);
-      if (group != null) {
-        mGroups.add(group);
-      }
-    }
-    if (mGroups.size() > 1) {
+    final EquivGroup group1 = objectMap.get(o1);
+    final EquivGroup group2 = objectMap.get(o2);
+    if (group1 != null && group2 != null && group1 != group2) {
       if (!mayMerge) {
         throw new RuntimeException("equivalence mapping violation");

Review Comment:
   If we touch this change please the exception message to Uppercase the first word



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [WIP] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5037:
URL: https://github.com/apache/hive/pull/5037#issuecomment-1913134667

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5037&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5037)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org