[GitHub] [hive] kgyrtkirk commented on a diff in pull request #3253: HIVE-26184: COLLECT_SET with GROUP BY is very slow when some keys are highly skewed

[GitBox <gi...@apache.org>]

kgyrtkirk commented on code in PR #3253:
URL: https://github.com/apache/hive/pull/3253#discussion_r860749534


##########
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java:
##########
@@ -95,11 +95,27 @@ public MkArrayAggregationBuffer() {
         throw new RuntimeException("Buffer type unknown");
       }
     }
+
+    private void reset() {
+      if (bufferType == BufferType.LIST) {
+        container.clear();
+      } else if (bufferType == BufferType.SET) {
+        // Don't reuse a container because HashSet#clear can be very slow. The operation takes O(N)

Review Comment:
   why did the entries got skewed in the firstplace? don't we miss or have incorrect implementation of some `hashCode()` method?
   
   could you please add a testcase which reproduces the issue?
   maybe you could probably write a test against the UDF itself..



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org