[GitHub] [hive] okumin commented on a diff in pull request #3253: HIVE-26184: COLLECT_SET with GROUP BY is very slow when some keys are highly skewed

[GitBox <gi...@apache.org>]

okumin commented on code in PR #3253:
URL: https://github.com/apache/hive/pull/3253#discussion_r860788021


##########
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java:
##########
@@ -95,11 +95,27 @@ public MkArrayAggregationBuffer() {
         throw new RuntimeException("Buffer type unknown");
       }
     }
+
+    private void reset() {
+      if (bufferType == BufferType.LIST) {
+        container.clear();
+      } else if (bufferType == BufferType.SET) {
+        // Don't reuse a container because HashSet#clear can be very slow. The operation takes O(N)

Review Comment:
   @kgyrtkirk Thanks for your quick review!
   
   I wanted to mean skew of GROUP BY keys here, not elements of HashSet. Let me illustrate that with the following query, mapping articles into their comments. If a certain article accidentally gets very popular, it has much more comments than the others. My `skew` means such situation.
   
   ```
   SELECT article_id, COLLECT_SET(comment) FROM comments GROUP BY article_id
   ```
   
   The capacity of the internal hash table of `MkArrayAggregationBuffer#container` will eventually grow as much as it can retain all comments tied to the most skewed article so far. Also, the internal hash table will never get smaller because resizing happens only when new entries are added(precisely, this point depends on the implementation of JDK).
   From the nature of hash tables, the duration of `HashSet#clear` relies on the capacity of an internal hash table. It's an operation to fill all cells with NULLs.
   
   Because of the two points, GroupByOperator suddenly slows down once it processes a skewed key. For example, assuming the first `article_id=1` has 1,000,000 comments, `GenericUDAFMkCollectionEvaluator#reset` has to fill a very big hash table with many NULLs every time even if all following articles have 0 comments.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org