Re: [PR] HIVE-28549: Limit the maximum number of operators merged by SharedWorkOptimizer [hive]

["ngsg (via GitHub)" <gi...@apache.org>]

ngsg commented on code in PR #5492:
URL: https://github.com/apache/hive/pull/5492#discussion_r1831948116


##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java:
##########
@@ -248,6 +249,32 @@ public ParseContext transform(ParseContext pctx) throws SemanticException {
     return pctx;
   }
 
+  private static List<List<TableScanOperator>> groupTableScanOperators(List<Entry<String, Long>> sortedTables,
+      ArrayListMultimap<String, TableScanOperator> tableNameToOps, int batchSize) {
+    final List<List<TableScanOperator>> batches = new ArrayList<>();
+    for (Entry<String, Long> tablePair : sortedTables) {
+      final String tableName = tablePair.getKey();
+      final List<TableScanOperator> scans = tableNameToOps.get(tableName);
+      if (batchSize == -1) {
+        batches.add(scans);
+        continue;
+      }
+
+      final int limit = scans.size();
+      int from = 0;
+      while (from != limit) {
+        final int to = Math.min(limit, from + batchSize);
+        // We have to copy the list because it is mutated later
+        final List<TableScanOperator> subList = new ArrayList<>(scans.subList(from, to));
+        Preconditions.checkState(!subList.isEmpty());

Review Comment:
   Could we change this Preconditions to an assertion or some other expressions that guarantee `!scans.subList(from, to).isEmpty()`? (e.g. `Preconditions.checkArgument(batchSize > 0)` at the top of the method.) Using Preconditions to check the invariant of code, rather than caller's or client's requirements, seems a bit odd to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org