[GitHub] justinborromeo commented on a change in pull request #7133: 6088 - Time Ordering On Scans

[GitBox <gi...@apache.org>]

justinborromeo commented on a change in pull request #7133: 6088 - Time Ordering On Scans
URL: https://github.com/apache/incubator-druid/pull/7133#discussion_r259561320
 
 

 ##########
 File path: processing/src/main/java/org/apache/druid/query/scan/ScanQueryRunnerFactory.java
 ##########
 @@ -129,4 +255,129 @@ public ScanQueryRunner(ScanQueryEngine engine, Segment segment)
       return engine.process((ScanQuery) query, segment, responseContext);
     }
   }
+
+  /**
+   * This iterator supports iteration through any Iterable of unbatched ScanResultValues (1 event/ScanResultValue) and
+   * aggregates events into ScanResultValues with {@code batchSize} events.  The columns from the first event per
+   * ScanResultValue will be used to populate the column section.
+   */
+  @VisibleForTesting
+  static class ScanBatchedIterator implements CloseableIterator<ScanResultValue>
+  {
+    private final Iterator<ScanResultValue> itr;
+    private final int batchSize;
+
+    public ScanBatchedIterator(Iterator<ScanResultValue> iterator, int batchSize)
+    {
+      this.itr = iterator;
+      this.batchSize = batchSize;
+    }
+
+    @Override
+    public void close() throws IOException
+    {
+    }
+
+    @Override
+    public boolean hasNext()
+    {
+      return itr.hasNext();
+    }
+
+    @Override
+    public ScanResultValue next()
+    {
+      // Create new ScanResultValue from event map
+      List<Object> eventsToAdd = new ArrayList<>(batchSize);
+      List<String> columns = new ArrayList<>();
+      while (eventsToAdd.size() < batchSize && itr.hasNext()) {
+        ScanResultValue srv = itr.next();
+        // Only replace once using the columns from the first event
+        columns = columns.isEmpty() ? srv.getColumns() : columns;
+        eventsToAdd.add(Iterables.getOnlyElement((List) srv.getEvents()));
+      }
+      return new ScanResultValue(null, columns, eventsToAdd);
+    }
+  }
+
+  @VisibleForTesting
+  static class ScanResultValueBatchingSequence extends YieldingSequenceBase<ScanResultValue>
 
 Review comment:
   This class is super hacky and I'm 99% sure there's a better way to do it but I'm not sure how.  Basically, what it does is it takes a Sequence of single-event ScanResultValues and batches them together into a Sequence of multi-event (# events = batchSize) ScanResultValues.
   
   Right now, it's a Sequence that takes another Sequence as an argument and iterates through the single-event sequence batching together ScanResultValues.  However, on line 356, there's a super sketchy type-cast that I added because I couldn't make toYielder() non-generic (with respect to OutType).
   
   Is there a more Sequence-oriented/cleaner way to perform this batching?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org