You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/12/06 08:05:57 UTC

[GitHub] [druid] gianm opened a new pull request, #13506: Sort-merge join and hash shuffles for MSQ.

gianm opened a new pull request, #13506:
URL: https://github.com/apache/druid/pull/13506

   The main changes are in the processing, multi-stage-query, and sql modules.
   
   processing module:
   
   1) Rename SortColumn to KeyColumn, replace boolean descending with KeyOrder.
      This makes it nicer to model hash keys, which use KeyOrder.NONE.
   
   2) Add nullability checkers to the FieldReader interface, and an
      "isPartiallyNullKey" method to FrameComparisonWidget. The join
      processor uses this to detect null keys.
   
   3) Add WritableFrameChannel.isClosed and OutputChannel.isReadableChannelReady
      so callers can tell which OutputChannels are ready for reading and which
      aren't.
   
   4) Specialize FrameProcessors.makeCursor to return FrameCursor, a random-access
      implementation. The join processor uses this to rewind when it needs to
      replay a set of rows with a particular key.
   
   5) Add MemoryAllocatorFactory, which is embedded inside FrameWriterFactory
      instead of a particular MemoryAllocator. This allows FrameWriterFactory
      to be shared in more scenarios.
   
   multi-stage-query module:
   
   1) ShuffleSpec: Add hash-based shuffles. New enum ShuffleKind helps callers
      figure out what kind of shuffle is happening. The change from SortColumn
      to KeyColumn allows ClusterBy to be used for both hash-based and sort-based
      shuffling.
   
   2) WorkerImpl: Add ability to handle hash-based shuffles. Refactor the logic
      to be more readable by moving the work-order-running code to the inner
      class RunWorkOrder, and the shuffle-pipeline-building code to the inner
      class ShufflePipelineBuilder.
   
   3) Add SortMergeJoinFrameProcessor and factory.
   
   4) WorkerMemoryParameters: Adjust logic to reserve space for output frames
      for hash partitioning. (We need one frame per partition.)
   
   sql module:
   
   1) Add sqlJoinAlgorithm context parameter; can be "broadcast" or
      "sortMerge". With native, it must always be "broadcast", or it's a
      validation error. MSQ supports both. Default is "broadcast" in
      both engines.
   
   2) Validate that MSQs do not use broadcast join with RIGHT or FULL join,
      as results are not correct for broadcast join with those types. Allow
      this in native for two reasons: legacy (the docs caution against it,
      but it's always been allowed), and the fact that it actually *does*
      generate correct results in native when the join is processed on the
      Broker. It is much less likely that MSQ will plan in such a way that
      generates correct results.
   
   3) Remove subquery penalty in DruidJoinQueryRel when using sort-merge
      join, because subqueries are always required, so there's no reason
      to penalize them.
   
   4) Move previously-disabled join reordering and manipulation rules to
      FANCY_JOIN_RULES, and enable them when using sort-merge join. Helps
      get to better plans where projections and filters are pushed down.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] lgtm-com[bot] commented on pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13506:
URL: https://github.com/apache/druid/pull/13506#issuecomment-1339108436

   This pull request **introduces 1 alert** and **fixes 1** when merging 7b6c47a02493ec9552b990788bc062321036f059 into 136322d13b68f1e14552ea606ac70101ff6af400 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-32bf58be8452aee572491be90c8e70902634023d)
   
   **new alerts:**
   
   * 1 for Spurious Javadoc @param tags
   
   **fixed alerts:**
   
   * 1 for Uncontrolled data used in path expression
   ---
   
   **Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1114156705


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Names of the key columns on each side of the join, unprefixed. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<List<KeyColumn>> keyColumns;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputs = ImmutableList.of(left, right);
+    this.inputChannels = inputs.stream().map(ReadableInput::getChannel).collect(Collectors.toList());
+    this.keyColumns = keyColumns;
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(new Tracker(LEFT), new Tracker(RIGHT));
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()
+           && !(trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd())) {

Review Comment:
   I tried to simplify it. It looks like this now:
   
   ```
       while (!allTrackersAreAtEnd()
              && !trackers.get(LEFT).needsMoreData()
              && !trackers.get(RIGHT).needsMoreData()) {
         // Algorithm can proceed: not all trackers are at the end of their streams, and no tracker needs more data to
         // read the current cursor or move it forward.
   
   ```
   
   The comment is new too. It spells out the logic as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1042611343


##########
docs/multi-stage-query/reference.md:
##########
@@ -198,13 +198,99 @@ The following table lists the context parameters for the MSQ task engine:
 | `maxNumTasks` | SELECT, INSERT, REPLACE<br /><br />The maximum total number of tasks to launch, including the controller task. The lowest possible value for this setting is 2: one controller and one worker. All tasks must be able to launch simultaneously. If they cannot, the query returns a `TaskStartTimeout` error code after approximately 10 minutes.<br /><br />May also be provided as `numTasks`. If both are present, `maxNumTasks` takes priority.| 2 |
 | `taskAssignment` | SELECT, INSERT, REPLACE<br /><br />Determines how many tasks to use. Possible values include: <ul><li>`max`: Uses as many tasks as possible, up to `maxNumTasks`.</li><li>`auto`: When file sizes can be determined through directory listing (for example: local files, S3, GCS, HDFS) uses as few tasks as possible without exceeding 10 GiB or 10,000 files per task, unless exceeding these limits is necessary to stay within `maxNumTasks`. When file sizes cannot be determined through directory listing (for example: http), behaves the same as `max`.</li></ul> | `max` |
 | `finalizeAggregations` | SELECT, INSERT, REPLACE<br /><br />Determines the type of aggregation to return. If true, Druid finalizes the results of complex aggregations that directly appear in query results. If false, Druid returns the aggregation's intermediate type rather than finalized type. This parameter is useful during ingestion, where it enables storing sketches directly in Druid tables. For more information about aggregations, see [SQL aggregation functions](../querying/sql-aggregations.md). | true |
+| `sqlJoinAlgorithm` | SELECT, INSERT, REPLACE<br /><br />Algorithm to use for JOIN. Use `broadcast` (the default) for broadcast hash join or `sortMerge` for sort-merge join. Affects all JOIN operations in the query. See [Joins](#joins) for more details. | `broadcast` |

Review Comment:
   Unfortunately, hints are not available in the version of Calcite version we use. Newer versions have this: https://calcite.apache.org/docs/reference.html#sql-hints
   
   @abhishekagarwal87 found some reasons we couldn't do it right now, as referenced by this comment: https://github.com/apache/druid/pull/13153#issuecomment-1261204764. @abhishekagarwal87 would you mind writing up your notes as to what blocks an upgrade, and making an issue about that, titled something like `Upgrade Calcite past <whatever version introduced the blocking problem>`? That way, we have an issue we can refer to and use to discuss possible ways to fix the blockers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on PR #13506:
URL: https://github.com/apache/druid/pull/13506#issuecomment-1439930345

   @imply-cheddar I've pushed up changes responsive to your comments. Thanks for reviewing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1042615629


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerContext.java:
##########
@@ -73,4 +74,9 @@
   DruidNode selfNode();
 
   Bouncer processorBouncer();
+
+  default File tempDir(int stageNumber, String id)

Review Comment:
   I was modeling it off `File tempDir()` that is already here without args. I can change both? This one doesn't matter much to me either way. Thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1127471391


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1075 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Input channels for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputChannels = ImmutableList.of(left.getChannel(), right.getChannel());
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(
+        new Tracker(left, keyColumns.get(LEFT)),
+        new Tracker(right, keyColumns.get(RIGHT))
+    );
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!allTrackersAreAtEnd()
+           && !trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()) {
+      // Algorithm can proceed: not all trackers are at the end of their streams, and no tracker needs more data to
+      // read the current cursor or move it forward.
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+
+      // Two rows match if the keys compare equal _and_ neither key has a null component. (x JOIN y ON x.a = y.a does
+      // not match rows where "x.a" is null.)
+      final boolean match = markCmp == 0 && trackers.get(LEFT).hasCompletelyNonNullMark();
+
+      // If marked keys are equal on both sides ("match"), at least one side must have a complete set of rows
+      // for the marked key.
+      if (match && trackerWithCompleteSetForCurrentKey < 0) {
+        for (int i = 0; i < inputChannels.size(); i++) {
+          final Tracker tracker = trackers.get(i);
+
+          // Fetch up to one frame from each tracker, to check if that tracker has a complete set.
+          // Can't fetch more than one frame, because channels are only guaranteed to have one frame per run.
+          if (tracker.hasCompleteSetForMark() || (pushNextFrame(i) && tracker.hasCompleteSetForMark())) {
+            trackerWithCompleteSetForCurrentKey = i;
+            break;
+          }
+        }
+
+        if (trackerWithCompleteSetForCurrentKey < 0) {
+          // Algorithm cannot proceed; fetch more frames on the next run.
+          return nextAwait();
+        }
+      }
+
+      if (match || (markCmp <= 0 && joinType.isLefty()) || (markCmp >= 0 && joinType.isRighty())) {
+        // Emit row, if there's room in the current frameWriter.
+        joinColumnSelectorFactory.cmp = markCmp;
+        joinColumnSelectorFactory.match = match;
+
+        if (!frameWriter.addSelection()) {
+          if (frameWriter.getNumRows() > 0) {
+            // Out of space in the current frame. Run again without moving cursors.
+            flushCurrentFrame();
+            return ReturnOrAwait.runAgain();
+          } else {
+            throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
+          }
+        }
+      }
+
+      // Advance one or both trackers.
+      if (match) {
+        // Matching keys. First advance the tracker with the complete set.
+        final Tracker tracker = trackers.get(trackerWithCompleteSetForCurrentKey);
+        final Tracker otherTracker = trackers.get(trackerWithCompleteSetForCurrentKey == LEFT ? RIGHT : LEFT);
+
+        tracker.advance();
+        if (!tracker.isCurrentSameKeyAsMark()) {
+          // Reached end of complete set. Advance the other tracker.
+          otherTracker.advance();
+
+          // On next iteration (when we're sure to have data) either rewind the complete-set tracker, or update marks
+          // of both, as appropriate.
+          onNextIteration(() -> {
+            if (otherTracker.isCurrentSameKeyAsMark()) {
+              otherTracker.markCurrent(); // Set mark to enable cleanup of old frames.
+              tracker.rewindToMark();
+            } else {
+              // Reached end of the other side too. Advance marks on both trackers.
+              tracker.markCurrent();
+              otherTracker.markCurrent();
+              trackerWithCompleteSetForCurrentKey = -1;
+            }
+          });
+        }
+      } else {
+        final int trackerToAdvance;
+
+        if (markCmp < 0) {
+          trackerToAdvance = LEFT;
+        } else if (markCmp > 0) {
+          trackerToAdvance = RIGHT;
+        } else {
+          // Key is null on both sides. Note that there is a preference for running through the left side first
+          // on a FULL join. It doesn't really matter which side we run through first, but we do need to be consistent
+          // for the benefit of the logic in "shouldEmitColumnValue".
+          trackerToAdvance = joinType.isLefty() ? LEFT : RIGHT;
+        }
+
+        final Tracker tracker = trackers.get(trackerToAdvance);
+
+        tracker.advance();
+
+        // On next iteration (when we're sure to have data), update mark if the key changed.
+        onNextIteration(() -> {
+          if (!tracker.isCurrentSameKeyAsMark()) {
+            tracker.markCurrent();
+            trackerWithCompleteSetForCurrentKey = -1;
+          }
+        });
+      }
+    }
+
+    if (allTrackersAreAtEnd()) {
+      flushCurrentFrame();
+      return ReturnOrAwait.returnObject(0L);
+    } else {
+      // Keep reading.
+      return nextAwait();
+    }
+  }
+
+  @Override
+  public void cleanup() throws IOException
+  {
+    FrameProcessors.closeAll(inputChannels(), outputChannels(), frameWriter, () -> trackers.forEach(Tracker::clear));
+  }
+
+  /**
+   * Returns a {@link ReturnOrAwait#awaitAll} for the channel numbers that need more data and have not yet hit their
+   * buffered-bytes limit, {@link Limits#MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN}.
+   *
+   * If all channels have hit their limit, throws {@link MSQException} with {@link TooManyRowsWithSameKeyFault}.
+   */
+  private ReturnOrAwait<Long> nextAwait()
+  {
+    final IntSet awaitSet = new IntOpenHashSet();
+    int trackerAtLimit = -1;
+
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.needsMoreData()) {
+        if (tracker.totalBytesBuffered() < Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+          awaitSet.add(i);
+        } else if (trackerAtLimit < 0) {
+          trackerAtLimit = i;
+        }
+      }
+    }
+
+    if (awaitSet.isEmpty() && trackerAtLimit > 0) {
+      // All trackers that need more data are at their max buffered bytes limit. Generate a nice exception.
+      final Tracker tracker = trackers.get(trackerAtLimit);
+      if (tracker.totalBytesBuffered() > Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+        // Generate a nice exception.
+        throw new MSQException(
+            new TooManyRowsWithSameKeyFault(
+                tracker.readMarkKey(),
+                tracker.totalBytesBuffered(),
+                Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN
+            )
+        );
+      }
+    }
+
+    return ReturnOrAwait.awaitAll(awaitSet);
+  }
+
+  /**
+   * Whether all trackers return true from {@link Tracker#isAtEnd()}.
+   */
+  private boolean allTrackersAreAtEnd()
+  {
+    for (Tracker tracker : trackers) {
+      if (!tracker.isAtEnd()) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  /**
+   * Compares the marked rows of the two {@link #trackers}.
+   *
+   * @throws IllegalStateException if either tracker does not have a marked row and is not completely done
+   */
+  private int compareMarks()
+  {
+    final Tracker leftTracker = trackers.get(LEFT);
+    final Tracker rightTracker = trackers.get(RIGHT);
+
+    Preconditions.checkState(leftTracker.hasMark() || leftTracker.isAtEnd(), "left.hasMark || left.isAtEnd");
+    Preconditions.checkState(rightTracker.hasMark() || rightTracker.isAtEnd(), "right.hasMark || right.isAtEnd");
+
+    if (!leftTracker.hasMark()) {
+      return rightTracker.markFrame < 0 ? 0 : 1;
+    } else if (!rightTracker.hasMark()) {
+      return -1;
+    } else {
+      final FrameHolder leftHolder = leftTracker.holders.get(leftTracker.markFrame);
+      final FrameHolder rightHolder = rightTracker.holders.get(rightTracker.markFrame);
+      return leftHolder.comparisonWidget.compare(
+          leftTracker.markRow,
+          rightHolder.comparisonWidget,
+          rightTracker.markRow
+      );
+    }
+  }
+
+  /**
+   * Pushes a frame from the indicated channel into the appropriate tracker. Returns true if a frame was pushed
+   * or if the channel is finished.
+   */
+  private boolean pushNextFrame(final int channelNumber)
+  {
+    final ReadableFrameChannel channel = inputChannels.get(channelNumber);
+    final Tracker tracker = trackers.get(channelNumber);
+
+    if (!channel.isFinished() && !channel.canRead()) {
+      return false;
+    } else if (channel.isFinished()) {
+      tracker.push(null);
+      return true;
+    } else {
+      final Frame frame = channel.read();
+
+      if (frame.numRows() == 0) {
+        // Skip, read next.
+        return false;
+      } else {
+        tracker.push(frame);
+        return true;
+      }
+    }
+  }
+
+  private void onNextIteration(final Runnable runnable)
+  {
+    if (nextIterationRunnable != null) {
+      throw new ISE("postAdvanceRunnable already set");

Review Comment:
   Not that I can think of. If this ever happens, it's a bug in the class and someone's gonna have a fun time in the degbugger.



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1075 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Input channels for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputChannels = ImmutableList.of(left.getChannel(), right.getChannel());
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(
+        new Tracker(left, keyColumns.get(LEFT)),
+        new Tracker(right, keyColumns.get(RIGHT))
+    );
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!allTrackersAreAtEnd()
+           && !trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()) {
+      // Algorithm can proceed: not all trackers are at the end of their streams, and no tracker needs more data to
+      // read the current cursor or move it forward.
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+
+      // Two rows match if the keys compare equal _and_ neither key has a null component. (x JOIN y ON x.a = y.a does
+      // not match rows where "x.a" is null.)
+      final boolean match = markCmp == 0 && trackers.get(LEFT).hasCompletelyNonNullMark();
+
+      // If marked keys are equal on both sides ("match"), at least one side must have a complete set of rows
+      // for the marked key.
+      if (match && trackerWithCompleteSetForCurrentKey < 0) {
+        for (int i = 0; i < inputChannels.size(); i++) {
+          final Tracker tracker = trackers.get(i);
+
+          // Fetch up to one frame from each tracker, to check if that tracker has a complete set.
+          // Can't fetch more than one frame, because channels are only guaranteed to have one frame per run.
+          if (tracker.hasCompleteSetForMark() || (pushNextFrame(i) && tracker.hasCompleteSetForMark())) {
+            trackerWithCompleteSetForCurrentKey = i;
+            break;
+          }
+        }
+
+        if (trackerWithCompleteSetForCurrentKey < 0) {
+          // Algorithm cannot proceed; fetch more frames on the next run.
+          return nextAwait();
+        }
+      }
+
+      if (match || (markCmp <= 0 && joinType.isLefty()) || (markCmp >= 0 && joinType.isRighty())) {
+        // Emit row, if there's room in the current frameWriter.
+        joinColumnSelectorFactory.cmp = markCmp;
+        joinColumnSelectorFactory.match = match;
+
+        if (!frameWriter.addSelection()) {
+          if (frameWriter.getNumRows() > 0) {
+            // Out of space in the current frame. Run again without moving cursors.
+            flushCurrentFrame();
+            return ReturnOrAwait.runAgain();
+          } else {
+            throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
+          }
+        }
+      }
+
+      // Advance one or both trackers.
+      if (match) {
+        // Matching keys. First advance the tracker with the complete set.
+        final Tracker tracker = trackers.get(trackerWithCompleteSetForCurrentKey);
+        final Tracker otherTracker = trackers.get(trackerWithCompleteSetForCurrentKey == LEFT ? RIGHT : LEFT);
+
+        tracker.advance();
+        if (!tracker.isCurrentSameKeyAsMark()) {
+          // Reached end of complete set. Advance the other tracker.
+          otherTracker.advance();
+
+          // On next iteration (when we're sure to have data) either rewind the complete-set tracker, or update marks
+          // of both, as appropriate.
+          onNextIteration(() -> {
+            if (otherTracker.isCurrentSameKeyAsMark()) {
+              otherTracker.markCurrent(); // Set mark to enable cleanup of old frames.
+              tracker.rewindToMark();
+            } else {
+              // Reached end of the other side too. Advance marks on both trackers.
+              tracker.markCurrent();
+              otherTracker.markCurrent();
+              trackerWithCompleteSetForCurrentKey = -1;
+            }
+          });
+        }
+      } else {
+        final int trackerToAdvance;
+
+        if (markCmp < 0) {
+          trackerToAdvance = LEFT;
+        } else if (markCmp > 0) {
+          trackerToAdvance = RIGHT;
+        } else {
+          // Key is null on both sides. Note that there is a preference for running through the left side first
+          // on a FULL join. It doesn't really matter which side we run through first, but we do need to be consistent
+          // for the benefit of the logic in "shouldEmitColumnValue".
+          trackerToAdvance = joinType.isLefty() ? LEFT : RIGHT;
+        }
+
+        final Tracker tracker = trackers.get(trackerToAdvance);
+
+        tracker.advance();
+
+        // On next iteration (when we're sure to have data), update mark if the key changed.
+        onNextIteration(() -> {
+          if (!tracker.isCurrentSameKeyAsMark()) {
+            tracker.markCurrent();
+            trackerWithCompleteSetForCurrentKey = -1;
+          }
+        });
+      }
+    }
+
+    if (allTrackersAreAtEnd()) {
+      flushCurrentFrame();
+      return ReturnOrAwait.returnObject(0L);
+    } else {
+      // Keep reading.
+      return nextAwait();
+    }
+  }
+
+  @Override
+  public void cleanup() throws IOException
+  {
+    FrameProcessors.closeAll(inputChannels(), outputChannels(), frameWriter, () -> trackers.forEach(Tracker::clear));
+  }
+
+  /**
+   * Returns a {@link ReturnOrAwait#awaitAll} for the channel numbers that need more data and have not yet hit their
+   * buffered-bytes limit, {@link Limits#MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN}.
+   *
+   * If all channels have hit their limit, throws {@link MSQException} with {@link TooManyRowsWithSameKeyFault}.
+   */
+  private ReturnOrAwait<Long> nextAwait()
+  {
+    final IntSet awaitSet = new IntOpenHashSet();
+    int trackerAtLimit = -1;
+
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.needsMoreData()) {
+        if (tracker.totalBytesBuffered() < Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+          awaitSet.add(i);
+        } else if (trackerAtLimit < 0) {
+          trackerAtLimit = i;
+        }
+      }
+    }
+
+    if (awaitSet.isEmpty() && trackerAtLimit > 0) {
+      // All trackers that need more data are at their max buffered bytes limit. Generate a nice exception.
+      final Tracker tracker = trackers.get(trackerAtLimit);
+      if (tracker.totalBytesBuffered() > Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+        // Generate a nice exception.
+        throw new MSQException(
+            new TooManyRowsWithSameKeyFault(
+                tracker.readMarkKey(),
+                tracker.totalBytesBuffered(),
+                Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN
+            )
+        );
+      }
+    }
+
+    return ReturnOrAwait.awaitAll(awaitSet);
+  }
+
+  /**
+   * Whether all trackers return true from {@link Tracker#isAtEnd()}.
+   */
+  private boolean allTrackersAreAtEnd()
+  {
+    for (Tracker tracker : trackers) {
+      if (!tracker.isAtEnd()) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  /**
+   * Compares the marked rows of the two {@link #trackers}.
+   *
+   * @throws IllegalStateException if either tracker does not have a marked row and is not completely done
+   */
+  private int compareMarks()
+  {
+    final Tracker leftTracker = trackers.get(LEFT);
+    final Tracker rightTracker = trackers.get(RIGHT);
+
+    Preconditions.checkState(leftTracker.hasMark() || leftTracker.isAtEnd(), "left.hasMark || left.isAtEnd");
+    Preconditions.checkState(rightTracker.hasMark() || rightTracker.isAtEnd(), "right.hasMark || right.isAtEnd");
+
+    if (!leftTracker.hasMark()) {
+      return rightTracker.markFrame < 0 ? 0 : 1;
+    } else if (!rightTracker.hasMark()) {
+      return -1;
+    } else {
+      final FrameHolder leftHolder = leftTracker.holders.get(leftTracker.markFrame);
+      final FrameHolder rightHolder = rightTracker.holders.get(rightTracker.markFrame);
+      return leftHolder.comparisonWidget.compare(
+          leftTracker.markRow,
+          rightHolder.comparisonWidget,
+          rightTracker.markRow
+      );
+    }
+  }
+
+  /**
+   * Pushes a frame from the indicated channel into the appropriate tracker. Returns true if a frame was pushed
+   * or if the channel is finished.
+   */
+  private boolean pushNextFrame(final int channelNumber)
+  {
+    final ReadableFrameChannel channel = inputChannels.get(channelNumber);
+    final Tracker tracker = trackers.get(channelNumber);
+
+    if (!channel.isFinished() && !channel.canRead()) {
+      return false;
+    } else if (channel.isFinished()) {
+      tracker.push(null);
+      return true;
+    } else {
+      final Frame frame = channel.read();
+
+      if (frame.numRows() == 0) {
+        // Skip, read next.
+        return false;
+      } else {
+        tracker.push(frame);
+        return true;
+      }
+    }
+  }
+
+  private void onNextIteration(final Runnable runnable)
+  {
+    if (nextIterationRunnable != null) {
+      throw new ISE("postAdvanceRunnable already set");

Review Comment:
   Not that I can think of. If this ever happens, it's a bug in the class and someone's gonna have a fun time in the debugger.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1127469101


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java:
##########
@@ -212,24 +219,51 @@ public boolean doesShuffle()
 
   public boolean doesSortDuringShuffle()
   {
-    if (shuffleSpec == null) {
+    if (shuffleSpec == null || shuffleSpec.clusterBy().isEmpty()) {
       return false;
     } else {
-      return !shuffleSpec.getClusterBy().getColumns().isEmpty() || shuffleSpec.needsStatistics();
+      return shuffleSpec.clusterBy().sortable();
     }
   }
 
-  public Optional<ShuffleSpec> getShuffleSpec()
+  /**
+   * Returns the {@link ShuffleSpec} for this stage, if {@link #doesShuffle()}.
+   *
+   * @throws IllegalStateException if this stage does not shuffle
+   */
+  public ShuffleSpec getShuffleSpec()
   {
-    return Optional.ofNullable(shuffleSpec);
+    if (shuffleSpec == null) {
+      throw new IllegalStateException("Stage does not shuffle");
+    }
+
+    return shuffleSpec;
   }

Review Comment:
   Yes: `doesShuffle`



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java:
##########
@@ -212,24 +219,51 @@ public boolean doesShuffle()
 
   public boolean doesSortDuringShuffle()
   {
-    if (shuffleSpec == null) {
+    if (shuffleSpec == null || shuffleSpec.clusterBy().isEmpty()) {
       return false;
     } else {
-      return !shuffleSpec.getClusterBy().getColumns().isEmpty() || shuffleSpec.needsStatistics();
+      return shuffleSpec.clusterBy().sortable();
     }
   }
 
-  public Optional<ShuffleSpec> getShuffleSpec()
+  /**
+   * Returns the {@link ShuffleSpec} for this stage, if {@link #doesShuffle()}.
+   *
+   * @throws IllegalStateException if this stage does not shuffle
+   */
+  public ShuffleSpec getShuffleSpec()
   {
-    return Optional.ofNullable(shuffleSpec);
+    if (shuffleSpec == null) {
+      throw new IllegalStateException("Stage does not shuffle");
+    }
+
+    return shuffleSpec;
   }

Review Comment:
   Yes: `doesShuffle()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1114228324


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Names of the key columns on each side of the join, unprefixed. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<List<KeyColumn>> keyColumns;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputs = ImmutableList.of(left, right);
+    this.inputChannels = inputs.stream().map(ReadableInput::getChannel).collect(Collectors.toList());
+    this.keyColumns = keyColumns;
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(new Tracker(LEFT), new Tracker(RIGHT));
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()
+           && !(trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd())) {
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+      final boolean match = markCmp == 0 && !trackers.get(LEFT).hasPartiallyNullMark();
+
+      // Fetch new frames until the algorithm can proceed:
+      //   1) If marked keys are equal on both sides, at least one side has a complete set of rows for the marked key.
+      //   2) Otherwise, each side has at least one readable row, or is completely done. (Checked by "while" condition.)
+      if (match && trackerWithCompleteSetForCurrentKey < 0) {
+        for (int i = 0; i < inputChannels.size(); i++) {
+          final Tracker tracker = trackers.get(i);
+          if (tracker.hasCompleteSetForMark() || (pushNextFrame(i) && tracker.hasCompleteSetForMark())) {
+            trackerWithCompleteSetForCurrentKey = i;
+            break;
+          }
+        }
+
+        if (trackerWithCompleteSetForCurrentKey < 0) {
+          // Algorithm cannot proceed; fetch more frames.
+          return nextAwait();
+        }
+      }
+
+      if (match || (markCmp <= 0 && joinType.isLefty()) || (markCmp >= 0 && joinType.isRighty())) {
+        // Emit row, if there's room in the current frameWriter.
+        joinColumnSelectorFactory.cmp = markCmp;
+        joinColumnSelectorFactory.match = match;
+
+        if (!frameWriter.addSelection()) {
+          if (frameWriter.getNumRows() > 0) {
+            // Out of space in the current frame. Run again without moving cursors.
+            flushCurrentFrame();
+            return ReturnOrAwait.runAgain();
+          } else {
+            throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
+          }
+        }
+      }
+
+      // Advance one or both trackers.
+      if (match) {
+        // Matching keys. First advance the tracker with the complete set.
+        final Tracker tracker = trackers.get(trackerWithCompleteSetForCurrentKey);
+        final Tracker otherTracker = trackers.get(trackerWithCompleteSetForCurrentKey == LEFT ? RIGHT : LEFT);
+
+        tracker.advance();
+        if (!tracker.isCurrentSameKeyAsMark()) {
+          // Reached end of complete set. Advance the other tracker.
+          otherTracker.advance();
+
+          // On next iteration (when we're sure to have data) either rewind the complete-set tracker, or update marks
+          // of both, as appropriate.
+          onNextIteration(() -> {
+            if (otherTracker.isCurrentSameKeyAsMark()) {
+              otherTracker.markCurrent(); // Set mark to enable cleanup of old frames.
+              tracker.rewindToMark();
+            } else {
+              // Reached end of the other side too. Advance marks on both trackers.
+              tracker.markCurrent();
+              otherTracker.markCurrent();
+              trackerWithCompleteSetForCurrentKey = -1;
+            }
+          });
+        }
+      } else {
+        final int trackerToAdvance;
+
+        if (markCmp < 0) {
+          trackerToAdvance = LEFT;
+        } else if (markCmp > 0) {
+          trackerToAdvance = RIGHT;
+        } else {
+          // Key is null on both sides. Note that there is a preference for running through the left side first
+          // on a FULL join. It doesn't really matter which side we run through first, but we do need to be consistent
+          // for the benefit of the logic in "shouldEmitColumnValue".
+          trackerToAdvance = joinType.isLefty() ? LEFT : RIGHT;
+        }
+
+        final Tracker tracker = trackers.get(trackerToAdvance);
+
+        tracker.advance();
+
+        // On next iteration (when we're sure to have data), update mark if the key changed.
+        onNextIteration(() -> {
+          if (!tracker.isCurrentSameKeyAsMark()) {
+            tracker.markCurrent();
+            trackerWithCompleteSetForCurrentKey = -1;
+          }
+        });
+      }
+    }
+
+    if (trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd()) {
+      // Both channels completely done.
+      flushCurrentFrame();
+      return ReturnOrAwait.returnObject(0L);
+    } else {
+      // Keep reading.
+      return nextAwait();
+    }
+  }
+
+  @Override
+  public void cleanup() throws IOException
+  {
+    FrameProcessors.closeAll(inputChannels(), outputChannels(), frameWriter, () -> trackers.forEach(Tracker::clear));
+  }
+
+  /**
+   * Returns a {@link ReturnOrAwait#awaitAll} for the channel numbers that need more data and have not yet hit their
+   * buffered-bytes limit, {@link Limits#MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN}.
+   *
+   * If all channels have hit their limit, throws {@link MSQException} with {@link TooManyRowsWithSameKeyFault}.
+   */
+  private ReturnOrAwait<Long> nextAwait()
+  {
+    final IntSet awaitSet = new IntOpenHashSet();
+    int trackerAtLimit = -1;
+
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.needsMoreData()) {
+        if (tracker.totalBytesBuffered() < Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+          awaitSet.add(i);
+        } else if (trackerAtLimit < 0) {
+          trackerAtLimit = i;
+        }
+      }
+    }
+
+    if (awaitSet.isEmpty() && trackerAtLimit > 0) {
+      // All trackers that need more data are at their max buffered bytes limit. Generate a nice exception.
+      final Tracker tracker = trackers.get(trackerAtLimit);
+      if (tracker.totalBytesBuffered() > Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+        // Generate a nice exception.
+        throw new MSQException(
+            new TooManyRowsWithSameKeyFault(
+                tracker.readMarkKey(),
+                tracker.totalBytesBuffered(),
+                Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN
+            )
+        );
+      }
+    }
+
+    return ReturnOrAwait.awaitAll(awaitSet);
+  }
+
+  /**
+   * Compares the marked rows of the two {@link #trackers}.
+   *
+   * @throws IllegalStateException if either tracker does not have a marked row and is not completely done
+   */
+  private int compareMarks()
+  {
+    final Tracker leftTracker = trackers.get(LEFT);
+    final Tracker rightTracker = trackers.get(RIGHT);
+
+    Preconditions.checkState(leftTracker.hasMark() || leftTracker.isAtEnd(), "left.hasMark || left.isAtEnd");
+    Preconditions.checkState(rightTracker.hasMark() || rightTracker.isAtEnd(), "right.hasMark || right.isAtEnd");
+
+    if (leftTracker.markFrame < 0) {
+      return rightTracker.markFrame < 0 ? 0 : 1;
+    } else if (rightTracker.markFrame < 0) {
+      return -1;
+    } else {
+      final FrameHolder leftHolder = leftTracker.holders.get(leftTracker.markFrame);
+      final FrameHolder rightHolder = rightTracker.holders.get(rightTracker.markFrame);
+      return leftHolder.comparisonWidget.compare(
+          leftTracker.markRow,
+          rightHolder.comparisonWidget,
+          rightTracker.markRow
+      );
+    }
+  }
+
+  /**
+   * Pushes a frame from the indicated channel into the appropriate tracker. Returns true if a frame was pushed
+   * or if the channel is finished.
+   */
+  private boolean pushNextFrame(final int channelNumber)
+  {
+    final ReadableFrameChannel channel = inputChannels.get(channelNumber);
+    final Tracker tracker = trackers.get(channelNumber);
+
+    if (!channel.isFinished() && !channel.canRead()) {
+      return false;
+    } else if (channel.isFinished()) {
+      tracker.push(null);
+      return true;
+    } else {
+      final Frame frame = channel.read();
+
+      if (frame.numRows() == 0) {
+        // Skip, read next.
+        return false;
+      } else {
+        tracker.push(frame);
+        return true;
+      }
+    }
+  }
+
+  private void onNextIteration(final Runnable runnable)
+  {
+    if (nextIterationRunnable != null) {
+      throw new ISE("postAdvanceRunnable already set");
+    } else {
+      nextIterationRunnable = runnable;
+    }
+  }
+
+  private void startNewFrameIfNeeded()
+  {
+    if (frameWriter == null) {
+      frameWriter = frameWriterFactory.newFrameWriter(joinColumnSelectorFactory);
+    }
+  }
+
+  private void flushCurrentFrame() throws IOException
+  {
+    if (frameWriter != null) {
+      if (frameWriter.getNumRows() > 0) {
+        final Frame frame = Frame.wrap(frameWriter.toByteArray());
+        frameWriter.close();
+        frameWriter = null;
+        outputChannel.write(new FrameWithPartition(frame, FrameWithPartition.NO_PARTITION));
+      }
+    }
+  }
+
+  /**
+   * Tracks the current set of rows that have the same key from a sequence of frames.
+   *
+   * markFrame and markRow are set when we encounter a new key, which enables rewinding and re-reading data with the
+   * same key.
+   */
+  private class Tracker
+  {
+    private final List<FrameHolder> holders = new ArrayList<>();
+    private final int channelNumber;
+
+    // markFrame and markRow are the first frame and row with the current key.
+    private int markFrame = -1;
+    private int markRow = -1;
+
+    // currentFrame is the frame containing the current cursor row.
+    private int currentFrame = -1;
+
+    // done indicates that no more data is available in the channel.
+    private boolean done;
+
+    public Tracker(int channelNumber)
+    {
+      this.channelNumber = channelNumber;
+    }
+
+    /**
+     * Adds a holder for a frame. If this is the first frame, sets the current cursor position and mark to the first
+     * row of the frame. Otherwise, the cursor position and mark are not changed.
+     *
+     * Pushing a null frame indicates no more frames are coming.
+     *
+     * @param frame frame, or null indicating no more frames are coming
+     */
+    public void push(final Frame frame)
+    {
+      if (frame == null) {
+        done = true;
+        return;
+      }
+
+      if (done) {
+        throw new ISE("Cannot push frames when already done");
+      }
+
+      final boolean atEndOfPushedData = isAtEndOfPushedData();
+      final FrameReader frameReader = inputs.get(channelNumber).getChannelFrameReader();
+      final FrameCursor cursor = FrameProcessors.makeCursor(frame, frameReader);
+      final FrameComparisonWidget comparisonWidget =
+          frameReader.makeComparisonWidget(frame, keyColumns.get(channelNumber));
+
+      final RowSignature.Builder keySignatureBuilder = RowSignature.builder();
+      for (final KeyColumn keyColumn : keyColumns.get(channelNumber)) {
+        keySignatureBuilder.add(
+            keyColumn.columnName(),
+            frameReader.signature().getColumnType(keyColumn.columnName()).orElse(null)
+        );
+      }
+
+      holders.add(
+          new FrameHolder(
+              frame,
+              RowKeyReader.create(keySignatureBuilder.build()),
+              cursor,
+              comparisonWidget
+          )
+      );
+
+      if (atEndOfPushedData) {
+        // Move currentFrame so it points at the next row, which we now have, instead of an "isDone" cursor.
+        currentFrame = currentFrame < 0 ? 0 : currentFrame + 1;
+      }
+
+      if (markFrame < 0) {
+        // Cleared mark means we want the current row to be marked.
+        markFrame = currentFrame;
+        markRow = 0;
+      }
+    }
+
+    /**
+     * Number of bytes currently buffered in {@link #holders}.
+     */
+    public long totalBytesBuffered()
+    {
+      long bytes = 0;
+      for (final FrameHolder holder : holders) {
+        bytes += holder.frame.numBytes();
+      }
+      return bytes;
+    }
+
+    /**
+     * Cursor containing the current row.
+     */
+    @Nullable
+    public FrameCursor currentCursor()
+    {
+      if (currentFrame < 0) {
+        return null;
+      } else {
+        return holders.get(currentFrame).cursor;
+      }
+    }
+
+    /**
+     * Advances the current row (the current row of {@link #currentFrame}). After calling this method,
+     * {@link #isAtEndOfPushedData()} may start returning true.
+     */
+    public void advance()
+    {
+      assert !isAtEndOfPushedData();
+
+      final FrameHolder currentHolder = holders.get(currentFrame);
+
+      currentHolder.cursor.advance();
+
+      if (currentHolder.cursor.isDone() && currentFrame + 1 < holders.size()) {
+        currentFrame++;
+        holders.get(currentFrame).cursor.reset();
+      }
+    }
+
+    /**
+     * Whether this tracker has a marked row.
+     */
+    public boolean hasMark()
+    {
+      return markFrame >= 0;
+    }
+
+    /**
+     * Whether this tracker has a marked row that is partially null.
+     */
+    public boolean hasPartiallyNullMark()
+    {
+      return hasMark() && holders.get(markFrame).comparisonWidget.isPartiallyNullKey(markRow);
+    }
+
+    /**
+     * Reads the current marked key.
+     */
+    @Nullable
+    public List<Object> readMarkKey()
+    {
+      if (!hasMark()) {
+        return null;
+      }
+
+      final FrameHolder markHolder = holders.get(markFrame);
+      final RowKey markKey = markHolder.comparisonWidget.readKey(markRow);
+      return markHolder.keyReader.read(markKey);
+    }
+
+    /**
+     * Rewind to the mark row: the first one with the current key.
+     *
+     * @throws IllegalStateException if there is no marked row
+     */
+    public void rewindToMark()
+    {
+      if (markFrame < 0) {
+        throw new ISE("No mark");
+      }
+
+      currentFrame = markFrame;
+      holders.get(currentFrame).cursor.setCurrentRow(markRow);
+    }
+
+    /**
+     * Set the mark row to the current row. Used when data from the old mark to the current row is no longer needed.
+     */
+    public void markCurrent()
+    {
+      if (isAtEndOfPushedData()) {
+        clear();
+      } else {
+        // Remove unnecessary holders.
+        while (currentFrame > 0) {
+          if (currentFrame == holders.size() - 1) {
+            final FrameHolder lastHolder = holders.get(currentFrame);
+            holders.clear();
+            holders.add(lastHolder);
+            currentFrame = 0;
+          } else {
+            holders.remove(0);
+            currentFrame--;
+          }
+        }
+
+        markFrame = 0;
+        markRow = holders.get(currentFrame).cursor.getCurrentRow();
+      }
+    }
+
+    /**
+     * Whether the current cursor is past the end of the last frame for which we have data.
+     */
+    public boolean isAtEndOfPushedData()
+    {
+      return currentFrame < 0 || (currentFrame == holders.size() - 1 && holders.get(currentFrame).cursor.isDone());
+    }
+
+    /**
+     * Whether the current cursor is past the end of all data that will ever be pushed.
+     */
+    public boolean isAtEnd()
+    {
+      return done && isAtEndOfPushedData();
+    }
+
+    /**
+     * Whether this tracker needs more data in order to read the current cursor location or move it forward.
+     */
+    public boolean needsMoreData()
+    {
+      return !done && isAtEndOfPushedData();
+    }
+
+    /**
+     * Whether this tracker contains all rows for the marked key.
+     *
+     * @throws IllegalStateException if there is no marked key
+     */
+    public boolean hasCompleteSetForMark()
+    {
+      if (markFrame < 0) {
+        throw new ISE("No mark");
+      }
+
+      if (done) {
+        return true;
+      }
+
+      final FrameHolder lastHolder = holders.get(holders.size() - 1);
+      return !isSameKeyAsMark(lastHolder, lastHolder.frame.numRows() - 1);
+    }
+
+    /**
+     * Whether the current position (the current row of the {@link #currentFrame}) compares equally to the mark row.
+     * If {@link #isAtEnd()}, returns true iff there is no mark row.
+     */
+    public boolean isCurrentSameKeyAsMark()
+    {
+      if (isAtEnd()) {
+        return markFrame < 0;
+      } else {
+        assert !isAtEndOfPushedData();
+        final FrameHolder headHolder = holders.get(currentFrame);
+        return isSameKeyAsMark(headHolder, headHolder.cursor.getCurrentRow());
+      }
+    }
+
+    /**
+     * Clears the current mark and all buffered frames. Does not change {@link #done}.
+     */
+    public void clear()
+    {
+      holders.clear();
+      markFrame = -1;
+      markRow = -1;
+      currentFrame = -1;
+    }
+
+    /**
+     * Whether the provided frame and row compares equally to the mark row. The provided row must be at, or after,
+     * the mark row.
+     */
+    private boolean isSameKeyAsMark(final FrameHolder holder, final int row)
+    {
+      // Mark row must exist.
+      assert markFrame >= 0;
+
+      // Row must exist.
+      assert row >= 0 && row < holder.frame.numRows();
+
+      final FrameHolder markHolder = holders.get(markFrame);
+      final int cmp = markHolder.comparisonWidget.compare(markRow, holder.comparisonWidget, row);
+
+      assert cmp <= 0;

Review Comment:
   It was important to me during development 🙂
   
   I replaced these with if statements.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1127458820


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/HashShuffleSpec.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.kernel;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.frame.key.ClusterBy;
+import org.apache.druid.frame.key.ClusterByPartitions;
+import org.apache.druid.java.util.common.Either;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.msq.statistics.ClusterByStatisticsCollector;
+
+import javax.annotation.Nullable;
+
+public class HashShuffleSpec implements ShuffleSpec
+{
+  public static final String TYPE = "hash";
+
+  private final ClusterBy clusterBy;
+  private final int numPartitions;
+
+  @JsonCreator
+  public HashShuffleSpec(
+      @JsonProperty("clusterBy") final ClusterBy clusterBy,
+      @JsonProperty("partitions") final int numPartitions
+  )
+  {
+    this.clusterBy = clusterBy;
+    this.numPartitions = numPartitions;
+
+    if (clusterBy.getBucketByCount() > 0) {
+      // Only GlobalSortTargetSizeShuffleSpec supports bucket-by.
+      throw new IAE("Cannot bucket with %s partitioning", TYPE);
+    }

Review Comment:
   > Is there no way to make it so that the only thing that supports it is the only thing that takes it?
   
   It could be done. It makes this code cleaner but then makes other code elsewhere dirtier. On balance I think it's better this way.
   
   > Additionally, if you were to interpolate the clusterBy into the error message, would it generate something that tells the user which part of their query is doing weird things?
   
   If the planner is non-buggy then there would be no way a user sees this. (Unless they write their own MSQ spec, without going through SQL.) That being said, I think we might as well include `clusterBy` here to help debug any potential planner bugs. I added it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1042622786


##########
processing/src/main/java/org/apache/druid/frame/key/KeyOrder.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.frame.key;
+
+public enum KeyOrder
+{
+  NONE(false),

Review Comment:
   Hmm. What I was thinking is not that "none" means "not ordered" (in ref to the enum name KeyOrder). Like, a "none" ordered thing might be used as a hash code, or it might not. I'll clarify this with a comment. If you still think it should be HASHED, let me know so I can think about it more.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1043395582


##########
docs/multi-stage-query/reference.md:
##########
@@ -198,13 +198,99 @@ The following table lists the context parameters for the MSQ task engine:
 | `maxNumTasks` | SELECT, INSERT, REPLACE<br /><br />The maximum total number of tasks to launch, including the controller task. The lowest possible value for this setting is 2: one controller and one worker. All tasks must be able to launch simultaneously. If they cannot, the query returns a `TaskStartTimeout` error code after approximately 10 minutes.<br /><br />May also be provided as `numTasks`. If both are present, `maxNumTasks` takes priority.| 2 |
 | `taskAssignment` | SELECT, INSERT, REPLACE<br /><br />Determines how many tasks to use. Possible values include: <ul><li>`max`: Uses as many tasks as possible, up to `maxNumTasks`.</li><li>`auto`: When file sizes can be determined through directory listing (for example: local files, S3, GCS, HDFS) uses as few tasks as possible without exceeding 10 GiB or 10,000 files per task, unless exceeding these limits is necessary to stay within `maxNumTasks`. When file sizes cannot be determined through directory listing (for example: http), behaves the same as `max`.</li></ul> | `max` |
 | `finalizeAggregations` | SELECT, INSERT, REPLACE<br /><br />Determines the type of aggregation to return. If true, Druid finalizes the results of complex aggregations that directly appear in query results. If false, Druid returns the aggregation's intermediate type rather than finalized type. This parameter is useful during ingestion, where it enables storing sketches directly in Druid tables. For more information about aggregations, see [SQL aggregation functions](../querying/sql-aggregations.md). | true |
+| `sqlJoinAlgorithm` | SELECT, INSERT, REPLACE<br /><br />Algorithm to use for JOIN. Use `broadcast` (the default) for broadcast hash join or `sortMerge` for sort-merge join. Affects all JOIN operations in the query. See [Joins](#joins) for more details. | `broadcast` |

Review Comment:
   @gianm - Here it is. https://github.com/apache/druid/issues/13532. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] lgtm-com[bot] commented on pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13506:
URL: https://github.com/apache/druid/pull/13506#issuecomment-1341914743

   This pull request **fixes 1 alert** when merging 4514d2363f4324baa3fa2f559a18c62bde10cabf into b56855b837b73644c080f5305b2726b4518e10f5 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-dd42945bca19f1da51f53a1353b2a6e266df9c8c)
   
   **fixed alerts:**
   
   * 1 for Uncontrolled data used in path expression
   ---
   
   **Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1114156011


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;

Review Comment:
   As to the lists vs. using 2 fields: I was thinking that someday this code would be extended to handle joining more than two channels at once.
   
   As to the container thingy: I decided to put most of that stuff into the Tracker, which is already a kind of container. Except for `inputChannels`, because that's used as a return value for `inputChannels()`. It's a bit more readable, IMO.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1114174262


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Names of the key columns on each side of the join, unprefixed. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<List<KeyColumn>> keyColumns;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputs = ImmutableList.of(left, right);
+    this.inputChannels = inputs.stream().map(ReadableInput::getChannel).collect(Collectors.toList());
+    this.keyColumns = keyColumns;
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(new Tracker(LEFT), new Tracker(RIGHT));
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()
+           && !(trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd())) {
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+      final boolean match = markCmp == 0 && !trackers.get(LEFT).hasPartiallyNullMark();

Review Comment:
   It's to achieve proper join semantics. I added a comment:
   
   ```
         // Two rows match if the keys compare equal _and_ neither key has a null component. (x JOIN y ON x.a = y.a does
         // not match rows where "x.a" is null.)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1114188415


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Names of the key columns on each side of the join, unprefixed. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<List<KeyColumn>> keyColumns;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputs = ImmutableList.of(left, right);
+    this.inputChannels = inputs.stream().map(ReadableInput::getChannel).collect(Collectors.toList());
+    this.keyColumns = keyColumns;
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(new Tracker(LEFT), new Tracker(RIGHT));
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()
+           && !(trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd())) {
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+      final boolean match = markCmp == 0 && !trackers.get(LEFT).hasPartiallyNullMark();
+
+      // Fetch new frames until the algorithm can proceed:
+      //   1) If marked keys are equal on both sides, at least one side has a complete set of rows for the marked key.
+      //   2) Otherwise, each side has at least one readable row, or is completely done. (Checked by "while" condition.)
+      if (match && trackerWithCompleteSetForCurrentKey < 0) {
+        for (int i = 0; i < inputChannels.size(); i++) {
+          final Tracker tracker = trackers.get(i);
+          if (tracker.hasCompleteSetForMark() || (pushNextFrame(i) && tracker.hasCompleteSetForMark())) {
+            trackerWithCompleteSetForCurrentKey = i;
+            break;
+          }
+        }
+
+        if (trackerWithCompleteSetForCurrentKey < 0) {
+          // Algorithm cannot proceed; fetch more frames.
+          return nextAwait();
+        }
+      }
+
+      if (match || (markCmp <= 0 && joinType.isLefty()) || (markCmp >= 0 && joinType.isRighty())) {
+        // Emit row, if there's room in the current frameWriter.
+        joinColumnSelectorFactory.cmp = markCmp;
+        joinColumnSelectorFactory.match = match;
+
+        if (!frameWriter.addSelection()) {
+          if (frameWriter.getNumRows() > 0) {
+            // Out of space in the current frame. Run again without moving cursors.
+            flushCurrentFrame();
+            return ReturnOrAwait.runAgain();
+          } else {
+            throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
+          }
+        }
+      }
+
+      // Advance one or both trackers.
+      if (match) {
+        // Matching keys. First advance the tracker with the complete set.
+        final Tracker tracker = trackers.get(trackerWithCompleteSetForCurrentKey);
+        final Tracker otherTracker = trackers.get(trackerWithCompleteSetForCurrentKey == LEFT ? RIGHT : LEFT);
+
+        tracker.advance();
+        if (!tracker.isCurrentSameKeyAsMark()) {
+          // Reached end of complete set. Advance the other tracker.
+          otherTracker.advance();
+
+          // On next iteration (when we're sure to have data) either rewind the complete-set tracker, or update marks
+          // of both, as appropriate.
+          onNextIteration(() -> {
+            if (otherTracker.isCurrentSameKeyAsMark()) {
+              otherTracker.markCurrent(); // Set mark to enable cleanup of old frames.
+              tracker.rewindToMark();
+            } else {
+              // Reached end of the other side too. Advance marks on both trackers.
+              tracker.markCurrent();
+              otherTracker.markCurrent();
+              trackerWithCompleteSetForCurrentKey = -1;
+            }
+          });
+        }
+      } else {
+        final int trackerToAdvance;
+
+        if (markCmp < 0) {
+          trackerToAdvance = LEFT;
+        } else if (markCmp > 0) {
+          trackerToAdvance = RIGHT;
+        } else {
+          // Key is null on both sides. Note that there is a preference for running through the left side first
+          // on a FULL join. It doesn't really matter which side we run through first, but we do need to be consistent
+          // for the benefit of the logic in "shouldEmitColumnValue".
+          trackerToAdvance = joinType.isLefty() ? LEFT : RIGHT;
+        }
+
+        final Tracker tracker = trackers.get(trackerToAdvance);
+
+        tracker.advance();
+
+        // On next iteration (when we're sure to have data), update mark if the key changed.
+        onNextIteration(() -> {
+          if (!tracker.isCurrentSameKeyAsMark()) {
+            tracker.markCurrent();
+            trackerWithCompleteSetForCurrentKey = -1;
+          }
+        });
+      }
+    }
+
+    if (trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd()) {
+      // Both channels completely done.
+      flushCurrentFrame();
+      return ReturnOrAwait.returnObject(0L);
+    } else {
+      // Keep reading.
+      return nextAwait();
+    }
+  }
+
+  @Override
+  public void cleanup() throws IOException
+  {
+    FrameProcessors.closeAll(inputChannels(), outputChannels(), frameWriter, () -> trackers.forEach(Tracker::clear));
+  }
+
+  /**
+   * Returns a {@link ReturnOrAwait#awaitAll} for the channel numbers that need more data and have not yet hit their
+   * buffered-bytes limit, {@link Limits#MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN}.
+   *
+   * If all channels have hit their limit, throws {@link MSQException} with {@link TooManyRowsWithSameKeyFault}.
+   */
+  private ReturnOrAwait<Long> nextAwait()
+  {
+    final IntSet awaitSet = new IntOpenHashSet();
+    int trackerAtLimit = -1;
+
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.needsMoreData()) {
+        if (tracker.totalBytesBuffered() < Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+          awaitSet.add(i);
+        } else if (trackerAtLimit < 0) {
+          trackerAtLimit = i;
+        }
+      }
+    }
+
+    if (awaitSet.isEmpty() && trackerAtLimit > 0) {
+      // All trackers that need more data are at their max buffered bytes limit. Generate a nice exception.
+      final Tracker tracker = trackers.get(trackerAtLimit);
+      if (tracker.totalBytesBuffered() > Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+        // Generate a nice exception.
+        throw new MSQException(
+            new TooManyRowsWithSameKeyFault(
+                tracker.readMarkKey(),
+                tracker.totalBytesBuffered(),
+                Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN
+            )
+        );
+      }
+    }
+
+    return ReturnOrAwait.awaitAll(awaitSet);
+  }
+
+  /**
+   * Compares the marked rows of the two {@link #trackers}.
+   *
+   * @throws IllegalStateException if either tracker does not have a marked row and is not completely done
+   */
+  private int compareMarks()
+  {
+    final Tracker leftTracker = trackers.get(LEFT);
+    final Tracker rightTracker = trackers.get(RIGHT);
+
+    Preconditions.checkState(leftTracker.hasMark() || leftTracker.isAtEnd(), "left.hasMark || left.isAtEnd");
+    Preconditions.checkState(rightTracker.hasMark() || rightTracker.isAtEnd(), "right.hasMark || right.isAtEnd");
+
+    if (leftTracker.markFrame < 0) {
+      return rightTracker.markFrame < 0 ? 0 : 1;
+    } else if (rightTracker.markFrame < 0) {
+      return -1;
+    } else {
+      final FrameHolder leftHolder = leftTracker.holders.get(leftTracker.markFrame);
+      final FrameHolder rightHolder = rightTracker.holders.get(rightTracker.markFrame);
+      return leftHolder.comparisonWidget.compare(
+          leftTracker.markRow,
+          rightHolder.comparisonWidget,
+          rightTracker.markRow
+      );
+    }
+  }
+
+  /**
+   * Pushes a frame from the indicated channel into the appropriate tracker. Returns true if a frame was pushed
+   * or if the channel is finished.
+   */
+  private boolean pushNextFrame(final int channelNumber)
+  {
+    final ReadableFrameChannel channel = inputChannels.get(channelNumber);
+    final Tracker tracker = trackers.get(channelNumber);
+
+    if (!channel.isFinished() && !channel.canRead()) {
+      return false;
+    } else if (channel.isFinished()) {
+      tracker.push(null);
+      return true;
+    } else {
+      final Frame frame = channel.read();
+
+      if (frame.numRows() == 0) {
+        // Skip, read next.
+        return false;
+      } else {
+        tracker.push(frame);
+        return true;
+      }
+    }
+  }
+
+  private void onNextIteration(final Runnable runnable)
+  {
+    if (nextIterationRunnable != null) {
+      throw new ISE("postAdvanceRunnable already set");
+    } else {
+      nextIterationRunnable = runnable;
+    }
+  }
+
+  private void startNewFrameIfNeeded()
+  {
+    if (frameWriter == null) {
+      frameWriter = frameWriterFactory.newFrameWriter(joinColumnSelectorFactory);
+    }
+  }
+
+  private void flushCurrentFrame() throws IOException
+  {
+    if (frameWriter != null) {
+      if (frameWriter.getNumRows() > 0) {
+        final Frame frame = Frame.wrap(frameWriter.toByteArray());
+        frameWriter.close();
+        frameWriter = null;
+        outputChannel.write(new FrameWithPartition(frame, FrameWithPartition.NO_PARTITION));
+      }
+    }
+  }
+
+  /**
+   * Tracks the current set of rows that have the same key from a sequence of frames.
+   *
+   * markFrame and markRow are set when we encounter a new key, which enables rewinding and re-reading data with the
+   * same key.
+   */
+  private class Tracker
+  {
+    private final List<FrameHolder> holders = new ArrayList<>();
+    private final int channelNumber;
+
+    // markFrame and markRow are the first frame and row with the current key.
+    private int markFrame = -1;
+    private int markRow = -1;
+
+    // currentFrame is the frame containing the current cursor row.
+    private int currentFrame = -1;
+
+    // done indicates that no more data is available in the channel.
+    private boolean done;
+
+    public Tracker(int channelNumber)
+    {
+      this.channelNumber = channelNumber;
+    }
+
+    /**
+     * Adds a holder for a frame. If this is the first frame, sets the current cursor position and mark to the first
+     * row of the frame. Otherwise, the cursor position and mark are not changed.
+     *
+     * Pushing a null frame indicates no more frames are coming.
+     *
+     * @param frame frame, or null indicating no more frames are coming
+     */
+    public void push(final Frame frame)
+    {
+      if (frame == null) {
+        done = true;
+        return;
+      }
+
+      if (done) {
+        throw new ISE("Cannot push frames when already done");
+      }
+
+      final boolean atEndOfPushedData = isAtEndOfPushedData();
+      final FrameReader frameReader = inputs.get(channelNumber).getChannelFrameReader();
+      final FrameCursor cursor = FrameProcessors.makeCursor(frame, frameReader);
+      final FrameComparisonWidget comparisonWidget =
+          frameReader.makeComparisonWidget(frame, keyColumns.get(channelNumber));
+
+      final RowSignature.Builder keySignatureBuilder = RowSignature.builder();
+      for (final KeyColumn keyColumn : keyColumns.get(channelNumber)) {
+        keySignatureBuilder.add(
+            keyColumn.columnName(),
+            frameReader.signature().getColumnType(keyColumn.columnName()).orElse(null)
+        );
+      }
+
+      holders.add(
+          new FrameHolder(
+              frame,
+              RowKeyReader.create(keySignatureBuilder.build()),
+              cursor,
+              comparisonWidget
+          )
+      );
+
+      if (atEndOfPushedData) {
+        // Move currentFrame so it points at the next row, which we now have, instead of an "isDone" cursor.
+        currentFrame = currentFrame < 0 ? 0 : currentFrame + 1;
+      }
+
+      if (markFrame < 0) {
+        // Cleared mark means we want the current row to be marked.
+        markFrame = currentFrame;
+        markRow = 0;
+      }
+    }
+
+    /**
+     * Number of bytes currently buffered in {@link #holders}.
+     */
+    public long totalBytesBuffered()
+    {
+      long bytes = 0;
+      for (final FrameHolder holder : holders) {
+        bytes += holder.frame.numBytes();
+      }
+      return bytes;
+    }
+
+    /**
+     * Cursor containing the current row.
+     */
+    @Nullable
+    public FrameCursor currentCursor()
+    {
+      if (currentFrame < 0) {
+        return null;
+      } else {
+        return holders.get(currentFrame).cursor;
+      }
+    }
+
+    /**
+     * Advances the current row (the current row of {@link #currentFrame}). After calling this method,
+     * {@link #isAtEndOfPushedData()} may start returning true.
+     */
+    public void advance()
+    {
+      assert !isAtEndOfPushedData();
+
+      final FrameHolder currentHolder = holders.get(currentFrame);
+
+      currentHolder.cursor.advance();
+
+      if (currentHolder.cursor.isDone() && currentFrame + 1 < holders.size()) {
+        currentFrame++;
+        holders.get(currentFrame).cursor.reset();
+      }

Review Comment:
   There's a `holders.clear()` in `markCurrent()`. We can't get rid of the old holders until the mark moves on. (The purpose of holding on to them is to enable us to collect a complete set of rows for the marked key.)
   
   I added comments to `holders` clarifying this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1127468735


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/HashShuffleSpec.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.kernel;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.frame.key.ClusterBy;
+import org.apache.druid.frame.key.ClusterByPartitions;
+import org.apache.druid.java.util.common.Either;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.msq.statistics.ClusterByStatisticsCollector;
+
+import javax.annotation.Nullable;
+
+public class HashShuffleSpec implements ShuffleSpec
+{
+  public static final String TYPE = "hash";
+
+  private final ClusterBy clusterBy;
+  private final int numPartitions;
+
+  @JsonCreator
+  public HashShuffleSpec(
+      @JsonProperty("clusterBy") final ClusterBy clusterBy,
+      @JsonProperty("partitions") final int numPartitions
+  )
+  {
+    this.clusterBy = clusterBy;
+    this.numPartitions = numPartitions;
+
+    if (clusterBy.getBucketByCount() > 0) {
+      // Only GlobalSortTargetSizeShuffleSpec supports bucket-by.
+      throw new IAE("Cannot bucket with %s partitioning", TYPE);
+    }
+  }
+
+  @Override
+  public ShuffleKind kind()
+  {
+    return clusterBy.sortable() && !clusterBy.isEmpty() ? ShuffleKind.HASH_LOCAL_SORT : ShuffleKind.HASH;
+  }
+
+  @Override
+  @JsonProperty
+  public ClusterBy clusterBy()
+  {
+    return clusterBy;
+  }
+
+  @Override
+  public boolean doesAggregate()
+  {
+    return false;
+  }
+
+  @Override
+  public boolean needsStatistics()
+  {
+    return false;
+  }
+
+  @Override
+  public Either<Long, ClusterByPartitions> generatePartitionsForGlobalSort(
+      @Nullable ClusterByStatisticsCollector collector,
+      int maxNumPartitions
+  )
+  {
+    throw new IllegalStateException("Not a global sort");
+  }

Review Comment:
   Hmm. Yes. I think it would need some kind of `.class` or casting stuff.
   
   I rejiggered things to have a specialized GlobalSortShuffleSpec interface that only the global-sort stuff implements. Calling code knows to check for this interface if the kind is `GLOBAL_SORT`. I think it's a bit cleaner. Thanks for the suggestion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1127470521


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java:
##########
@@ -288,37 +322,72 @@ public boolean mustGatherResultKeyStatistics()
     return shuffleSpec != null && shuffleSpec.needsStatistics();
   }
 
-  public Either<Long, ClusterByPartitions> generatePartitionsForShuffle(
+  public Either<Long, ClusterByPartitions> generatePartitionBoundariesForShuffle(
       @Nullable ClusterByStatisticsCollector collector
   )
   {
     if (shuffleSpec == null) {
       throw new ISE("No shuffle for stage[%d]", getStageNumber());
+    } else if (shuffleSpec.kind() != ShuffleKind.GLOBAL_SORT) {
+      throw new ISE(
+          "Shuffle of kind [%s] cannot generate partition boundaries for stage[%d]",
+          shuffleSpec.kind(),
+          getStageNumber()
+      );
     } else if (mustGatherResultKeyStatistics() && collector == null) {
       throw new ISE("Statistics required, but not gathered for stage[%d]", getStageNumber());
     } else if (!mustGatherResultKeyStatistics() && collector != null) {
       throw new ISE("Statistics gathered, but not required for stage[%d]", getStageNumber());
     } else {
-      return shuffleSpec.generatePartitions(collector, MAX_PARTITIONS);
+      return shuffleSpec.generatePartitionsForGlobalSort(collector, MAX_PARTITIONS);
     }
   }
 
   public ClusterByStatisticsCollector createResultKeyStatisticsCollector(final int maxRetainedBytes)
   {
     if (!mustGatherResultKeyStatistics()) {
-      throw new ISE("No statistics needed");
+      throw new ISE("No statistics needed for stage[%d]", getStageNumber());
     }
 
     return ClusterByStatisticsCollectorImpl.create(
-        shuffleSpec.getClusterBy(),
+        shuffleSpec.clusterBy(),
         signature,
         maxRetainedBytes,
         PARTITION_STATS_MAX_BUCKETS,
-        shuffleSpec.doesAggregateByClusterKey(),
+        shuffleSpec.doesAggregate(),
         shuffleCheckHasMultipleValues
     );

Review Comment:
   It's specifically talking about gathering result key statistics for the purposes of generating partitions for the global sort. (It's populating sketches to figure out range cut points.) The user doesn't enter into it really. I updated some javadoc to hopefully be more clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1042611343


##########
docs/multi-stage-query/reference.md:
##########
@@ -198,13 +198,99 @@ The following table lists the context parameters for the MSQ task engine:
 | `maxNumTasks` | SELECT, INSERT, REPLACE<br /><br />The maximum total number of tasks to launch, including the controller task. The lowest possible value for this setting is 2: one controller and one worker. All tasks must be able to launch simultaneously. If they cannot, the query returns a `TaskStartTimeout` error code after approximately 10 minutes.<br /><br />May also be provided as `numTasks`. If both are present, `maxNumTasks` takes priority.| 2 |
 | `taskAssignment` | SELECT, INSERT, REPLACE<br /><br />Determines how many tasks to use. Possible values include: <ul><li>`max`: Uses as many tasks as possible, up to `maxNumTasks`.</li><li>`auto`: When file sizes can be determined through directory listing (for example: local files, S3, GCS, HDFS) uses as few tasks as possible without exceeding 10 GiB or 10,000 files per task, unless exceeding these limits is necessary to stay within `maxNumTasks`. When file sizes cannot be determined through directory listing (for example: http), behaves the same as `max`.</li></ul> | `max` |
 | `finalizeAggregations` | SELECT, INSERT, REPLACE<br /><br />Determines the type of aggregation to return. If true, Druid finalizes the results of complex aggregations that directly appear in query results. If false, Druid returns the aggregation's intermediate type rather than finalized type. This parameter is useful during ingestion, where it enables storing sketches directly in Druid tables. For more information about aggregations, see [SQL aggregation functions](../querying/sql-aggregations.md). | true |
+| `sqlJoinAlgorithm` | SELECT, INSERT, REPLACE<br /><br />Algorithm to use for JOIN. Use `broadcast` (the default) for broadcast hash join or `sortMerge` for sort-merge join. Affects all JOIN operations in the query. See [Joins](#joins) for more details. | `broadcast` |

Review Comment:
   Unfortunately, hints are not available in the version of Calcite version we use. Newer versions have this: https://calcite.apache.org/docs/reference.html#sql-hints
   
   @abhishekagarwal87 found some reasons we couldn't do it right now, as referenced by this comment: https://github.com/apache/druid/pull/13153#issuecomment-1261204764. @abhishekagarwal87 would you mind writing up your notes as to what blocks an upgrade, and making an issue about that, titled something like `Upgrade Calcite past <whatever version introduced the blocking problem>`? That way, we have an issue we can refer to and use to discuss possible ways to fix the blockers.
   
   Once we sort that out, I'd like to deprecate the context parameter and move things to use hints (and eventually statistics as well) instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1042613906


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java:
##########
@@ -595,8 +596,8 @@ public void updatePartialKeyStatisticsInformation(int stageNumber, int workerNum
           final StageDefinition stageDef = queryKernel.getStageDefinition(stageId);
           final ObjectMapper mapper = MSQTasks.decorateObjectMapperForKeyCollectorSnapshot(
               context.jsonMapper(),
-              stageDef.getShuffleSpec().get().getClusterBy(),
-              stageDef.getShuffleSpec().get().doesAggregateByClusterKey()
+              stageDef.getShuffleSpec().clusterBy(),
+              stageDef.getShuffleSpec().doesAggregate()

Review Comment:
   I don't love them for holder objects that don't "do" anything. Just personal preference I guess.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1042989583


##########
processing/src/main/java/org/apache/druid/frame/key/ClusterBy.java:
##########
@@ -58,6 +59,21 @@ public ClusterBy(
     if (bucketByCount < 0 || bucketByCount > columns.size()) {
       throw new IAE("Invalid bucketByCount [%d]", bucketByCount);
     }
+
+    // Key must be 100% sortable or 100% nonsortable. If empty, call it unsortable.

Review Comment:
   Is an empty key really unsortable?  An empty key would effectively be equivalent to a constant value, and all data sets are sorted by a constant value.



##########
processing/src/main/java/org/apache/druid/frame/field/StringFieldReader.java:
##########
@@ -87,6 +87,14 @@ public DimensionSelector makeDimensionSelector(
     return new Selector(memory, fieldPointer, extractionFn, false);
   }
 
+  @Override
+  public boolean isNull(Memory memory, long position)
+  {
+    return memory.getByte(position) == StringFieldWriter.NULL_BYTE
+           && memory.getByte(position + 1) == StringFieldWriter.VALUE_TERMINATOR
+           && memory.getByte(position + 2) == StringFieldWriter.ROW_TERMINATOR;

Review Comment:
   Maybe either read a 3-byte array and test for equality or, if there's padding, you could read an int, mask away a byte and test equals.



##########
processing/src/main/java/org/apache/druid/frame/key/KeyOrder.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.frame.key;
+
+public enum KeyOrder
+{
+  NONE(false),

Review Comment:
   In your description on this PR, I took away that adding NONE here allows you to use that to mean HASHED partitioning.
   
   Maybe I'm not understanding the point of this class, but I think I assumed it would only be instantiated if someone was effecting an ordering/clustering/partitioning of some sort?  I guess, even on the comment, what does it mean for the key to be used for non-sorting purposes, if the point of creating the object is for sort-type purposes?  The answer left to me is "well, then it's used for clustering/partitioning" in which case, I don't know of an algorithm other than HASH, so seems like we can just name it how we expect it to be used and then ask ourselves what to do when we come up with a new usage that doesn't fit the name HASH?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] lgtm-com[bot] commented on pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13506:
URL: https://github.com/apache/druid/pull/13506#issuecomment-1340312353

   This pull request **fixes 1 alert** when merging d2d56f32a1c57a3f74f792d84b2acd8d446837d7 into 83261f9641a750b121dbbb13b1cf53dd0c40f126 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-6ee3a4af29ae1d131ec498a73e40f2220d6d9e86)
   
   **fixed alerts:**
   
   * 1 for Uncontrolled data used in path expression
   ---
   
   **Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1114230688


##########
processing/src/main/java/org/apache/druid/frame/key/ClusterBy.java:
##########
@@ -58,6 +59,21 @@ public ClusterBy(
     if (bucketByCount < 0 || bucketByCount > columns.size()) {
       throw new IAE("Invalid bucketByCount [%d]", bucketByCount);
     }
+
+    // Key must be 100% sortable or 100% nonsortable. If empty, call it unsortable.

Review Comment:
   After thinking about it a bit, I changed this so an empty key is considered sortable, since I agree that makes more sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on PR #13506:
URL: https://github.com/apache/druid/pull/13506#issuecomment-1442379367

   > CodeQL found more than 10 potential problems in the proposed changes. Check the [Files changed](/apache/druid/pull/13506/files) tab for more details.
   
   Fixed a couple of unused parameters, and dismissed the rest, which looked like false positives. They were all about the `partition` field in FrameChannelHashPartitioner, which comes from HashPartitionVirtualColumn.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] github-code-scanning[bot] commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "github-code-scanning[bot] (via GitHub)" <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1116256993


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java:
##########
@@ -1370,6 +934,818 @@
     }
   }
 
+  /**
+   * Main worker logic for executing a {@link WorkOrder}.
+   */
+  private class RunWorkOrder
+  {
+    private final WorkerStageKernel kernel;
+    private final InputChannelFactory inputChannelFactory;
+    private final CounterTracker counterTracker;
+    private final FrameProcessorExecutor exec;
+    private final String cancellationId;
+    private final int parallelism;
+    private final FrameContext frameContext;
+    private final MSQWarningReportPublisher warningPublisher;
+
+    private InputSliceReader inputSliceReader;
+    private OutputChannelFactory workOutputChannelFactory;
+    private OutputChannelFactory shuffleOutputChannelFactory;
+    private ResultAndChannels<?> workResultAndOutputChannels;
+    private SettableFuture<ClusterByPartitions> stagePartitionBoundariesFuture;
+    private ListenableFuture<OutputChannels> shuffleOutputChannelsFuture;
+
+    public RunWorkOrder(
+        final WorkerStageKernel kernel,
+        final InputChannelFactory inputChannelFactory,
+        final CounterTracker counterTracker,
+        final FrameProcessorExecutor exec,
+        final String cancellationId,
+        final int parallelism,
+        final FrameContext frameContext,
+        final MSQWarningReportPublisher warningPublisher
+    )
+    {
+      this.kernel = kernel;
+      this.inputChannelFactory = inputChannelFactory;
+      this.counterTracker = counterTracker;
+      this.exec = exec;
+      this.cancellationId = cancellationId;
+      this.parallelism = parallelism;
+      this.frameContext = frameContext;
+      this.warningPublisher = warningPublisher;
+    }
+
+    private void start() throws IOException
+    {
+      final WorkOrder workOrder = kernel.getWorkOrder();
+      final StageDefinition stageDef = workOrder.getStageDefinition();
+
+      makeInputSliceReader();
+      makeWorkOutputChannelFactory();
+      makeShuffleOutputChannelFactory();
+      makeAndRunWorkProcessors();
+
+      if (stageDef.doesShuffle()) {
+        makeAndRunShuffleProcessors();
+      } else {
+        // No shuffling: work output _is_ shuffle output. Retain read-only versions to reduce memory footprint.
+        shuffleOutputChannelsFuture =
+            Futures.immediateFuture(workResultAndOutputChannels.getOutputChannels().readOnly());
+      }
+
+      setUpCompletionCallbacks();
+    }
+
+    /**
+     * Settable {@link ClusterByPartitions} future for global sort. Necessary because we don't know ahead of time
+     * what the boundaries will be. The controller decides based on statistics from all workers. Once the controller
+     * decides, its decision is written to this future, which allows sorting on workers to proceed.
+     */
+    @Nullable
+    public SettableFuture<ClusterByPartitions> getStagePartitionBoundariesFuture()
+    {
+      return stagePartitionBoundariesFuture;
+    }
+
+    private void makeInputSliceReader()
+    {
+      if (inputSliceReader != null) {
+        throw new ISE("inputSliceReader already created");
+      }
+
+      final WorkOrder workOrder = kernel.getWorkOrder();
+      final String queryId = workOrder.getQueryDefinition().getQueryId();
+
+      final InputChannels inputChannels =
+          new InputChannelsImpl(
+              workOrder.getQueryDefinition(),
+              InputSlices.allReadablePartitions(workOrder.getInputs()),
+              inputChannelFactory,
+              () -> ArenaMemoryAllocator.createOnHeap(frameContext.memoryParameters().getStandardFrameSize()),
+              exec,
+              cancellationId
+          );
+
+      inputSliceReader = new MapInputSliceReader(
+          ImmutableMap.<Class<? extends InputSlice>, InputSliceReader>builder()
+                      .put(NilInputSlice.class, NilInputSliceReader.INSTANCE)
+                      .put(StageInputSlice.class, new StageInputSliceReader(queryId, inputChannels))
+                      .put(ExternalInputSlice.class, new ExternalInputSliceReader(frameContext.tempDir()))
+                      .put(SegmentsInputSlice.class, new SegmentsInputSliceReader(frameContext.dataSegmentProvider()))
+                      .build()
+      );
+    }
+
+    private void makeWorkOutputChannelFactory()
+    {
+      if (workOutputChannelFactory != null) {
+        throw new ISE("processorOutputChannelFactory already created");
+      }
+
+      final OutputChannelFactory baseOutputChannelFactory;
+
+      if (kernel.getStageDefinition().doesShuffle()) {
+        // Writing to a consumer in the same JVM (which will be set up later on in this method). Use the large frame
+        // size if we're writing to a SuperSorter, since we'll generate fewer temp files if we use larger frames.
+        // Otherwise, use the standard frame size.
+        final int frameSize;
+
+        if (kernel.getStageDefinition().getShuffleSpec().kind().isSort()) {
+          frameSize = frameContext.memoryParameters().getLargeFrameSize();
+        } else {
+          frameSize = frameContext.memoryParameters().getStandardFrameSize();
+        }
+
+        baseOutputChannelFactory = new BlockingQueueOutputChannelFactory(frameSize);
+      } else {
+        // Writing stage output.
+        baseOutputChannelFactory =
+            makeStageOutputChannelFactory(frameContext, kernel.getStageDefinition().getStageNumber());
+      }
+
+      workOutputChannelFactory = new CountingOutputChannelFactory(
+          baseOutputChannelFactory,
+          counterTracker.channel(CounterNames.outputChannel())
+      );
+    }
+
+    private void makeShuffleOutputChannelFactory()
+    {
+      shuffleOutputChannelFactory =
+          new CountingOutputChannelFactory(
+              makeStageOutputChannelFactory(frameContext, kernel.getStageDefinition().getStageNumber()),
+              counterTracker.channel(CounterNames.shuffleChannel())
+          );
+    }
+
+    private <FactoryType extends FrameProcessorFactory<I, WorkerClass, T, R>, I, WorkerClass extends FrameProcessor<T>, T, R> void makeAndRunWorkProcessors()
+        throws IOException
+    {
+      if (workResultAndOutputChannels != null) {
+        throw new ISE("workResultAndOutputChannels already set");
+      }
+
+      @SuppressWarnings("unchecked")
+      final FactoryType processorFactory = (FactoryType) kernel.getStageDefinition().getProcessorFactory();
+
+      @SuppressWarnings("unchecked")
+      final ProcessorsAndChannels<WorkerClass, T> processors =
+          processorFactory.makeProcessors(
+              kernel.getStageDefinition(),
+              kernel.getWorkOrder().getWorkerNumber(),
+              kernel.getWorkOrder().getInputs(),
+              inputSliceReader,
+              (I) kernel.getWorkOrder().getExtraInfo(),
+              workOutputChannelFactory,
+              frameContext,
+              parallelism,
+              counterTracker,
+              e -> warningPublisher.publishException(kernel.getStageDefinition().getStageNumber(), e)
+          );
+
+      final Sequence<WorkerClass> processorSequence = processors.processors();
+
+      final int maxOutstandingProcessors;
+
+      if (processors.getOutputChannels().getAllChannels().isEmpty()) {
+        // No output channels: run up to "parallelism" processors at once.
+        maxOutstandingProcessors = Math.max(1, parallelism);
+      } else {
+        // If there are output channels, that acts as a ceiling on the number of processors that can run at once.
+        maxOutstandingProcessors =
+            Math.max(1, Math.min(parallelism, processors.getOutputChannels().getAllChannels().size()));
+      }
+
+      final ListenableFuture<R> workResultFuture = exec.runAllFully(
+          processorSequence,
+          processorFactory.newAccumulatedResult(),
+          processorFactory::accumulateResult,
+          maxOutstandingProcessors,
+          processorBouncer,
+          cancellationId
+      );
+
+      workResultAndOutputChannels = new ResultAndChannels<>(workResultFuture, processors.getOutputChannels());
+    }
+
+    private void makeAndRunShuffleProcessors()
+    {
+      if (shuffleOutputChannelsFuture != null) {
+        throw new ISE("shuffleOutputChannelsFuture already set");
+      }
+
+      final ShuffleSpec shuffleSpec = kernel.getWorkOrder().getStageDefinition().getShuffleSpec();
+
+      final ShufflePipelineBuilder shufflePipeline = new ShufflePipelineBuilder(
+          kernel,
+          counterTracker,
+          exec,
+          cancellationId,
+          frameContext
+      );
+
+      shufflePipeline.initialize(workResultAndOutputChannels);
+
+      switch (shuffleSpec.kind()) {
+        case MIX:
+          shufflePipeline.mix(shuffleOutputChannelFactory);
+          break;
+
+        case HASH:
+          shufflePipeline.hashPartition(shuffleOutputChannelFactory);
+          break;
+
+        case HASH_LOCAL_SORT:
+          final OutputChannelFactory hashOutputChannelFactory;
+
+          if (shuffleSpec.partitionCount() == 1) {
+            // Single partition; no need to write temporary files.
+            hashOutputChannelFactory =
+                new BlockingQueueOutputChannelFactory(frameContext.memoryParameters().getStandardFrameSize());
+          } else {
+            // Multi-partition; write temporary files and then sort each one file-by-file.
+            hashOutputChannelFactory =
+                new FileOutputChannelFactory(
+                    context.tempDir(kernel.getStageDefinition().getStageNumber(), "hash-parts"),
+                    frameContext.memoryParameters().getStandardFrameSize(),
+                    null
+                );
+          }
+
+          shufflePipeline.hashPartition(hashOutputChannelFactory);
+          shufflePipeline.localSort(shuffleOutputChannelFactory);
+          break;
+
+        case GLOBAL_SORT:
+          shufflePipeline.gatherResultKeyStatisticsIfNeeded();
+          shufflePipeline.globalSort(shuffleOutputChannelFactory, makeGlobalSortPartitionBoundariesFuture());
+          break;
+
+        default:
+          throw new UOE("Cannot handle shuffle kind [%s]", shuffleSpec.kind());
+      }
+
+      shuffleOutputChannelsFuture = shufflePipeline.build();
+    }
+
+    private ListenableFuture<ClusterByPartitions> makeGlobalSortPartitionBoundariesFuture()
+    {
+      if (kernel.getStageDefinition().mustGatherResultKeyStatistics()) {
+        if (stagePartitionBoundariesFuture != null) {
+          throw new ISE("Cannot call 'makeGlobalSortPartitionBoundariesFuture' twice");
+        }
+
+        return (stagePartitionBoundariesFuture = SettableFuture.create());
+      } else {
+        return Futures.immediateFuture(kernel.getResultPartitionBoundaries());
+      }
+    }
+
+    private void setUpCompletionCallbacks()
+    {
+      final StageDefinition stageDef = kernel.getStageDefinition();
+
+      Futures.addCallback(
+          Futures.allAsList(
+              Arrays.asList(
+                  workResultAndOutputChannels.getResultFuture(),
+                  shuffleOutputChannelsFuture
+              )
+          ),
+          new FutureCallback<List<Object>>()
+          {
+            @Override
+            public void onSuccess(final List<Object> workerResultAndOutputChannelsResolved)
+            {
+              final Object resultObject = workerResultAndOutputChannelsResolved.get(0);
+              final OutputChannels outputChannels = (OutputChannels) workerResultAndOutputChannelsResolved.get(1);
+
+              for (OutputChannel channel : outputChannels.getAllChannels()) {
+                try {
+                  stageOutputs.computeIfAbsent(stageDef.getId(), ignored1 -> new ConcurrentHashMap<>())
+                              .computeIfAbsent(channel.getPartitionNumber(), ignored2 -> channel.getReadableChannel());
+                }
+                catch (Exception e) {
+                  kernelManipulationQueue.add(holder -> {
+                    throw new RE(e, "Worker completion callback error for stage [%s]", stageDef.getId());
+                  });
+
+                  // Don't make the "setResultsComplete" call below.
+                  return;
+                }
+              }
+
+              // Once the outputs channels have been resolved and are ready for reading, write success file, if
+              // using durable storage.
+              writeDurableStorageSuccessFileIfNeeded(stageDef.getStageNumber());
+
+              kernelManipulationQueue.add(holder -> holder.getStageKernelMap()
+                                                          .get(stageDef.getId())
+                                                          .setResultsComplete(resultObject));
+            }
+
+            @Override
+            public void onFailure(final Throwable t)
+            {
+              kernelManipulationQueue.add(
+                  kernelHolder ->
+                      kernelHolder.getStageKernelMap().get(stageDef.getId()).fail(t)
+              );
+            }
+          }
+      );
+    }
+
+    /**
+     * Write {@link DurableStorageUtils#SUCCESS_MARKER_FILENAME} for a particular stage, if durable storage is enabled.
+     */
+    private void writeDurableStorageSuccessFileIfNeeded(final int stageNumber)
+    {
+      if (!durableStageStorageEnabled) {
+        return;
+      }
+
+      DurableStorageOutputChannelFactory durableStorageOutputChannelFactory =
+          DurableStorageOutputChannelFactory.createStandardImplementation(
+              task.getControllerTaskId(),
+              task().getWorkerNumber(),
+              stageNumber,
+              task().getId(),
+              frameContext.memoryParameters().getStandardFrameSize(),
+              MSQTasks.makeStorageConnector(context.injector()),
+              context.tempDir()
+          );
+      try {
+        durableStorageOutputChannelFactory.createSuccessFile(task.getId());
+      }
+      catch (IOException e) {
+        throw new ISE(
+            e,
+            "Unable to create the success file [%s] at the location [%s]",
+            DurableStorageUtils.SUCCESS_MARKER_FILENAME,
+            DurableStorageUtils.getSuccessFilePath(
+                task.getControllerTaskId(),
+                stageNumber,
+                task().getWorkerNumber()
+            )
+        );
+      }
+    }
+  }
+
+  /**
+   * Helper for {@link RunWorkOrder#makeAndRunShuffleProcessors()}. Builds a {@link FrameProcessor} pipeline to
+   * handle the shuffle.
+   */
+  private class ShufflePipelineBuilder
+  {
+    private final WorkerStageKernel kernel;
+    private final CounterTracker counterTracker;
+    private final FrameProcessorExecutor exec;
+    private final String cancellationId;
+    private final FrameContext frameContext;
+
+    // Current state of the pipeline. It's a future to allow pipeline construction to be deferred if necessary.
+    private ListenableFuture<ResultAndChannels<?>> pipelineFuture;
+
+    public ShufflePipelineBuilder(
+        final WorkerStageKernel kernel,
+        final CounterTracker counterTracker,
+        final FrameProcessorExecutor exec,
+        final String cancellationId,
+        final FrameContext frameContext
+    )
+    {
+      this.kernel = kernel;
+      this.counterTracker = counterTracker;
+      this.exec = exec;
+      this.cancellationId = cancellationId;
+      this.frameContext = frameContext;
+    }
+
+    /**
+     * Start the pipeline with the outputs of the main processor.
+     */
+    public void initialize(final ResultAndChannels<?> resultAndChannels)
+    {
+      if (pipelineFuture != null) {
+        throw new ISE("already initialized");
+      }
+
+      pipelineFuture = Futures.immediateFuture(resultAndChannels);
+    }
+
+    /**
+     * Add {@link FrameChannelMixer}, which mixes all current outputs into a single channel from the provided factory.
+     */
+    public void mix(final OutputChannelFactory outputChannelFactory)
+    {
+      // No sorting or statistics gathering, just combining all outputs into one big partition. Use a mixer to get
+      // everything into one file. Note: even if there is only one output channel, we'll run it through the mixer
+      // anyway, to ensure the data gets written to a file. (httpGetChannelData requires files.)
+
+      push(
+          resultAndChannels -> {
+            final OutputChannel outputChannel = outputChannelFactory.openChannel(0);
+
+            final FrameChannelMixer mixer =
+                new FrameChannelMixer(
+                    resultAndChannels.getOutputChannels().getAllReadableChannels(),
+                    outputChannel.getWritableChannel()
+                );
+
+            return new ResultAndChannels<>(
+                exec.runFully(mixer, cancellationId),
+                OutputChannels.wrap(Collections.singletonList(outputChannel.readOnly()))
+            );
+          }
+      );
+    }
+
+    /**
+     * Add {@link KeyStatisticsCollectionProcessor} if {@link StageDefinition#mustGatherResultKeyStatistics()}.
+     */
+    public void gatherResultKeyStatisticsIfNeeded()
+    {
+      push(
+          resultAndChannels -> {
+            final StageDefinition stageDefinition = kernel.getStageDefinition();
+            final OutputChannels channels = resultAndChannels.getOutputChannels();
+
+            if (channels.getAllChannels().isEmpty()) {
+              // No data coming out of this processor. Report empty statistics, if the kernel is expecting statistics.
+              if (stageDefinition.mustGatherResultKeyStatistics()) {
+                kernelManipulationQueue.add(
+                    holder ->
+                        holder.getStageKernelMap().get(stageDefinition.getId())
+                              .setResultKeyStatisticsSnapshot(ClusterByStatisticsSnapshot.empty())
+                );
+              }
+
+              // Generate one empty channel so the SuperSorter has something to do.
+              final BlockingQueueFrameChannel channel = BlockingQueueFrameChannel.minimal();
+              channel.writable().close();
+
+              final OutputChannel outputChannel = OutputChannel.readOnly(
+                  channel.readable(),
+                  FrameWithPartition.NO_PARTITION
+              );
+
+              return new ResultAndChannels<>(
+                  Futures.immediateFuture(null),
+                  OutputChannels.wrap(Collections.singletonList(outputChannel))
+              );
+            } else if (stageDefinition.mustGatherResultKeyStatistics()) {
+              return gatherResultKeyStatistics(channels);
+            } else {
+              return resultAndChannels;
+            }
+          }
+      );
+    }
+
+    /**
+     * Add a {@link SuperSorter} using {@link StageDefinition#getSortKey()} and partition boundaries
+     * from {@code partitionBoundariesFuture}.
+     */
+    public void globalSort(
+        final OutputChannelFactory outputChannelFactory,
+        final ListenableFuture<ClusterByPartitions> partitionBoundariesFuture
+    )
+    {
+      pushAsync(
+          resultAndChannels -> {
+            final StageDefinition stageDefinition = kernel.getStageDefinition();
+
+            final File sorterTmpDir = context.tempDir(stageDefinition.getStageNumber(), "super-sort");
+            FileUtils.mkdirp(sorterTmpDir);
+            if (!sorterTmpDir.isDirectory()) {
+              throw new IOException("Cannot create directory: " + sorterTmpDir);
+            }
+
+            final WorkerMemoryParameters memoryParameters = frameContext.memoryParameters();
+            final SuperSorter sorter = new SuperSorter(
+                resultAndChannels.getOutputChannels().getAllReadableChannels(),
+                stageDefinition.getFrameReader(),
+                stageDefinition.getSortKey(),
+                partitionBoundariesFuture,
+                exec,
+                outputChannelFactory,
+                makeSuperSorterIntermediateOutputChannelFactory(
+                    frameContext,
+                    stageDefinition.getStageNumber(),
+                    sorterTmpDir
+                ),
+                memoryParameters.getSuperSorterMaxActiveProcessors(),
+                memoryParameters.getSuperSorterMaxChannelsPerProcessor(),
+                -1,
+                cancellationId,
+                counterTracker.sortProgress()
+            );
+
+            return FutureUtils.transform(
+                sorter.run(),
+                sortedChannels -> new ResultAndChannels<>(Futures.immediateFuture(null), sortedChannels)
+            );
+          }
+      );
+    }
+
+    /**
+     * Add a {@link FrameChannelHashPartitioner} using {@link StageDefinition#getSortKey()}.
+     */
+    public void hashPartition(final OutputChannelFactory outputChannelFactory)
+    {
+      pushAsync(
+          resultAndChannels -> {
+            final ShuffleSpec shuffleSpec = kernel.getStageDefinition().getShuffleSpec();
+            final int partitions = shuffleSpec.partitionCount();
+
+            final List<OutputChannel> outputChannels = new ArrayList<>();
+
+            for (int i = 0; i < partitions; i++) {
+              outputChannels.add(outputChannelFactory.openChannel(i));
+            }
+
+            final FrameChannelHashPartitioner partitioner = new FrameChannelHashPartitioner(
+                resultAndChannels.getOutputChannels().getAllReadableChannels(),
+                outputChannels.stream().map(OutputChannel::getWritableChannel).collect(Collectors.toList()),
+                kernel.getStageDefinition().getFrameReader(),
+                kernel.getStageDefinition().getClusterBy().getColumns().size(),
+                FrameWriters.makeFrameWriterFactory(
+                    FrameType.ROW_BASED,
+                    new ArenaMemoryAllocatorFactory(frameContext.memoryParameters().getStandardFrameSize()),
+                    kernel.getStageDefinition().getSignature(),
+                    kernel.getStageDefinition().getSortKey()
+                )
+            );
+
+            final ListenableFuture<Long> partitionerFuture = exec.runFully(partitioner, cancellationId);
+
+            final ResultAndChannels<Long> retVal =
+                new ResultAndChannels<>(partitionerFuture, OutputChannels.wrap(outputChannels));
+
+            if (retVal.getOutputChannels().areReadableChannelsReady()) {
+              return Futures.immediateFuture(retVal);
+            } else {
+              return FutureUtils.transform(partitionerFuture, ignored -> retVal);
+            }
+          }
+      );
+    }
+
+    /**
+     * Add a sequence of {@link SuperSorter}, operating on each current output channel in order, one at a time.
+     */
+    public void localSort(final OutputChannelFactory outputChannelFactory)
+    {
+      pushAsync(
+          resultAndChannels -> {
+            final WorkerMemoryParameters memoryParameters = frameContext.memoryParameters();

Review Comment:
   ## Unread local variable
   
   Variable 'WorkerMemoryParameters memoryParameters' is never read.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/4318)



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1075 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Input channels for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputChannels = ImmutableList.of(left.getChannel(), right.getChannel());
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(
+        new Tracker(left, keyColumns.get(LEFT)),
+        new Tracker(right, keyColumns.get(RIGHT))
+    );
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!allTrackersAreAtEnd()
+           && !trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()) {
+      // Algorithm can proceed: not all trackers are at the end of their streams, and no tracker needs more data to
+      // read the current cursor or move it forward.
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+
+      // Two rows match if the keys compare equal _and_ neither key has a null component. (x JOIN y ON x.a = y.a does
+      // not match rows where "x.a" is null.)
+      final boolean match = markCmp == 0 && trackers.get(LEFT).hasCompletelyNonNullMark();
+
+      // If marked keys are equal on both sides ("match"), at least one side must have a complete set of rows
+      // for the marked key.
+      if (match && trackerWithCompleteSetForCurrentKey < 0) {
+        for (int i = 0; i < inputChannels.size(); i++) {
+          final Tracker tracker = trackers.get(i);
+
+          // Fetch up to one frame from each tracker, to check if that tracker has a complete set.
+          // Can't fetch more than one frame, because channels are only guaranteed to have one frame per run.
+          if (tracker.hasCompleteSetForMark() || (pushNextFrame(i) && tracker.hasCompleteSetForMark())) {
+            trackerWithCompleteSetForCurrentKey = i;
+            break;
+          }
+        }
+
+        if (trackerWithCompleteSetForCurrentKey < 0) {
+          // Algorithm cannot proceed; fetch more frames on the next run.
+          return nextAwait();
+        }
+      }
+
+      if (match || (markCmp <= 0 && joinType.isLefty()) || (markCmp >= 0 && joinType.isRighty())) {
+        // Emit row, if there's room in the current frameWriter.
+        joinColumnSelectorFactory.cmp = markCmp;
+        joinColumnSelectorFactory.match = match;
+
+        if (!frameWriter.addSelection()) {
+          if (frameWriter.getNumRows() > 0) {
+            // Out of space in the current frame. Run again without moving cursors.
+            flushCurrentFrame();
+            return ReturnOrAwait.runAgain();
+          } else {
+            throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
+          }
+        }
+      }
+
+      // Advance one or both trackers.
+      if (match) {
+        // Matching keys. First advance the tracker with the complete set.
+        final Tracker tracker = trackers.get(trackerWithCompleteSetForCurrentKey);
+        final Tracker otherTracker = trackers.get(trackerWithCompleteSetForCurrentKey == LEFT ? RIGHT : LEFT);
+
+        tracker.advance();
+        if (!tracker.isCurrentSameKeyAsMark()) {
+          // Reached end of complete set. Advance the other tracker.
+          otherTracker.advance();
+
+          // On next iteration (when we're sure to have data) either rewind the complete-set tracker, or update marks
+          // of both, as appropriate.
+          onNextIteration(() -> {
+            if (otherTracker.isCurrentSameKeyAsMark()) {
+              otherTracker.markCurrent(); // Set mark to enable cleanup of old frames.
+              tracker.rewindToMark();
+            } else {
+              // Reached end of the other side too. Advance marks on both trackers.
+              tracker.markCurrent();
+              otherTracker.markCurrent();
+              trackerWithCompleteSetForCurrentKey = -1;
+            }
+          });
+        }
+      } else {
+        final int trackerToAdvance;
+
+        if (markCmp < 0) {
+          trackerToAdvance = LEFT;
+        } else if (markCmp > 0) {
+          trackerToAdvance = RIGHT;
+        } else {
+          // Key is null on both sides. Note that there is a preference for running through the left side first
+          // on a FULL join. It doesn't really matter which side we run through first, but we do need to be consistent
+          // for the benefit of the logic in "shouldEmitColumnValue".
+          trackerToAdvance = joinType.isLefty() ? LEFT : RIGHT;
+        }
+
+        final Tracker tracker = trackers.get(trackerToAdvance);
+
+        tracker.advance();
+
+        // On next iteration (when we're sure to have data), update mark if the key changed.
+        onNextIteration(() -> {
+          if (!tracker.isCurrentSameKeyAsMark()) {
+            tracker.markCurrent();
+            trackerWithCompleteSetForCurrentKey = -1;
+          }
+        });
+      }
+    }
+
+    if (allTrackersAreAtEnd()) {
+      flushCurrentFrame();
+      return ReturnOrAwait.returnObject(0L);
+    } else {
+      // Keep reading.
+      return nextAwait();
+    }
+  }
+
+  @Override
+  public void cleanup() throws IOException
+  {
+    FrameProcessors.closeAll(inputChannels(), outputChannels(), frameWriter, () -> trackers.forEach(Tracker::clear));
+  }
+
+  /**
+   * Returns a {@link ReturnOrAwait#awaitAll} for the channel numbers that need more data and have not yet hit their
+   * buffered-bytes limit, {@link Limits#MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN}.
+   *
+   * If all channels have hit their limit, throws {@link MSQException} with {@link TooManyRowsWithSameKeyFault}.
+   */
+  private ReturnOrAwait<Long> nextAwait()
+  {
+    final IntSet awaitSet = new IntOpenHashSet();
+    int trackerAtLimit = -1;
+
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.needsMoreData()) {
+        if (tracker.totalBytesBuffered() < Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+          awaitSet.add(i);
+        } else if (trackerAtLimit < 0) {
+          trackerAtLimit = i;
+        }
+      }
+    }
+
+    if (awaitSet.isEmpty() && trackerAtLimit > 0) {
+      // All trackers that need more data are at their max buffered bytes limit. Generate a nice exception.
+      final Tracker tracker = trackers.get(trackerAtLimit);
+      if (tracker.totalBytesBuffered() > Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+        // Generate a nice exception.
+        throw new MSQException(
+            new TooManyRowsWithSameKeyFault(
+                tracker.readMarkKey(),
+                tracker.totalBytesBuffered(),
+                Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN
+            )
+        );
+      }
+    }
+
+    return ReturnOrAwait.awaitAll(awaitSet);
+  }
+
+  /**
+   * Whether all trackers return true from {@link Tracker#isAtEnd()}.
+   */
+  private boolean allTrackersAreAtEnd()
+  {
+    for (Tracker tracker : trackers) {
+      if (!tracker.isAtEnd()) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  /**
+   * Compares the marked rows of the two {@link #trackers}.
+   *
+   * @throws IllegalStateException if either tracker does not have a marked row and is not completely done
+   */
+  private int compareMarks()
+  {
+    final Tracker leftTracker = trackers.get(LEFT);
+    final Tracker rightTracker = trackers.get(RIGHT);
+
+    Preconditions.checkState(leftTracker.hasMark() || leftTracker.isAtEnd(), "left.hasMark || left.isAtEnd");
+    Preconditions.checkState(rightTracker.hasMark() || rightTracker.isAtEnd(), "right.hasMark || right.isAtEnd");
+
+    if (!leftTracker.hasMark()) {
+      return rightTracker.markFrame < 0 ? 0 : 1;
+    } else if (!rightTracker.hasMark()) {
+      return -1;
+    } else {
+      final FrameHolder leftHolder = leftTracker.holders.get(leftTracker.markFrame);
+      final FrameHolder rightHolder = rightTracker.holders.get(rightTracker.markFrame);
+      return leftHolder.comparisonWidget.compare(
+          leftTracker.markRow,
+          rightHolder.comparisonWidget,
+          rightTracker.markRow
+      );
+    }
+  }
+
+  /**
+   * Pushes a frame from the indicated channel into the appropriate tracker. Returns true if a frame was pushed
+   * or if the channel is finished.
+   */
+  private boolean pushNextFrame(final int channelNumber)
+  {
+    final ReadableFrameChannel channel = inputChannels.get(channelNumber);
+    final Tracker tracker = trackers.get(channelNumber);
+
+    if (!channel.isFinished() && !channel.canRead()) {
+      return false;
+    } else if (channel.isFinished()) {
+      tracker.push(null);
+      return true;
+    } else {
+      final Frame frame = channel.read();
+
+      if (frame.numRows() == 0) {
+        // Skip, read next.
+        return false;
+      } else {
+        tracker.push(frame);
+        return true;
+      }
+    }
+  }
+
+  private void onNextIteration(final Runnable runnable)
+  {
+    if (nextIterationRunnable != null) {
+      throw new ISE("postAdvanceRunnable already set");
+    } else {
+      nextIterationRunnable = runnable;
+    }
+  }
+
+  private void startNewFrameIfNeeded()
+  {
+    if (frameWriter == null) {
+      frameWriter = frameWriterFactory.newFrameWriter(joinColumnSelectorFactory);
+    }
+  }
+
+  private void flushCurrentFrame() throws IOException
+  {
+    if (frameWriter != null) {
+      if (frameWriter.getNumRows() > 0) {
+        final Frame frame = Frame.wrap(frameWriter.toByteArray());
+        frameWriter.close();
+        frameWriter = null;
+        outputChannel.write(new FrameWithPartition(frame, FrameWithPartition.NO_PARTITION));
+      }
+    }
+  }
+
+  /**
+   * Tracks the current set of rows that have the same key from a sequence of frames.
+   *
+   * markFrame and markRow are set when we encounter a new key, which enables rewinding and re-reading data with the
+   * same key.
+   */
+  private static class Tracker
+  {
+    /**
+     * Frame holders for the current frame, as well as immediately prior frames that share the same marked key.
+     * Prior frames are cleared on each call to {@link #markCurrent()}.
+     */
+    private final List<FrameHolder> holders = new ArrayList<>();
+    private final ReadableInput input;
+    private final List<KeyColumn> keyColumns;
+
+    // markFrame and markRow are the first frame and row with the current key.
+    private int markFrame = -1;
+    private int markRow = -1;
+
+    // currentFrame is the frame containing the current cursor row.
+    private int currentFrame = -1;
+
+    // done indicates that no more data is available in the channel.
+    private boolean done;
+
+    public Tracker(ReadableInput input, List<KeyColumn> keyColumns)
+    {
+      this.input = input;
+      this.keyColumns = keyColumns;
+    }
+
+    /**
+     * Adds a holder for a frame. If this is the first frame, sets the current cursor position and mark to the first
+     * row of the frame. Otherwise, the cursor position and mark are not changed.
+     *
+     * Pushing a null frame indicates no more frames are coming.
+     *
+     * @param frame frame, or null indicating no more frames are coming
+     */
+    public void push(final Frame frame)
+    {
+      if (frame == null) {
+        done = true;
+        return;
+      }
+
+      if (done) {
+        throw new ISE("Cannot push frames when already done");
+      }
+
+      final boolean atEndOfPushedData = isAtEndOfPushedData();
+      final FrameReader frameReader = input.getChannelFrameReader();
+      final FrameCursor cursor = FrameProcessors.makeCursor(frame, frameReader);
+      final FrameComparisonWidget comparisonWidget =
+          frameReader.makeComparisonWidget(frame, keyColumns);
+
+      final RowSignature.Builder keySignatureBuilder = RowSignature.builder();
+      for (final KeyColumn keyColumn : keyColumns) {
+        keySignatureBuilder.add(
+            keyColumn.columnName(),
+            frameReader.signature().getColumnType(keyColumn.columnName()).orElse(null)
+        );
+      }
+
+      holders.add(
+          new FrameHolder(
+              frame,
+              RowKeyReader.create(keySignatureBuilder.build()),
+              cursor,
+              comparisonWidget
+          )
+      );
+
+      if (atEndOfPushedData) {
+        // Move currentFrame so it points at the next row, which we now have, instead of an "isDone" cursor.
+        currentFrame = currentFrame < 0 ? 0 : currentFrame + 1;
+      }
+
+      if (markFrame < 0) {
+        // Cleared mark means we want the current row to be marked.
+        markFrame = currentFrame;
+        markRow = 0;
+      }
+    }
+
+    /**
+     * Number of bytes currently buffered in {@link #holders}.
+     */
+    public long totalBytesBuffered()
+    {
+      long bytes = 0;
+      for (final FrameHolder holder : holders) {
+        bytes += holder.frame.numBytes();
+      }
+      return bytes;
+    }
+
+    /**
+     * Cursor containing the current row.
+     */
+    @Nullable
+    public FrameCursor currentCursor()
+    {
+      if (currentFrame < 0) {
+        return null;
+      } else {
+        return holders.get(currentFrame).cursor;
+      }
+    }
+
+    /**
+     * Advances the current row (the current row of {@link #currentFrame}). After calling this method,
+     * {@link #isAtEndOfPushedData()} may start returning true.
+     */
+    public void advance()
+    {
+      assert !isAtEndOfPushedData();
+
+      final FrameHolder currentHolder = holders.get(currentFrame);
+
+      currentHolder.cursor.advance();
+
+      if (currentHolder.cursor.isDone() && currentFrame + 1 < holders.size()) {
+        currentFrame++;
+        holders.get(currentFrame).cursor.reset();
+      }
+    }
+
+    /**
+     * Whether this tracker has a marked row.
+     */
+    public boolean hasMark()
+    {
+      return markFrame >= 0;
+    }
+
+    /**
+     * Whether this tracker has a marked row that is completely nonnull.
+     */
+    public boolean hasCompletelyNonNullMark()
+    {
+      return hasMark() && !holders.get(markFrame).comparisonWidget.isPartiallyNullKey(markRow);
+    }
+
+    /**
+     * Reads the current marked key.
+     */
+    @Nullable
+    public List<Object> readMarkKey()
+    {
+      if (!hasMark()) {
+        return null;
+      }
+
+      final FrameHolder markHolder = holders.get(markFrame);
+      final RowKey markKey = markHolder.comparisonWidget.readKey(markRow);
+      return markHolder.keyReader.read(markKey);
+    }
+
+    /**
+     * Rewind to the mark row: the first one with the current key.
+     *
+     * @throws IllegalStateException if there is no marked row
+     */
+    public void rewindToMark()
+    {
+      if (markFrame < 0) {
+        throw new ISE("No mark");
+      }
+
+      currentFrame = markFrame;
+      holders.get(currentFrame).cursor.setCurrentRow(markRow);
+    }
+
+    /**
+     * Set the mark row to the current row. Used when data from the old mark to the current row is no longer needed.
+     */
+    public void markCurrent()
+    {
+      if (isAtEndOfPushedData()) {
+        clear();
+      } else {
+        // Remove unnecessary holders, now that the mark has moved on.
+        while (currentFrame > 0) {
+          if (currentFrame == holders.size() - 1) {
+            final FrameHolder lastHolder = holders.get(currentFrame);
+            holders.clear();
+            holders.add(lastHolder);
+            currentFrame = 0;
+          } else {
+            holders.remove(0);
+            currentFrame--;
+          }
+        }
+
+        markFrame = 0;
+        markRow = holders.get(currentFrame).cursor.getCurrentRow();
+      }
+    }
+
+    /**
+     * Whether the current cursor is past the end of the last frame for which we have data.
+     */
+    public boolean isAtEndOfPushedData()
+    {
+      return currentFrame < 0 || (currentFrame == holders.size() - 1 && holders.get(currentFrame).cursor.isDone());
+    }
+
+    /**
+     * Whether the current cursor is past the end of all data that will ever be pushed.
+     */
+    public boolean isAtEnd()
+    {
+      return done && isAtEndOfPushedData();
+    }
+
+    /**
+     * Whether this tracker needs more data in order to read the current cursor location or move it forward.
+     */
+    public boolean needsMoreData()
+    {
+      return !done && isAtEndOfPushedData();
+    }
+
+    /**
+     * Whether this tracker contains all rows for the marked key.
+     *
+     * @throws IllegalStateException if there is no marked key
+     */
+    public boolean hasCompleteSetForMark()
+    {
+      if (markFrame < 0) {
+        throw new ISE("No mark");
+      }
+
+      if (done) {
+        return true;
+      }
+
+      final FrameHolder lastHolder = holders.get(holders.size() - 1);
+      return !isSameKeyAsMark(lastHolder, lastHolder.frame.numRows() - 1);
+    }
+
+    /**
+     * Whether the current position (the current row of the {@link #currentFrame}) compares equally to the mark row.
+     * If {@link #isAtEnd()}, returns true iff there is no mark row.
+     */
+    public boolean isCurrentSameKeyAsMark()
+    {
+      if (isAtEnd()) {
+        return markFrame < 0;
+      } else {
+        assert !isAtEndOfPushedData();
+        final FrameHolder headHolder = holders.get(currentFrame);
+        return isSameKeyAsMark(headHolder, headHolder.cursor.getCurrentRow());
+      }
+    }
+
+    /**
+     * Clears the current mark and all buffered frames. Does not change {@link #done}.
+     */
+    public void clear()
+    {
+      holders.clear();
+      markFrame = -1;
+      markRow = -1;
+      currentFrame = -1;
+    }
+
+    /**
+     * Whether the provided frame and row compares equally to the mark row. The provided row must be at, or after,
+     * the mark row.
+     */
+    private boolean isSameKeyAsMark(final FrameHolder holder, final int row)
+    {
+      if (markFrame < 0) {
+        throw new ISE("No marked frame");
+      }
+      if (row < 0 || row >= holder.frame.numRows()) {
+        throw new ISE("Row [%d] out of bounds", row);
+      }
+
+      final FrameHolder markHolder = holders.get(markFrame);
+      final int cmp = markHolder.comparisonWidget.compare(markRow, holder.comparisonWidget, row);
+
+      if (cmp > 0) {
+        // The provided row is at, or after, the marked row.
+        // Therefore, cmp > 0 may indicate that input was provided out of order.
+        throw new ISE("Row compares higher than mark; out-of-order input?");
+      }
+
+      return cmp == 0;
+    }
+  }
+
+  /**
+   * Selector for joined rows. This is used as an input to {@link #frameWriter}.
+   */
+  private class JoinColumnSelectorFactory implements ColumnSelectorFactory
+  {
+    /**
+     * Current key comparison between left- and right-hand side.
+     */
+    private int cmp;
+
+    /**
+     * Whether there is a match between the left- and right-hand side. Not equivalent to {@code cmp == 0} in
+     * the case where the key on both sides is null.
+     */
+    private boolean match;
+
+    @Override
+    public DimensionSelector makeDimensionSelector(DimensionSpec dimensionSpec)
+    {
+      if (dimensionSpec.getExtractionFn() != null || dimensionSpec.mustDecorate()) {

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [DimensionSpec.getExtractionFn](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/4320)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1127471929


##########
processing/src/main/java/org/apache/druid/frame/key/ClusterBy.java:
##########
@@ -58,6 +59,21 @@ public ClusterBy(
     if (bucketByCount < 0 || bucketByCount > columns.size()) {
       throw new IAE("Invalid bucketByCount [%d]", bucketByCount);
     }
+
+    // Key must be 100% sortable or 100% nonsortable. If empty, call it sortable.
+    boolean sortable = true;
+
+    for (int i = 0; i < columns.size(); i++) {
+      final KeyColumn column = columns.get(i);
+
+      if (i == 0) {
+        sortable = column.order().sortable();
+      } else if (sortable != column.order().sortable()) {
+        throw new IAE("Cannot mix sortable and unsortable key columns");
+      }
+    }
+
+    this.sortable = sortable;

Review Comment:
   I kept it as-is since they both seem equally good to me and I'm lazy 🙂



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "imply-cheddar (via GitHub)" <gi...@apache.org>.
imply-cheddar commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1125836072


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java:
##########
@@ -212,24 +219,51 @@ public boolean doesShuffle()
 
   public boolean doesSortDuringShuffle()
   {
-    if (shuffleSpec == null) {
+    if (shuffleSpec == null || shuffleSpec.clusterBy().isEmpty()) {
       return false;
     } else {
-      return !shuffleSpec.getClusterBy().getColumns().isEmpty() || shuffleSpec.needsStatistics();
+      return shuffleSpec.clusterBy().sortable();
     }
   }
 
-  public Optional<ShuffleSpec> getShuffleSpec()
+  /**
+   * Returns the {@link ShuffleSpec} for this stage, if {@link #doesShuffle()}.
+   *
+   * @throws IllegalStateException if this stage does not shuffle
+   */
+  public ShuffleSpec getShuffleSpec()
   {
-    return Optional.ofNullable(shuffleSpec);
+    if (shuffleSpec == null) {
+      throw new IllegalStateException("Stage does not shuffle");
+    }
+
+    return shuffleSpec;
   }

Review Comment:
   Is there a way to know that the stage shuffles or not before calling this method (to avoid the ISE)?
   
   I see changes to `doesSortDuringShuffle`, but that seems to indicate more than just "does the current stage "shuffle".



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/GlobalSortMaxCountShuffleSpec.java:
##########
@@ -93,15 +110,26 @@ public Either<Long, ClusterByPartitions> generatePartitions(
 
   @Override
   @JsonProperty
-  public ClusterBy getClusterBy()
+  public ClusterBy clusterBy()
   {
     return clusterBy;
   }
 
-  @JsonProperty
-  int getPartitions()
+  @Override
+  public int partitionCount()
+  {
+    if (maxPartitions == 1) {
+      return maxPartitions;
+    } else {
+      // Number of actual partitions may be less than maxPartitions.
+      throw new IllegalStateException("Number of partitions not known.");
+    }
+  }

Review Comment:
   This logic makes no sense to me?  The partitionCount equals the maxPartitions if maxPartitions == 1?  Doesn't that mean that partitionCount is either 1 or an exception?



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1075 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Input channels for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputChannels = ImmutableList.of(left.getChannel(), right.getChannel());
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(
+        new Tracker(left, keyColumns.get(LEFT)),
+        new Tracker(right, keyColumns.get(RIGHT))
+    );
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!allTrackersAreAtEnd()
+           && !trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()) {
+      // Algorithm can proceed: not all trackers are at the end of their streams, and no tracker needs more data to
+      // read the current cursor or move it forward.
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+
+      // Two rows match if the keys compare equal _and_ neither key has a null component. (x JOIN y ON x.a = y.a does
+      // not match rows where "x.a" is null.)
+      final boolean match = markCmp == 0 && trackers.get(LEFT).hasCompletelyNonNullMark();
+
+      // If marked keys are equal on both sides ("match"), at least one side must have a complete set of rows
+      // for the marked key.
+      if (match && trackerWithCompleteSetForCurrentKey < 0) {
+        for (int i = 0; i < inputChannels.size(); i++) {
+          final Tracker tracker = trackers.get(i);
+
+          // Fetch up to one frame from each tracker, to check if that tracker has a complete set.
+          // Can't fetch more than one frame, because channels are only guaranteed to have one frame per run.
+          if (tracker.hasCompleteSetForMark() || (pushNextFrame(i) && tracker.hasCompleteSetForMark())) {
+            trackerWithCompleteSetForCurrentKey = i;
+            break;
+          }
+        }
+
+        if (trackerWithCompleteSetForCurrentKey < 0) {
+          // Algorithm cannot proceed; fetch more frames on the next run.
+          return nextAwait();
+        }
+      }
+
+      if (match || (markCmp <= 0 && joinType.isLefty()) || (markCmp >= 0 && joinType.isRighty())) {
+        // Emit row, if there's room in the current frameWriter.
+        joinColumnSelectorFactory.cmp = markCmp;
+        joinColumnSelectorFactory.match = match;
+
+        if (!frameWriter.addSelection()) {
+          if (frameWriter.getNumRows() > 0) {
+            // Out of space in the current frame. Run again without moving cursors.
+            flushCurrentFrame();
+            return ReturnOrAwait.runAgain();
+          } else {
+            throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
+          }
+        }
+      }
+
+      // Advance one or both trackers.
+      if (match) {
+        // Matching keys. First advance the tracker with the complete set.
+        final Tracker tracker = trackers.get(trackerWithCompleteSetForCurrentKey);
+        final Tracker otherTracker = trackers.get(trackerWithCompleteSetForCurrentKey == LEFT ? RIGHT : LEFT);
+
+        tracker.advance();
+        if (!tracker.isCurrentSameKeyAsMark()) {
+          // Reached end of complete set. Advance the other tracker.
+          otherTracker.advance();
+
+          // On next iteration (when we're sure to have data) either rewind the complete-set tracker, or update marks
+          // of both, as appropriate.
+          onNextIteration(() -> {
+            if (otherTracker.isCurrentSameKeyAsMark()) {
+              otherTracker.markCurrent(); // Set mark to enable cleanup of old frames.
+              tracker.rewindToMark();
+            } else {
+              // Reached end of the other side too. Advance marks on both trackers.
+              tracker.markCurrent();
+              otherTracker.markCurrent();
+              trackerWithCompleteSetForCurrentKey = -1;
+            }
+          });
+        }
+      } else {
+        final int trackerToAdvance;
+
+        if (markCmp < 0) {
+          trackerToAdvance = LEFT;
+        } else if (markCmp > 0) {
+          trackerToAdvance = RIGHT;
+        } else {
+          // Key is null on both sides. Note that there is a preference for running through the left side first
+          // on a FULL join. It doesn't really matter which side we run through first, but we do need to be consistent
+          // for the benefit of the logic in "shouldEmitColumnValue".
+          trackerToAdvance = joinType.isLefty() ? LEFT : RIGHT;
+        }
+
+        final Tracker tracker = trackers.get(trackerToAdvance);
+
+        tracker.advance();
+
+        // On next iteration (when we're sure to have data), update mark if the key changed.
+        onNextIteration(() -> {
+          if (!tracker.isCurrentSameKeyAsMark()) {
+            tracker.markCurrent();
+            trackerWithCompleteSetForCurrentKey = -1;
+          }
+        });
+      }
+    }
+
+    if (allTrackersAreAtEnd()) {
+      flushCurrentFrame();
+      return ReturnOrAwait.returnObject(0L);
+    } else {
+      // Keep reading.
+      return nextAwait();
+    }
+  }
+
+  @Override
+  public void cleanup() throws IOException
+  {
+    FrameProcessors.closeAll(inputChannels(), outputChannels(), frameWriter, () -> trackers.forEach(Tracker::clear));
+  }
+
+  /**
+   * Returns a {@link ReturnOrAwait#awaitAll} for the channel numbers that need more data and have not yet hit their
+   * buffered-bytes limit, {@link Limits#MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN}.
+   *
+   * If all channels have hit their limit, throws {@link MSQException} with {@link TooManyRowsWithSameKeyFault}.
+   */
+  private ReturnOrAwait<Long> nextAwait()
+  {
+    final IntSet awaitSet = new IntOpenHashSet();
+    int trackerAtLimit = -1;
+
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.needsMoreData()) {
+        if (tracker.totalBytesBuffered() < Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+          awaitSet.add(i);
+        } else if (trackerAtLimit < 0) {
+          trackerAtLimit = i;
+        }
+      }
+    }
+
+    if (awaitSet.isEmpty() && trackerAtLimit > 0) {
+      // All trackers that need more data are at their max buffered bytes limit. Generate a nice exception.
+      final Tracker tracker = trackers.get(trackerAtLimit);
+      if (tracker.totalBytesBuffered() > Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+        // Generate a nice exception.
+        throw new MSQException(
+            new TooManyRowsWithSameKeyFault(
+                tracker.readMarkKey(),
+                tracker.totalBytesBuffered(),
+                Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN
+            )
+        );
+      }
+    }
+
+    return ReturnOrAwait.awaitAll(awaitSet);
+  }
+
+  /**
+   * Whether all trackers return true from {@link Tracker#isAtEnd()}.
+   */
+  private boolean allTrackersAreAtEnd()
+  {
+    for (Tracker tracker : trackers) {
+      if (!tracker.isAtEnd()) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  /**
+   * Compares the marked rows of the two {@link #trackers}.
+   *
+   * @throws IllegalStateException if either tracker does not have a marked row and is not completely done
+   */
+  private int compareMarks()
+  {
+    final Tracker leftTracker = trackers.get(LEFT);
+    final Tracker rightTracker = trackers.get(RIGHT);
+
+    Preconditions.checkState(leftTracker.hasMark() || leftTracker.isAtEnd(), "left.hasMark || left.isAtEnd");
+    Preconditions.checkState(rightTracker.hasMark() || rightTracker.isAtEnd(), "right.hasMark || right.isAtEnd");
+
+    if (!leftTracker.hasMark()) {
+      return rightTracker.markFrame < 0 ? 0 : 1;
+    } else if (!rightTracker.hasMark()) {
+      return -1;
+    } else {
+      final FrameHolder leftHolder = leftTracker.holders.get(leftTracker.markFrame);
+      final FrameHolder rightHolder = rightTracker.holders.get(rightTracker.markFrame);
+      return leftHolder.comparisonWidget.compare(
+          leftTracker.markRow,
+          rightHolder.comparisonWidget,
+          rightTracker.markRow
+      );
+    }
+  }
+
+  /**
+   * Pushes a frame from the indicated channel into the appropriate tracker. Returns true if a frame was pushed
+   * or if the channel is finished.
+   */
+  private boolean pushNextFrame(final int channelNumber)
+  {
+    final ReadableFrameChannel channel = inputChannels.get(channelNumber);
+    final Tracker tracker = trackers.get(channelNumber);
+
+    if (!channel.isFinished() && !channel.canRead()) {
+      return false;
+    } else if (channel.isFinished()) {
+      tracker.push(null);
+      return true;
+    } else {
+      final Frame frame = channel.read();
+
+      if (frame.numRows() == 0) {
+        // Skip, read next.
+        return false;
+      } else {
+        tracker.push(frame);
+        return true;
+      }
+    }
+  }
+
+  private void onNextIteration(final Runnable runnable)
+  {
+    if (nextIterationRunnable != null) {
+      throw new ISE("postAdvanceRunnable already set");

Review Comment:
   Is there any extra information that can be interpolated or anything in here to try to help the person who runs into this exception figure out what is going on?



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/HashShuffleSpec.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.kernel;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.frame.key.ClusterBy;
+import org.apache.druid.frame.key.ClusterByPartitions;
+import org.apache.druid.java.util.common.Either;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.msq.statistics.ClusterByStatisticsCollector;
+
+import javax.annotation.Nullable;
+
+public class HashShuffleSpec implements ShuffleSpec
+{
+  public static final String TYPE = "hash";
+
+  private final ClusterBy clusterBy;
+  private final int numPartitions;
+
+  @JsonCreator
+  public HashShuffleSpec(
+      @JsonProperty("clusterBy") final ClusterBy clusterBy,
+      @JsonProperty("partitions") final int numPartitions
+  )
+  {
+    this.clusterBy = clusterBy;
+    this.numPartitions = numPartitions;
+
+    if (clusterBy.getBucketByCount() > 0) {
+      // Only GlobalSortTargetSizeShuffleSpec supports bucket-by.
+      throw new IAE("Cannot bucket with %s partitioning", TYPE);
+    }

Review Comment:
   Why does this spec have to validate for something that is only supported by something else?  Is there no way to make it so that the only thing that supports it is the only thing that takes it?
   
   Additionally, if you were to interpolate the `clusterBy` into the error message, would it generate something that tells the user which part of their query is doing weird things?  (I'm imagining a complex query with lots of WITH and sub-queries, it can be hard to figure out which one is acting weird sometimes)



##########
processing/src/main/java/org/apache/druid/frame/key/ClusterBy.java:
##########
@@ -58,6 +59,21 @@ public ClusterBy(
     if (bucketByCount < 0 || bucketByCount > columns.size()) {
       throw new IAE("Invalid bucketByCount [%d]", bucketByCount);
     }
+
+    // Key must be 100% sortable or 100% nonsortable. If empty, call it sortable.
+    boolean sortable = true;
+
+    for (int i = 0; i < columns.size(); i++) {
+      final KeyColumn column = columns.get(i);
+
+      if (i == 0) {
+        sortable = column.order().sortable();
+      } else if (sortable != column.order().sortable()) {
+        throw new IAE("Cannot mix sortable and unsortable key columns");
+      }
+    }
+
+    this.sortable = sortable;

Review Comment:
   Code nit, could also do:
   
   ```
   if (columns.size == 0) {
     // no columns means that it's sortable/sorted as they are effectively all sorted by the same non-existant, constant value.
     this.sortable = true;
   } else {
     boolean sortable = columns.get(0).order().sortable();
     for (int i = 1; i < columns.size; ++i)  {
       if (sortable != column.order().sortable()) {
         throw new IAE("Cannot mix sortable and unsortable key columns");
       }
     }
   }
   ```



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java:
##########
@@ -288,37 +322,72 @@ public boolean mustGatherResultKeyStatistics()
     return shuffleSpec != null && shuffleSpec.needsStatistics();
   }
 
-  public Either<Long, ClusterByPartitions> generatePartitionsForShuffle(
+  public Either<Long, ClusterByPartitions> generatePartitionBoundariesForShuffle(
       @Nullable ClusterByStatisticsCollector collector
   )
   {
     if (shuffleSpec == null) {
       throw new ISE("No shuffle for stage[%d]", getStageNumber());
+    } else if (shuffleSpec.kind() != ShuffleKind.GLOBAL_SORT) {
+      throw new ISE(
+          "Shuffle of kind [%s] cannot generate partition boundaries for stage[%d]",
+          shuffleSpec.kind(),
+          getStageNumber()
+      );
     } else if (mustGatherResultKeyStatistics() && collector == null) {
       throw new ISE("Statistics required, but not gathered for stage[%d]", getStageNumber());
     } else if (!mustGatherResultKeyStatistics() && collector != null) {
       throw new ISE("Statistics gathered, but not required for stage[%d]", getStageNumber());
     } else {
-      return shuffleSpec.generatePartitions(collector, MAX_PARTITIONS);
+      return shuffleSpec.generatePartitionsForGlobalSort(collector, MAX_PARTITIONS);
     }
   }
 
   public ClusterByStatisticsCollector createResultKeyStatisticsCollector(final int maxRetainedBytes)
   {
     if (!mustGatherResultKeyStatistics()) {
-      throw new ISE("No statistics needed");
+      throw new ISE("No statistics needed for stage[%d]", getStageNumber());
     }
 
     return ClusterByStatisticsCollectorImpl.create(
-        shuffleSpec.getClusterBy(),
+        shuffleSpec.clusterBy(),
         signature,
         maxRetainedBytes,
         PARTITION_STATS_MAX_BUCKETS,
-        shuffleSpec.doesAggregateByClusterKey(),
+        shuffleSpec.doesAggregate(),
         shuffleCheckHasMultipleValues
     );

Review Comment:
   I'm not sure the semantics of "mustGatherStatistics", but "must gather" and "might choose to gather if the user asked nicely" could be seen as different things.  If a user asked for full statistics on over their run, would the "mustGather" method end up returning true?



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/HashShuffleSpec.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.kernel;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.frame.key.ClusterBy;
+import org.apache.druid.frame.key.ClusterByPartitions;
+import org.apache.druid.java.util.common.Either;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.msq.statistics.ClusterByStatisticsCollector;
+
+import javax.annotation.Nullable;
+
+public class HashShuffleSpec implements ShuffleSpec
+{
+  public static final String TYPE = "hash";
+
+  private final ClusterBy clusterBy;
+  private final int numPartitions;
+
+  @JsonCreator
+  public HashShuffleSpec(
+      @JsonProperty("clusterBy") final ClusterBy clusterBy,
+      @JsonProperty("partitions") final int numPartitions
+  )
+  {
+    this.clusterBy = clusterBy;
+    this.numPartitions = numPartitions;
+
+    if (clusterBy.getBucketByCount() > 0) {
+      // Only GlobalSortTargetSizeShuffleSpec supports bucket-by.
+      throw new IAE("Cannot bucket with %s partitioning", TYPE);
+    }
+  }
+
+  @Override
+  public ShuffleKind kind()
+  {
+    return clusterBy.sortable() && !clusterBy.isEmpty() ? ShuffleKind.HASH_LOCAL_SORT : ShuffleKind.HASH;
+  }
+
+  @Override
+  @JsonProperty
+  public ClusterBy clusterBy()
+  {
+    return clusterBy;
+  }
+
+  @Override
+  public boolean doesAggregate()
+  {
+    return false;
+  }
+
+  @Override
+  public boolean needsStatistics()
+  {
+    return false;
+  }
+
+  @Override
+  public Either<Long, ClusterByPartitions> generatePartitionsForGlobalSort(
+      @Nullable ClusterByStatisticsCollector collector,
+      int maxNumPartitions
+  )
+  {
+    throw new IllegalStateException("Not a global sort");
+  }

Review Comment:
   Can the global sort methods exist only on the things that do global sort?  One of the things that sucks the most about the AggregatorFactory and other interfaces is the number of methods that can be ignored and/or are only special purpose, would be nice to keep special things relegated to their own special location if at all possible.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1111165610


##########
docs/multi-stage-query/reference.md:
##########
@@ -198,13 +198,99 @@ The following table lists the context parameters for the MSQ task engine:
 | `maxNumTasks` | SELECT, INSERT, REPLACE<br /><br />The maximum total number of tasks to launch, including the controller task. The lowest possible value for this setting is 2: one controller and one worker. All tasks must be able to launch simultaneously. If they cannot, the query returns a `TaskStartTimeout` error code after approximately 10 minutes.<br /><br />May also be provided as `numTasks`. If both are present, `maxNumTasks` takes priority.| 2 |
 | `taskAssignment` | SELECT, INSERT, REPLACE<br /><br />Determines how many tasks to use. Possible values include: <ul><li>`max`: Uses as many tasks as possible, up to `maxNumTasks`.</li><li>`auto`: When file sizes can be determined through directory listing (for example: local files, S3, GCS, HDFS) uses as few tasks as possible without exceeding 10 GiB or 10,000 files per task, unless exceeding these limits is necessary to stay within `maxNumTasks`. When file sizes cannot be determined through directory listing (for example: http), behaves the same as `max`.</li></ul> | `max` |
 | `finalizeAggregations` | SELECT, INSERT, REPLACE<br /><br />Determines the type of aggregation to return. If true, Druid finalizes the results of complex aggregations that directly appear in query results. If false, Druid returns the aggregation's intermediate type rather than finalized type. This parameter is useful during ingestion, where it enables storing sketches directly in Druid tables. For more information about aggregations, see [SQL aggregation functions](../querying/sql-aggregations.md). | true |
+| `sqlJoinAlgorithm` | SELECT, INSERT, REPLACE<br /><br />Algorithm to use for JOIN. Use `broadcast` (the default) for broadcast hash join or `sortMerge` for sort-merge join. Affects all JOIN operations in the query. See [Joins](#joins) for more details. | `broadcast` |

Review Comment:
   Making some progress here, described on #13532. It will require another Calcite release, so I think it's best if we merge this sort-merge join PR first, then do the Calcite upgrade and add hints.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1114183648


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Names of the key columns on each side of the join, unprefixed. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<List<KeyColumn>> keyColumns;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputs = ImmutableList.of(left, right);
+    this.inputChannels = inputs.stream().map(ReadableInput::getChannel).collect(Collectors.toList());
+    this.keyColumns = keyColumns;
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(new Tracker(LEFT), new Tracker(RIGHT));
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()
+           && !(trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd())) {
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+      final boolean match = markCmp == 0 && !trackers.get(LEFT).hasPartiallyNullMark();
+
+      // Fetch new frames until the algorithm can proceed:
+      //   1) If marked keys are equal on both sides, at least one side has a complete set of rows for the marked key.
+      //   2) Otherwise, each side has at least one readable row, or is completely done. (Checked by "while" condition.)
+      if (match && trackerWithCompleteSetForCurrentKey < 0) {
+        for (int i = 0; i < inputChannels.size(); i++) {
+          final Tracker tracker = trackers.get(i);
+          if (tracker.hasCompleteSetForMark() || (pushNextFrame(i) && tracker.hasCompleteSetForMark())) {

Review Comment:
   I adjusted some comments to hopefully clarify this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm merged pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm merged PR #13506:
URL: https://github.com/apache/druid/pull/13506


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1042611343


##########
docs/multi-stage-query/reference.md:
##########
@@ -198,13 +198,99 @@ The following table lists the context parameters for the MSQ task engine:
 | `maxNumTasks` | SELECT, INSERT, REPLACE<br /><br />The maximum total number of tasks to launch, including the controller task. The lowest possible value for this setting is 2: one controller and one worker. All tasks must be able to launch simultaneously. If they cannot, the query returns a `TaskStartTimeout` error code after approximately 10 minutes.<br /><br />May also be provided as `numTasks`. If both are present, `maxNumTasks` takes priority.| 2 |
 | `taskAssignment` | SELECT, INSERT, REPLACE<br /><br />Determines how many tasks to use. Possible values include: <ul><li>`max`: Uses as many tasks as possible, up to `maxNumTasks`.</li><li>`auto`: When file sizes can be determined through directory listing (for example: local files, S3, GCS, HDFS) uses as few tasks as possible without exceeding 10 GiB or 10,000 files per task, unless exceeding these limits is necessary to stay within `maxNumTasks`. When file sizes cannot be determined through directory listing (for example: http), behaves the same as `max`.</li></ul> | `max` |
 | `finalizeAggregations` | SELECT, INSERT, REPLACE<br /><br />Determines the type of aggregation to return. If true, Druid finalizes the results of complex aggregations that directly appear in query results. If false, Druid returns the aggregation's intermediate type rather than finalized type. This parameter is useful during ingestion, where it enables storing sketches directly in Druid tables. For more information about aggregations, see [SQL aggregation functions](../querying/sql-aggregations.md). | true |
+| `sqlJoinAlgorithm` | SELECT, INSERT, REPLACE<br /><br />Algorithm to use for JOIN. Use `broadcast` (the default) for broadcast hash join or `sortMerge` for sort-merge join. Affects all JOIN operations in the query. See [Joins](#joins) for more details. | `broadcast` |

Review Comment:
   Unfortunately, hints are not available in the version of Calcite version we use. Newer versions have this: https://calcite.apache.org/docs/reference.html#sql-hints
   
   @abhishekagarwal87 found some reasons we couldn't do it right now, as referenced by this comment: https://github.com/apache/druid/pull/13153#issuecomment-1261204764. @abhishekagarwal87 would you mind writing up your notes as to what blocks an upgrade, and making an issue about that, titled something like `Upgrade Calcite past <whatever version introduced the blocking problem>`? That way, we have an issue we can refer to and use to discuss possible ways to fix the blockers.
   
   Once we sort that out, I'd like to deprecate the context parameter and move things to use hints instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "imply-cheddar (via GitHub)" <gi...@apache.org>.
imply-cheddar commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1121184142


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerContext.java:
##########
@@ -73,4 +74,9 @@
   DruidNode selfNode();
 
   Bouncer processorBouncer();
+
+  default File tempDir(int stageNumber, String id)

Review Comment:
   I like consistency.  If it's always `tmp` then it's much easier to say that next time it should be `tmp` as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1127455895


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/GlobalSortMaxCountShuffleSpec.java:
##########
@@ -93,15 +110,26 @@ public Either<Long, ClusterByPartitions> generatePartitions(
 
   @Override
   @JsonProperty
-  public ClusterBy getClusterBy()
+  public ClusterBy clusterBy()
   {
     return clusterBy;
   }
 
-  @JsonProperty
-  int getPartitions()
+  @Override
+  public int partitionCount()
+  {
+    if (maxPartitions == 1) {
+      return maxPartitions;
+    } else {
+      // Number of actual partitions may be less than maxPartitions.
+      throw new IllegalStateException("Number of partitions not known.");
+    }
+  }

Review Comment:
   Ah, this method is never actually called, because we never need to know the partition count ahead of time for global-sort shuffle specs. This is a relic of a past version of the patch where global-sort with maxPartitions = 1 was a way of encoding "mix everything into a single partition". Now, there's an explicit "mix" spec, so this isn't needed any longer.
   
   I replaced the entire thing with a `throw`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1114232945


##########
processing/src/main/java/org/apache/druid/frame/field/StringFieldReader.java:
##########
@@ -87,6 +87,14 @@ public DimensionSelector makeDimensionSelector(
     return new Selector(memory, fieldPointer, extractionFn, false);
   }
 
+  @Override
+  public boolean isNull(Memory memory, long position)
+  {
+    return memory.getByte(position) == StringFieldWriter.NULL_BYTE
+           && memory.getByte(position + 1) == StringFieldWriter.VALUE_TERMINATOR
+           && memory.getByte(position + 2) == StringFieldWriter.ROW_TERMINATOR;

Review Comment:
   There isn't necessarily going to be an extra byte of padding, and I'm not sure that reading a 3-byte array is going to be better. I'd prefer to leave this as-is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1114116023


##########
processing/src/main/java/org/apache/druid/frame/key/KeyOrder.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.frame.key;
+
+public enum KeyOrder
+{
+  NONE(false),

Review Comment:
   OK, I thought about it more, so much more that it took a few months 😄
   
   NONE actually does make sense rather than HASHED, since we can do hash partitioning along with sorting. (Meaning that ASCENDING or DESCENDING key order may still be involved in hash partitioning.)
   
   I'll add comments discussing this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1114229933


##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessorTest.java:
##########
@@ -0,0 +1,987 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.common.guava.FutureUtils;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.FrameType;
+import org.apache.druid.frame.allocation.ArenaMemoryAllocator;
+import org.apache.druid.frame.allocation.SingleMemoryAllocatorFactory;
+import org.apache.druid.frame.channel.BlockingQueueFrameChannel;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.ReadableNilFrameChannel;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.KeyOrder;
+import org.apache.druid.frame.processor.FrameProcessorExecutor;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameStorageAdapter;
+import org.apache.druid.frame.testutil.FrameSequenceBuilder;
+import org.apache.druid.frame.testutil.FrameTestUtil;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.frame.write.FrameWriters;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.guava.Sequence;
+import org.apache.druid.java.util.common.guava.Sequences;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.msq.kernel.StageId;
+import org.apache.druid.msq.kernel.StagePartition;
+import org.apache.druid.msq.test.LimitedFrameWriterFactory;
+import org.apache.druid.segment.RowBasedSegment;
+import org.apache.druid.segment.StorageAdapter;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.join.JoinTestHelper;
+import org.apache.druid.segment.join.JoinType;
+import org.apache.druid.testing.InitializedNullHandlingTest;
+import org.apache.druid.timeline.SegmentId;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+@RunWith(Parameterized.class)
+public class SortMergeJoinFrameProcessorTest extends InitializedNullHandlingTest
+{
+  private static final StagePartition STAGE_PARTITION = new StagePartition(new StageId("q", 0), 0);
+
+  private final int rowsPerInputFrame;
+  private final int rowsPerOutputFrame;
+
+  private FrameProcessorExecutor exec;
+
+  @Rule
+  public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+  public SortMergeJoinFrameProcessorTest(int rowsPerInputFrame, int rowsPerOutputFrame)
+  {
+    this.rowsPerInputFrame = rowsPerInputFrame;
+    this.rowsPerOutputFrame = rowsPerOutputFrame;
+  }
+
+  @Parameterized.Parameters(name = "rowsPerInputFrame = {0}, rowsPerOutputFrame = {1}")
+  public static Iterable<Object[]> constructorFeeder()
+  {
+    final List<Object[]> constructors = new ArrayList<>();
+
+    for (final int rowsPerInputFrame : new int[]{1, 2, 7, Integer.MAX_VALUE}) {
+      for (final int rowsPerOutputFrame : new int[]{1, 2, 7, Integer.MAX_VALUE}) {
+        constructors.add(new Object[]{rowsPerInputFrame, rowsPerOutputFrame});
+      }
+    }
+
+    return constructors;
+  }
+
+  @Before
+  public void setUp()
+  {
+    exec = new FrameProcessorExecutor(MoreExecutors.listeningDecorator(Execs.singleThreaded("test-exec")));
+  }
+
+  @After
+  public void tearDown() throws Exception
+  {
+    exec.getExecutorService().shutdownNow();
+    exec.getExecutorService().awaitTermination(10, TimeUnit.MINUTES);
+  }
+
+  @Test
+  public void testLeftJoinEmptyLeftSide() throws Exception
+  {
+    final ReadableInput factChannel = ReadableInput.channel(
+        ReadableNilFrameChannel.INSTANCE,
+        FrameReader.create(JoinTestHelper.FACT_SIGNATURE),
+        STAGE_PARTITION
+    );
+
+    final ReadableInput countriesChannel =
+        buildCountriesInput(ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)));
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryName", ColumnType.STRING)
+                    .add("j0.countryNumber", ColumnType.LONG)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        countriesChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.LEFT
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, Collections.emptyList());
+  }
+
+  @Test
+  public void testLeftJoinEmptyRightSide() throws Exception
+  {
+    final ReadableInput factChannel = buildFactInput(
+        ImmutableList.of(
+            new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+            new KeyColumn("page", KeyOrder.ASCENDING)
+        )
+    );
+
+
+    final ReadableInput countriesChannel = ReadableInput.channel(
+        ReadableNilFrameChannel.INSTANCE,
+        FrameReader.create(JoinTestHelper.COUNTRIES_SIGNATURE),
+        STAGE_PARTITION
+    );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryName", ColumnType.STRING)
+                    .add("j0.countryNumber", ColumnType.LONG)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        countriesChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.LEFT
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList("Agama mossambica", null, null, null, null),
+        Arrays.asList("Apamea abruzzorum", null, null, null, null),
+        Arrays.asList("Atractus flammigerus", null, null, null, null),
+        Arrays.asList("Rallicula", null, null, null, null),
+        Arrays.asList("Talk:Oswald Tilghman", null, null, null, null),
+        Arrays.asList("Peremptory norm", "AU", null, null, null),
+        Arrays.asList("Didier Leclair", "CA", null, null, null),
+        Arrays.asList("Les Argonautes", "CA", null, null, null),
+        Arrays.asList("Sarah Michelle Gellar", "CA", null, null, null),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "CL", null, null, null),
+        Arrays.asList("Diskussion:Sebastian Schulz", "DE", null, null, null),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "EC", null, null, null),
+        Arrays.asList("Saison 9 de Secret Story", "FR", null, null, null),
+        Arrays.asList("Glasgow", "GB", null, null, null),
+        Arrays.asList("Giusy Ferreri discography", "IT", null, null, null),
+        Arrays.asList("Roma-Bangkok", "IT", null, null, null),
+        Arrays.asList("青野武", "JP", null, null, null),
+        Arrays.asList("유희왕 GX", "KR", null, null, null),
+        Arrays.asList("History of Fourems", "MMMM", null, null, null),
+        Arrays.asList("Mathis Bolly", "MX", null, null, null),
+        Arrays.asList("Orange Soda", "MatchNothing", null, null, null),
+        Arrays.asList("Алиса в Зазеркалье", "NO", null, null, null),
+        Arrays.asList("Cream Soda", "SU", null, null, null),
+        Arrays.asList("Wendigo", "SV", null, null, null),
+        Arrays.asList("Carlo Curti", "US", null, null, null),
+        Arrays.asList("DirecTV", "US", null, null, null),
+        Arrays.asList("Old Anatolian Turkish", "US", null, null, null),
+        Arrays.asList("Otjiwarongo Airport", "US", null, null, null),
+        Arrays.asList("President of India", "US", null, null, null)
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testInnerJoinEmptyRightSide() throws Exception
+  {
+    final ReadableInput factChannel = buildFactInput(
+        ImmutableList.of(
+            new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+            new KeyColumn("page", KeyOrder.ASCENDING)
+        )
+    );
+
+    final ReadableInput countriesChannel = ReadableInput.channel(
+        ReadableNilFrameChannel.INSTANCE,
+        FrameReader.create(JoinTestHelper.COUNTRIES_SIGNATURE),
+        STAGE_PARTITION
+    );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryName", ColumnType.STRING)
+                    .add("j0.countryNumber", ColumnType.LONG)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        countriesChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.INNER
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, Collections.emptyList());
+  }
+
+  @Test
+  public void testLeftJoinCountryIsoCode() throws Exception
+  {
+    final ReadableInput factChannel = buildFactInput(
+        ImmutableList.of(
+            new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+            new KeyColumn("page", KeyOrder.ASCENDING)
+        )
+    );
+
+    final ReadableInput countriesChannel =
+        buildCountriesInput(ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)));
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryName", ColumnType.STRING)
+                    .add("j0.countryNumber", ColumnType.LONG)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        countriesChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.LEFT
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList("Agama mossambica", null, null, null, null),
+        Arrays.asList("Apamea abruzzorum", null, null, null, null),
+        Arrays.asList("Atractus flammigerus", null, null, null, null),
+        Arrays.asList("Rallicula", null, null, null, null),
+        Arrays.asList("Talk:Oswald Tilghman", null, null, null, null),
+        Arrays.asList("Peremptory norm", "AU", "AU", "Australia", 0L),
+        Arrays.asList("Didier Leclair", "CA", "CA", "Canada", 1L),
+        Arrays.asList("Les Argonautes", "CA", "CA", "Canada", 1L),
+        Arrays.asList("Sarah Michelle Gellar", "CA", "CA", "Canada", 1L),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "CL", "CL", "Chile", 2L),
+        Arrays.asList("Diskussion:Sebastian Schulz", "DE", "DE", "Germany", 3L),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "EC", "EC", "Ecuador", 4L),
+        Arrays.asList("Saison 9 de Secret Story", "FR", "FR", "France", 5L),
+        Arrays.asList("Glasgow", "GB", "GB", "United Kingdom", 6L),
+        Arrays.asList("Giusy Ferreri discography", "IT", "IT", "Italy", 7L),
+        Arrays.asList("Roma-Bangkok", "IT", "IT", "Italy", 7L),
+        Arrays.asList("青野武", "JP", "JP", "Japan", 8L),
+        Arrays.asList("유희왕 GX", "KR", "KR", "Republic of Korea", 9L),
+        Arrays.asList("History of Fourems", "MMMM", "MMMM", "Fourems", 205L),
+        Arrays.asList("Mathis Bolly", "MX", "MX", "Mexico", 10L),
+        Arrays.asList("Orange Soda", "MatchNothing", null, null, null),
+        Arrays.asList("Алиса в Зазеркалье", "NO", "NO", "Norway", 11L),
+        Arrays.asList("Cream Soda", "SU", "SU", "States United", 15L),
+        Arrays.asList("Wendigo", "SV", "SV", "El Salvador", 12L),
+        Arrays.asList("Carlo Curti", "US", "US", "United States", 13L),
+        Arrays.asList("DirecTV", "US", "US", "United States", 13L),
+        Arrays.asList("Old Anatolian Turkish", "US", "US", "United States", 13L),
+        Arrays.asList("Otjiwarongo Airport", "US", "US", "United States", 13L),
+        Arrays.asList("President of India", "US", "US", "United States", 13L)
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testCrossJoin() throws Exception
+  {
+    final ReadableInput factChannel = buildFactInput(
+        ImmutableList.of(
+            new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+            new KeyColumn("page", KeyOrder.ASCENDING)
+        )
+    );
+
+    final ReadableInput countriesChannel = makeChannelFromResourceWithLimit(
+        JoinTestHelper.COUNTRIES_RESOURCE,
+        JoinTestHelper.COUNTRIES_SIGNATURE,
+        ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)),
+        2
+    );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("j0.page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        countriesChannel,
+        factChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(Collections.emptyList(), Collections.emptyList()),
+        JoinType.INNER
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList("Agama mossambica", "AU"),
+        Arrays.asList("Agama mossambica", "CA"),
+        Arrays.asList("Apamea abruzzorum", "AU"),
+        Arrays.asList("Apamea abruzzorum", "CA"),
+        Arrays.asList("Atractus flammigerus", "AU"),
+        Arrays.asList("Atractus flammigerus", "CA"),
+        Arrays.asList("Rallicula", "AU"),
+        Arrays.asList("Rallicula", "CA"),
+        Arrays.asList("Talk:Oswald Tilghman", "AU"),
+        Arrays.asList("Talk:Oswald Tilghman", "CA"),
+        Arrays.asList("Peremptory norm", "AU"),
+        Arrays.asList("Peremptory norm", "CA"),
+        Arrays.asList("Didier Leclair", "AU"),
+        Arrays.asList("Didier Leclair", "CA"),
+        Arrays.asList("Les Argonautes", "AU"),
+        Arrays.asList("Les Argonautes", "CA"),
+        Arrays.asList("Sarah Michelle Gellar", "AU"),
+        Arrays.asList("Sarah Michelle Gellar", "CA"),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "AU"),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "CA"),
+        Arrays.asList("Diskussion:Sebastian Schulz", "AU"),
+        Arrays.asList("Diskussion:Sebastian Schulz", "CA"),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "AU"),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "CA"),
+        Arrays.asList("Saison 9 de Secret Story", "AU"),
+        Arrays.asList("Saison 9 de Secret Story", "CA"),
+        Arrays.asList("Glasgow", "AU"),
+        Arrays.asList("Glasgow", "CA"),
+        Arrays.asList("Giusy Ferreri discography", "AU"),
+        Arrays.asList("Giusy Ferreri discography", "CA"),
+        Arrays.asList("Roma-Bangkok", "AU"),
+        Arrays.asList("Roma-Bangkok", "CA"),
+        Arrays.asList("青野武", "AU"),
+        Arrays.asList("青野武", "CA"),
+        Arrays.asList("유희왕 GX", "AU"),
+        Arrays.asList("유희왕 GX", "CA"),
+        Arrays.asList("History of Fourems", "AU"),
+        Arrays.asList("History of Fourems", "CA"),
+        Arrays.asList("Mathis Bolly", "AU"),
+        Arrays.asList("Mathis Bolly", "CA"),
+        Arrays.asList("Orange Soda", "AU"),
+        Arrays.asList("Orange Soda", "CA"),
+        Arrays.asList("Алиса в Зазеркалье", "AU"),
+        Arrays.asList("Алиса в Зазеркалье", "CA"),
+        Arrays.asList("Cream Soda", "AU"),
+        Arrays.asList("Cream Soda", "CA"),
+        Arrays.asList("Wendigo", "AU"),
+        Arrays.asList("Wendigo", "CA"),
+        Arrays.asList("Carlo Curti", "AU"),
+        Arrays.asList("Carlo Curti", "CA"),
+        Arrays.asList("DirecTV", "AU"),
+        Arrays.asList("DirecTV", "CA"),
+        Arrays.asList("Old Anatolian Turkish", "AU"),
+        Arrays.asList("Old Anatolian Turkish", "CA"),
+        Arrays.asList("Otjiwarongo Airport", "AU"),
+        Arrays.asList("Otjiwarongo Airport", "CA"),
+        Arrays.asList("President of India", "AU"),
+        Arrays.asList("President of India", "CA")
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testLeftJoinRegions() throws Exception
+  {
+    final ReadableInput factChannel =
+        buildFactInput(
+            ImmutableList.of(
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("page", KeyOrder.ASCENDING)
+            )
+        );
+
+    final ReadableInput regionsChannel =
+        buildRegionsInput(
+            ImmutableList.of(
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING)
+            )
+        );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("j0.regionName", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        regionsChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING)
+            ),
+            ImmutableList.of(
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING)
+            )
+        ),
+        JoinType.LEFT
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList("Agama mossambica", null, null),
+        Arrays.asList("Apamea abruzzorum", null, null),
+        Arrays.asList("Atractus flammigerus", null, null),
+        Arrays.asList("Rallicula", null, null),
+        Arrays.asList("Talk:Oswald Tilghman", null, null),
+        Arrays.asList("Peremptory norm", "New South Wales", "AU"),
+        Arrays.asList("Didier Leclair", "Ontario", "CA"),
+        Arrays.asList("Sarah Michelle Gellar", "Ontario", "CA"),
+        Arrays.asList("Les Argonautes", "Quebec", "CA"),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "Santiago Metropolitan", "CL"),
+        Arrays.asList("Diskussion:Sebastian Schulz", "Hesse", "DE"),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "Provincia del Guayas", "EC"),
+        Arrays.asList("Saison 9 de Secret Story", "Val d'Oise", "FR"),
+        Arrays.asList("Glasgow", "Kingston upon Hull", "GB"),
+        Arrays.asList("Giusy Ferreri discography", "Provincia di Varese", "IT"),
+        Arrays.asList("Roma-Bangkok", "Provincia di Varese", "IT"),
+        Arrays.asList("青野武", "Tōkyō", "JP"),
+        Arrays.asList("유희왕 GX", "Seoul", "KR"),
+        Arrays.asList("History of Fourems", "Fourems Province", "MMMM"),
+        Arrays.asList("Mathis Bolly", "Mexico City", "MX"),
+        Arrays.asList("Orange Soda", null, "MatchNothing"),
+        Arrays.asList("Алиса в Зазеркалье", "Finnmark Fylke", "NO"),
+        Arrays.asList("Cream Soda", "Ainigriv", "SU"),
+        Arrays.asList("Wendigo", "Departamento de San Salvador", "SV"),
+        Arrays.asList("Carlo Curti", "California", "US"),
+        Arrays.asList("Otjiwarongo Airport", "California", "US"),
+        Arrays.asList("President of India", "California", "US"),
+        Arrays.asList("DirecTV", "North Carolina", "US"),
+        Arrays.asList("Old Anatolian Turkish", "Virginia", "US")
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testRightJoinRegionCodeOnly() throws Exception
+  {
+    // This join generates duplicates.
+
+    final ReadableInput factChannel =
+        buildFactInput(
+            ImmutableList.of(
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("page", KeyOrder.ASCENDING)
+            )
+        );
+
+    final ReadableInput regionsChannel =
+        buildRegionsInput(
+            ImmutableList.of(
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)
+            )
+        );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("j0.page", ColumnType.STRING)
+                    .add("regionName", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        regionsChannel,
+        factChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("regionIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("regionIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.RIGHT
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList("Agama mossambica", null, null),
+        Arrays.asList("Apamea abruzzorum", null, null),
+        Arrays.asList("Atractus flammigerus", null, null),
+        Arrays.asList("Rallicula", null, null),
+        Arrays.asList("Talk:Oswald Tilghman", null, null),
+        Arrays.asList("유희왕 GX", "Seoul", "KR"),
+        Arrays.asList("青野武", "Tōkyō", "JP"),
+        Arrays.asList("Алиса в Зазеркалье", "Finnmark Fylke", "NO"),
+        Arrays.asList("Saison 9 de Secret Story", "Val d'Oise", "FR"),
+        Arrays.asList("Cream Soda", "Ainigriv", "SU"),
+        Arrays.asList("Carlo Curti", "California", "US"),
+        Arrays.asList("Otjiwarongo Airport", "California", "US"),
+        Arrays.asList("President of India", "California", "US"),
+        Arrays.asList("Mathis Bolly", "Mexico City", "MX"),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "Provincia del Guayas", "EC"),
+        Arrays.asList("Diskussion:Sebastian Schulz", "Hesse", "DE"),
+        Arrays.asList("Glasgow", "Kingston upon Hull", "GB"),
+        Arrays.asList("History of Fourems", "Fourems Province", "MMMM"),
+        Arrays.asList("Orange Soda", null, "MatchNothing"),
+        Arrays.asList("DirecTV", "North Carolina", "US"),
+        Arrays.asList("Peremptory norm", "New South Wales", "AU"),
+        Arrays.asList("Didier Leclair", "Ontario", "CA"),
+        Arrays.asList("Sarah Michelle Gellar", "Ontario", "CA"),
+        Arrays.asList("Les Argonautes", "Quebec", "CA"),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "Santiago Metropolitan", "CL"),
+        Arrays.asList("Wendigo", "Departamento de San Salvador", "SV"),
+        Arrays.asList("Giusy Ferreri discography", "Provincia di Varese", "IT"),
+        Arrays.asList("Giusy Ferreri discography", "Virginia", "IT"),
+        Arrays.asList("Old Anatolian Turkish", "Provincia di Varese", "US"),
+        Arrays.asList("Old Anatolian Turkish", "Virginia", "US"),
+        Arrays.asList("Roma-Bangkok", "Provincia di Varese", "IT"),
+        Arrays.asList("Roma-Bangkok", "Virginia", "IT")
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testFullOuterJoinRegionCodeOnly() throws Exception
+  {
+    // This join generates duplicates.
+
+    final ReadableInput factChannel =
+        buildFactInput(
+            ImmutableList.of(
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("page", KeyOrder.ASCENDING)
+            )
+        );
+
+    final ReadableInput regionsChannel =
+        buildRegionsInput(
+            ImmutableList.of(
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)
+            )
+        );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("j0.page", ColumnType.STRING)
+                    .add("regionName", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        regionsChannel,
+        factChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("regionIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("regionIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.FULL
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList(null, "Nulland", null),
+        Arrays.asList("Agama mossambica", null, null),
+        Arrays.asList("Apamea abruzzorum", null, null),
+        Arrays.asList("Atractus flammigerus", null, null),
+        Arrays.asList("Rallicula", null, null),
+        Arrays.asList("Talk:Oswald Tilghman", null, null),
+        Arrays.asList("유희왕 GX", "Seoul", "KR"),
+        Arrays.asList("青野武", "Tōkyō", "JP"),
+        Arrays.asList("Алиса в Зазеркалье", "Finnmark Fylke", "NO"),
+        Arrays.asList("Saison 9 de Secret Story", "Val d'Oise", "FR"),
+        Arrays.asList(null, "Foureis Province", null),
+        Arrays.asList("Cream Soda", "Ainigriv", "SU"),
+        Arrays.asList("Carlo Curti", "California", "US"),
+        Arrays.asList("Otjiwarongo Airport", "California", "US"),
+        Arrays.asList("President of India", "California", "US"),
+        Arrays.asList("Mathis Bolly", "Mexico City", "MX"),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "Provincia del Guayas", "EC"),
+        Arrays.asList("Diskussion:Sebastian Schulz", "Hesse", "DE"),
+        Arrays.asList("Glasgow", "Kingston upon Hull", "GB"),
+        Arrays.asList("History of Fourems", "Fourems Province", "MMMM"),
+        Arrays.asList("Orange Soda", null, "MatchNothing"),
+        Arrays.asList("DirecTV", "North Carolina", "US"),
+        Arrays.asList("Peremptory norm", "New South Wales", "AU"),
+        Arrays.asList("Didier Leclair", "Ontario", "CA"),
+        Arrays.asList("Sarah Michelle Gellar", "Ontario", "CA"),
+        Arrays.asList("Les Argonautes", "Quebec", "CA"),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "Santiago Metropolitan", "CL"),
+        Arrays.asList("Wendigo", "Departamento de San Salvador", "SV"),
+        Arrays.asList("Giusy Ferreri discography", "Provincia di Varese", "IT"),
+        Arrays.asList("Giusy Ferreri discography", "Virginia", "IT"),
+        Arrays.asList("Old Anatolian Turkish", "Provincia di Varese", "US"),
+        Arrays.asList("Old Anatolian Turkish", "Virginia", "US"),
+        Arrays.asList("Roma-Bangkok", "Provincia di Varese", "IT"),
+        Arrays.asList("Roma-Bangkok", "Virginia", "IT"),
+        Arrays.asList(null, "Usca City", null)
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testLeftJoinCountryNumber() throws Exception
+  {
+    final ReadableInput factChannel = buildFactInput(
+        ImmutableList.of(
+            new KeyColumn("countryNumber", KeyOrder.ASCENDING),
+            new KeyColumn("page", KeyOrder.ASCENDING)
+        )
+    );
+
+    final ReadableInput countriesChannel =
+        buildCountriesInput(ImmutableList.of(new KeyColumn("countryNumber", KeyOrder.ASCENDING)));
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryName", ColumnType.STRING)
+                    .add("j0.countryNumber", ColumnType.LONG)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        countriesChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("countryNumber", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("countryNumber", KeyOrder.ASCENDING))
+        ),
+        JoinType.LEFT
+    );
+
+    final String countryCodeForNull;
+    final String countryNameForNull;
+    final Long countryNumberForNull;
+
+    if (NullHandling.sqlCompatible()) {
+      countryCodeForNull = null;
+      countryNameForNull = null;
+      countryNumberForNull = null;
+    } else {
+      // In default-value mode, null country number from the left-hand table converts to zero, which matches Australia.
+      countryCodeForNull = "AU";
+      countryNameForNull = "Australia";
+      countryNumberForNull = 0L;

Review Comment:
   Sure. Although, it's accurate as-is, because the null from the left-hand side really does convert to zero, since the left-hand side isn't using nested columns. Any suggestions on how to rewrite the comment to be better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on PR #13506:
URL: https://github.com/apache/druid/pull/13506#issuecomment-1439929888

   > Okay, I've now read through all of the classes. Or, read them as best I can. I'm a newbie to this code, so the best I can do is a surface-level review, which is what I've done. Most of my comments are stream-of-consciousness because, I fear that I lack the context to really comment meaningfully.
   
   Thanks, I still appreciate it.
   
   > In terms of approving merge or not approving merge. I don't know how "safe" this is. It seems to touch a lot of different files and there does seem to be good test coverage, is this safe to just merge and it'll work? Is there anything extra done in the code to ensure that the old code paths are untouched in the worst case that there is a bug in here?
   
   The SQL planning changes are designed to be low risk when not using sort-merge joins. The new code is generally additive and the bulk of it will not execute for the default (broadcast) join algorithm.
   
   The MSQ and processing changes, other than the core sort-merge logic, are mostly related to adding hash partitioning. I tried to keep this as minimally invasive as possible, but it did have to touch various places and it could not be purely additive. To minimize the risk, I've added new tests. I will fix any problems that turn up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1042823240


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;

Review Comment:
   Why not have 2 fields: a left and a right ReadableFrameChannel instead of a List that you do `.get()` on all the time.



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Names of the key columns on each side of the join, unprefixed. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<List<KeyColumn>> keyColumns;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputs = ImmutableList.of(left, right);
+    this.inputChannels = inputs.stream().map(ReadableInput::getChannel).collect(Collectors.toList());
+    this.keyColumns = keyColumns;
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(new Tracker(LEFT), new Tracker(RIGHT));
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()
+           && !(trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd())) {
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+      final boolean match = markCmp == 0 && !trackers.get(LEFT).hasPartiallyNullMark();
+
+      // Fetch new frames until the algorithm can proceed:
+      //   1) If marked keys are equal on both sides, at least one side has a complete set of rows for the marked key.
+      //   2) Otherwise, each side has at least one readable row, or is completely done. (Checked by "while" condition.)
+      if (match && trackerWithCompleteSetForCurrentKey < 0) {
+        for (int i = 0; i < inputChannels.size(); i++) {
+          final Tracker tracker = trackers.get(i);
+          if (tracker.hasCompleteSetForMark() || (pushNextFrame(i) && tracker.hasCompleteSetForMark())) {

Review Comment:
   > (pushNextFrame(i) && tracker.hasCompleteSetForMark())
   
   It's unclear to me why it is sufficient to only push a single next frame and check.  This sort of thing is usually in a while-loop, no?



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Names of the key columns on each side of the join, unprefixed. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<List<KeyColumn>> keyColumns;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputs = ImmutableList.of(left, right);
+    this.inputChannels = inputs.stream().map(ReadableInput::getChannel).collect(Collectors.toList());
+    this.keyColumns = keyColumns;
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(new Tracker(LEFT), new Tracker(RIGHT));
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()
+           && !(trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd())) {

Review Comment:
   This boolean is large...  There's lots of negation.  It might be nice to negate a large OR instead of 3 negated-ANDs, but if there's any way to make this boolean simpler, that might be nice.



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Names of the key columns on each side of the join, unprefixed. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<List<KeyColumn>> keyColumns;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputs = ImmutableList.of(left, right);
+    this.inputChannels = inputs.stream().map(ReadableInput::getChannel).collect(Collectors.toList());
+    this.keyColumns = keyColumns;
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(new Tracker(LEFT), new Tracker(RIGHT));
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()
+           && !(trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd())) {
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+      final boolean match = markCmp == 0 && !trackers.get(LEFT).hasPartiallyNullMark();
+
+      // Fetch new frames until the algorithm can proceed:
+      //   1) If marked keys are equal on both sides, at least one side has a complete set of rows for the marked key.
+      //   2) Otherwise, each side has at least one readable row, or is completely done. (Checked by "while" condition.)
+      if (match && trackerWithCompleteSetForCurrentKey < 0) {
+        for (int i = 0; i < inputChannels.size(); i++) {
+          final Tracker tracker = trackers.get(i);
+          if (tracker.hasCompleteSetForMark() || (pushNextFrame(i) && tracker.hasCompleteSetForMark())) {
+            trackerWithCompleteSetForCurrentKey = i;
+            break;
+          }
+        }
+
+        if (trackerWithCompleteSetForCurrentKey < 0) {
+          // Algorithm cannot proceed; fetch more frames.
+          return nextAwait();
+        }
+      }
+
+      if (match || (markCmp <= 0 && joinType.isLefty()) || (markCmp >= 0 && joinType.isRighty())) {
+        // Emit row, if there's room in the current frameWriter.
+        joinColumnSelectorFactory.cmp = markCmp;
+        joinColumnSelectorFactory.match = match;
+
+        if (!frameWriter.addSelection()) {
+          if (frameWriter.getNumRows() > 0) {
+            // Out of space in the current frame. Run again without moving cursors.
+            flushCurrentFrame();
+            return ReturnOrAwait.runAgain();
+          } else {
+            throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
+          }
+        }
+      }
+
+      // Advance one or both trackers.
+      if (match) {
+        // Matching keys. First advance the tracker with the complete set.
+        final Tracker tracker = trackers.get(trackerWithCompleteSetForCurrentKey);
+        final Tracker otherTracker = trackers.get(trackerWithCompleteSetForCurrentKey == LEFT ? RIGHT : LEFT);
+
+        tracker.advance();
+        if (!tracker.isCurrentSameKeyAsMark()) {
+          // Reached end of complete set. Advance the other tracker.
+          otherTracker.advance();
+
+          // On next iteration (when we're sure to have data) either rewind the complete-set tracker, or update marks
+          // of both, as appropriate.
+          onNextIteration(() -> {
+            if (otherTracker.isCurrentSameKeyAsMark()) {
+              otherTracker.markCurrent(); // Set mark to enable cleanup of old frames.
+              tracker.rewindToMark();
+            } else {
+              // Reached end of the other side too. Advance marks on both trackers.
+              tracker.markCurrent();
+              otherTracker.markCurrent();
+              trackerWithCompleteSetForCurrentKey = -1;
+            }
+          });
+        }
+      } else {
+        final int trackerToAdvance;
+
+        if (markCmp < 0) {
+          trackerToAdvance = LEFT;
+        } else if (markCmp > 0) {
+          trackerToAdvance = RIGHT;
+        } else {
+          // Key is null on both sides. Note that there is a preference for running through the left side first
+          // on a FULL join. It doesn't really matter which side we run through first, but we do need to be consistent
+          // for the benefit of the logic in "shouldEmitColumnValue".
+          trackerToAdvance = joinType.isLefty() ? LEFT : RIGHT;
+        }
+
+        final Tracker tracker = trackers.get(trackerToAdvance);
+
+        tracker.advance();
+
+        // On next iteration (when we're sure to have data), update mark if the key changed.
+        onNextIteration(() -> {
+          if (!tracker.isCurrentSameKeyAsMark()) {
+            tracker.markCurrent();
+            trackerWithCompleteSetForCurrentKey = -1;
+          }
+        });
+      }
+    }
+
+    if (trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd()) {
+      // Both channels completely done.
+      flushCurrentFrame();
+      return ReturnOrAwait.returnObject(0L);
+    } else {
+      // Keep reading.
+      return nextAwait();
+    }
+  }
+
+  @Override
+  public void cleanup() throws IOException
+  {
+    FrameProcessors.closeAll(inputChannels(), outputChannels(), frameWriter, () -> trackers.forEach(Tracker::clear));
+  }
+
+  /**
+   * Returns a {@link ReturnOrAwait#awaitAll} for the channel numbers that need more data and have not yet hit their
+   * buffered-bytes limit, {@link Limits#MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN}.
+   *
+   * If all channels have hit their limit, throws {@link MSQException} with {@link TooManyRowsWithSameKeyFault}.
+   */
+  private ReturnOrAwait<Long> nextAwait()
+  {
+    final IntSet awaitSet = new IntOpenHashSet();
+    int trackerAtLimit = -1;
+
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.needsMoreData()) {
+        if (tracker.totalBytesBuffered() < Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+          awaitSet.add(i);
+        } else if (trackerAtLimit < 0) {
+          trackerAtLimit = i;
+        }
+      }
+    }
+
+    if (awaitSet.isEmpty() && trackerAtLimit > 0) {
+      // All trackers that need more data are at their max buffered bytes limit. Generate a nice exception.
+      final Tracker tracker = trackers.get(trackerAtLimit);
+      if (tracker.totalBytesBuffered() > Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+        // Generate a nice exception.
+        throw new MSQException(
+            new TooManyRowsWithSameKeyFault(
+                tracker.readMarkKey(),
+                tracker.totalBytesBuffered(),
+                Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN
+            )
+        );
+      }
+    }
+
+    return ReturnOrAwait.awaitAll(awaitSet);
+  }
+
+  /**
+   * Compares the marked rows of the two {@link #trackers}.
+   *
+   * @throws IllegalStateException if either tracker does not have a marked row and is not completely done
+   */
+  private int compareMarks()
+  {
+    final Tracker leftTracker = trackers.get(LEFT);
+    final Tracker rightTracker = trackers.get(RIGHT);
+
+    Preconditions.checkState(leftTracker.hasMark() || leftTracker.isAtEnd(), "left.hasMark || left.isAtEnd");
+    Preconditions.checkState(rightTracker.hasMark() || rightTracker.isAtEnd(), "right.hasMark || right.isAtEnd");
+
+    if (leftTracker.markFrame < 0) {
+      return rightTracker.markFrame < 0 ? 0 : 1;
+    } else if (rightTracker.markFrame < 0) {
+      return -1;
+    } else {
+      final FrameHolder leftHolder = leftTracker.holders.get(leftTracker.markFrame);
+      final FrameHolder rightHolder = rightTracker.holders.get(rightTracker.markFrame);
+      return leftHolder.comparisonWidget.compare(
+          leftTracker.markRow,
+          rightHolder.comparisonWidget,
+          rightTracker.markRow
+      );
+    }
+  }
+
+  /**
+   * Pushes a frame from the indicated channel into the appropriate tracker. Returns true if a frame was pushed
+   * or if the channel is finished.
+   */
+  private boolean pushNextFrame(final int channelNumber)
+  {
+    final ReadableFrameChannel channel = inputChannels.get(channelNumber);
+    final Tracker tracker = trackers.get(channelNumber);
+
+    if (!channel.isFinished() && !channel.canRead()) {
+      return false;
+    } else if (channel.isFinished()) {
+      tracker.push(null);
+      return true;
+    } else {
+      final Frame frame = channel.read();
+
+      if (frame.numRows() == 0) {
+        // Skip, read next.
+        return false;
+      } else {
+        tracker.push(frame);
+        return true;
+      }
+    }
+  }
+
+  private void onNextIteration(final Runnable runnable)
+  {
+    if (nextIterationRunnable != null) {
+      throw new ISE("postAdvanceRunnable already set");
+    } else {
+      nextIterationRunnable = runnable;
+    }
+  }
+
+  private void startNewFrameIfNeeded()
+  {
+    if (frameWriter == null) {
+      frameWriter = frameWriterFactory.newFrameWriter(joinColumnSelectorFactory);
+    }
+  }
+
+  private void flushCurrentFrame() throws IOException
+  {
+    if (frameWriter != null) {
+      if (frameWriter.getNumRows() > 0) {
+        final Frame frame = Frame.wrap(frameWriter.toByteArray());
+        frameWriter.close();
+        frameWriter = null;
+        outputChannel.write(new FrameWithPartition(frame, FrameWithPartition.NO_PARTITION));
+      }
+    }
+  }
+
+  /**
+   * Tracks the current set of rows that have the same key from a sequence of frames.
+   *
+   * markFrame and markRow are set when we encounter a new key, which enables rewinding and re-reading data with the
+   * same key.
+   */
+  private class Tracker
+  {
+    private final List<FrameHolder> holders = new ArrayList<>();
+    private final int channelNumber;
+
+    // markFrame and markRow are the first frame and row with the current key.
+    private int markFrame = -1;
+    private int markRow = -1;
+
+    // currentFrame is the frame containing the current cursor row.
+    private int currentFrame = -1;
+
+    // done indicates that no more data is available in the channel.
+    private boolean done;
+
+    public Tracker(int channelNumber)
+    {
+      this.channelNumber = channelNumber;
+    }
+
+    /**
+     * Adds a holder for a frame. If this is the first frame, sets the current cursor position and mark to the first
+     * row of the frame. Otherwise, the cursor position and mark are not changed.
+     *
+     * Pushing a null frame indicates no more frames are coming.
+     *
+     * @param frame frame, or null indicating no more frames are coming
+     */
+    public void push(final Frame frame)
+    {
+      if (frame == null) {
+        done = true;
+        return;
+      }
+
+      if (done) {
+        throw new ISE("Cannot push frames when already done");
+      }
+
+      final boolean atEndOfPushedData = isAtEndOfPushedData();
+      final FrameReader frameReader = inputs.get(channelNumber).getChannelFrameReader();
+      final FrameCursor cursor = FrameProcessors.makeCursor(frame, frameReader);
+      final FrameComparisonWidget comparisonWidget =
+          frameReader.makeComparisonWidget(frame, keyColumns.get(channelNumber));
+
+      final RowSignature.Builder keySignatureBuilder = RowSignature.builder();
+      for (final KeyColumn keyColumn : keyColumns.get(channelNumber)) {
+        keySignatureBuilder.add(
+            keyColumn.columnName(),
+            frameReader.signature().getColumnType(keyColumn.columnName()).orElse(null)
+        );
+      }
+
+      holders.add(
+          new FrameHolder(
+              frame,
+              RowKeyReader.create(keySignatureBuilder.build()),
+              cursor,
+              comparisonWidget
+          )
+      );
+
+      if (atEndOfPushedData) {
+        // Move currentFrame so it points at the next row, which we now have, instead of an "isDone" cursor.
+        currentFrame = currentFrame < 0 ? 0 : currentFrame + 1;
+      }
+
+      if (markFrame < 0) {
+        // Cleared mark means we want the current row to be marked.
+        markFrame = currentFrame;
+        markRow = 0;
+      }
+    }
+
+    /**
+     * Number of bytes currently buffered in {@link #holders}.
+     */
+    public long totalBytesBuffered()
+    {
+      long bytes = 0;
+      for (final FrameHolder holder : holders) {
+        bytes += holder.frame.numBytes();
+      }
+      return bytes;
+    }
+
+    /**
+     * Cursor containing the current row.
+     */
+    @Nullable
+    public FrameCursor currentCursor()
+    {
+      if (currentFrame < 0) {
+        return null;
+      } else {
+        return holders.get(currentFrame).cursor;
+      }
+    }
+
+    /**
+     * Advances the current row (the current row of {@link #currentFrame}). After calling this method,
+     * {@link #isAtEndOfPushedData()} may start returning true.
+     */
+    public void advance()
+    {
+      assert !isAtEndOfPushedData();
+
+      final FrameHolder currentHolder = holders.get(currentFrame);
+
+      currentHolder.cursor.advance();
+
+      if (currentHolder.cursor.isDone() && currentFrame + 1 < holders.size()) {
+        currentFrame++;
+        holders.get(currentFrame).cursor.reset();
+      }

Review Comment:
   This appears to never be letting go of the previous FrameHolder objects?  It can effectively null out the old holder, right?



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Names of the key columns on each side of the join, unprefixed. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<List<KeyColumn>> keyColumns;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputs = ImmutableList.of(left, right);
+    this.inputChannels = inputs.stream().map(ReadableInput::getChannel).collect(Collectors.toList());
+    this.keyColumns = keyColumns;
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(new Tracker(LEFT), new Tracker(RIGHT));
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()
+           && !(trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd())) {
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+      final boolean match = markCmp == 0 && !trackers.get(LEFT).hasPartiallyNullMark();

Review Comment:
   As I read this line, I'm unclear why "partially null" matters.  Maybe it'll become clearer as I read more.  I went back up to the class level javadoc and skimmed it one more time, but didn't see the relevance of null called out.



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;

Review Comment:
   Or, given that you have multiple lists here.  Maybe what you need is a `JoinChannelContainer` or something like that which has a reference to all of `ReadableInput`, `ReadableFrameChannel`, `List<KeyColumn>`, `Tracker`?



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessor.java:
##########
@@ -0,0 +1,1054 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import it.unimi.dsi.fastutil.ints.IntOpenHashSet;
+import it.unimi.dsi.fastutil.ints.IntSet;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.channel.FrameWithPartition;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.WritableFrameChannel;
+import org.apache.druid.frame.key.FrameComparisonWidget;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.RowKey;
+import org.apache.druid.frame.key.RowKeyReader;
+import org.apache.druid.frame.processor.FrameProcessor;
+import org.apache.druid.frame.processor.FrameProcessors;
+import org.apache.druid.frame.processor.FrameRowTooLargeException;
+import org.apache.druid.frame.processor.ReturnOrAwait;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameCursor;
+import org.apache.druid.frame.write.FrameWriter;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.msq.exec.Limits;
+import org.apache.druid.msq.indexing.error.MSQException;
+import org.apache.druid.msq.indexing.error.TooManyRowsWithSameKeyFault;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.filter.ValueMatcher;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.ColumnValueSelector;
+import org.apache.druid.segment.Cursor;
+import org.apache.druid.segment.DimensionSelector;
+import org.apache.druid.segment.DimensionSelectorUtils;
+import org.apache.druid.segment.IdLookup;
+import org.apache.druid.segment.NilColumnValueSelector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.data.IndexedInts;
+import org.apache.druid.segment.data.ZeroIndexedInts;
+import org.apache.druid.segment.join.JoinPrefixUtils;
+import org.apache.druid.segment.join.JoinType;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Processor for a sort-merge join of two inputs.
+ *
+ * Prerequisites:
+ *
+ * 1) Two inputs, both of which are stages; i.e. {@link ReadableInput#hasChannel()}.
+ *
+ * 2) Conditions are all simple equalities. Validated by {@link SortMergeJoinFrameProcessorFactory#validateCondition}
+ * and then transformed to lists of key columns by {@link SortMergeJoinFrameProcessorFactory#toKeyColumns}.
+ *
+ * 3) Both inputs are comprised of {@link org.apache.druid.frame.FrameType#ROW_BASED} frames, are sorted by the same
+ * key, and that key can be used to check the provided condition. Validated by
+ * {@link SortMergeJoinFrameProcessorFactory#validateInputFrameSignatures}.
+ *
+ * Algorithm:
+ *
+ * 1) Read current key from each side of the join.
+ *
+ * 2) If there is no match, emit or skip the row for the earlier key, as appropriate, based on the join type.
+ *
+ * 3) If there is a match, identify a complete set on one side or the other. (It doesn't matter which side has the
+ * complete set, but we need it on one of them.) We mark the first row for the key using {@link Tracker#markCurrent()}
+ * and find complete sets using {@link Tracker#hasCompleteSetForMark()}. Once we find one, we store it in
+ * {@link #trackerWithCompleteSetForCurrentKey}. If both sides have a complete set, we break ties by choosing the
+ * left side.
+ *
+ * 4) Once a complete set for the current key is identified: for each row on the *other* side, loop through the entire
+ * set of rows on {@link #trackerWithCompleteSetForCurrentKey}, and emit that many joined rows.
+ *
+ * 5) Once we process the final row on the *other* side, reset both marks with {@link Tracker#markCurrent()} and
+ * continue the algorithm.
+ */
+public class SortMergeJoinFrameProcessor implements FrameProcessor<Long>
+{
+  private static final int LEFT = 0;
+  private static final int RIGHT = 1;
+
+  /**
+   * Two sides of the join. Must be channels; i.e. {@link ReadableInput#hasChannel()} must be true. Two-element array:
+   * {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableInput> inputs;
+
+  /**
+   * Channels from {@link #inputs}. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<ReadableFrameChannel> inputChannels;
+
+  /**
+   * Names of the key columns on each side of the join, unprefixed. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<List<KeyColumn>> keyColumns;
+
+  /**
+   * Trackers for each side of the join. Two-element array: {@link #LEFT} and {@link #RIGHT}.
+   */
+  private final List<Tracker> trackers;
+
+  private final WritableFrameChannel outputChannel;
+  private final FrameWriterFactory frameWriterFactory;
+  private final String rightPrefix;
+  private final JoinType joinType;
+  private final JoinColumnSelectorFactory joinColumnSelectorFactory = new JoinColumnSelectorFactory();
+  private FrameWriter frameWriter = null;
+
+  // Used by runIncrementally to defer certain logic to the next run.
+  private Runnable nextIterationRunnable = null;
+
+  // Used by runIncrementally to remember which tracker has the complete set for the current key.
+  private int trackerWithCompleteSetForCurrentKey = -1;
+
+  SortMergeJoinFrameProcessor(
+      ReadableInput left,
+      ReadableInput right,
+      WritableFrameChannel outputChannel,
+      FrameWriterFactory frameWriterFactory,
+      String rightPrefix,
+      List<List<KeyColumn>> keyColumns,
+      JoinType joinType
+  )
+  {
+    this.inputs = ImmutableList.of(left, right);
+    this.inputChannels = inputs.stream().map(ReadableInput::getChannel).collect(Collectors.toList());
+    this.keyColumns = keyColumns;
+    this.outputChannel = outputChannel;
+    this.frameWriterFactory = frameWriterFactory;
+    this.rightPrefix = rightPrefix;
+    this.joinType = joinType;
+    this.trackers = ImmutableList.of(new Tracker(LEFT), new Tracker(RIGHT));
+  }
+
+  @Override
+  public List<ReadableFrameChannel> inputChannels()
+  {
+    return inputChannels;
+  }
+
+  @Override
+  public List<WritableFrameChannel> outputChannels()
+  {
+    return Collections.singletonList(outputChannel);
+  }
+
+  @Override
+  public ReturnOrAwait<Long> runIncrementally(IntSet readableInputs) throws IOException
+  {
+    // Fetch enough frames such that each tracker has one readable row.
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.isAtEndOfPushedData() && !pushNextFrame(i)) {
+        return nextAwait();
+      }
+    }
+
+    // Initialize new output frame, if needed.
+    startNewFrameIfNeeded();
+
+    while (!trackers.get(LEFT).needsMoreData()
+           && !trackers.get(RIGHT).needsMoreData()
+           && !(trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd())) {
+      if (nextIterationRunnable != null) {
+        final Runnable tmp = nextIterationRunnable;
+        nextIterationRunnable = null;
+        tmp.run();
+      }
+
+      final int markCmp = compareMarks();
+      final boolean match = markCmp == 0 && !trackers.get(LEFT).hasPartiallyNullMark();
+
+      // Fetch new frames until the algorithm can proceed:
+      //   1) If marked keys are equal on both sides, at least one side has a complete set of rows for the marked key.
+      //   2) Otherwise, each side has at least one readable row, or is completely done. (Checked by "while" condition.)
+      if (match && trackerWithCompleteSetForCurrentKey < 0) {
+        for (int i = 0; i < inputChannels.size(); i++) {
+          final Tracker tracker = trackers.get(i);
+          if (tracker.hasCompleteSetForMark() || (pushNextFrame(i) && tracker.hasCompleteSetForMark())) {
+            trackerWithCompleteSetForCurrentKey = i;
+            break;
+          }
+        }
+
+        if (trackerWithCompleteSetForCurrentKey < 0) {
+          // Algorithm cannot proceed; fetch more frames.
+          return nextAwait();
+        }
+      }
+
+      if (match || (markCmp <= 0 && joinType.isLefty()) || (markCmp >= 0 && joinType.isRighty())) {
+        // Emit row, if there's room in the current frameWriter.
+        joinColumnSelectorFactory.cmp = markCmp;
+        joinColumnSelectorFactory.match = match;
+
+        if (!frameWriter.addSelection()) {
+          if (frameWriter.getNumRows() > 0) {
+            // Out of space in the current frame. Run again without moving cursors.
+            flushCurrentFrame();
+            return ReturnOrAwait.runAgain();
+          } else {
+            throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
+          }
+        }
+      }
+
+      // Advance one or both trackers.
+      if (match) {
+        // Matching keys. First advance the tracker with the complete set.
+        final Tracker tracker = trackers.get(trackerWithCompleteSetForCurrentKey);
+        final Tracker otherTracker = trackers.get(trackerWithCompleteSetForCurrentKey == LEFT ? RIGHT : LEFT);
+
+        tracker.advance();
+        if (!tracker.isCurrentSameKeyAsMark()) {
+          // Reached end of complete set. Advance the other tracker.
+          otherTracker.advance();
+
+          // On next iteration (when we're sure to have data) either rewind the complete-set tracker, or update marks
+          // of both, as appropriate.
+          onNextIteration(() -> {
+            if (otherTracker.isCurrentSameKeyAsMark()) {
+              otherTracker.markCurrent(); // Set mark to enable cleanup of old frames.
+              tracker.rewindToMark();
+            } else {
+              // Reached end of the other side too. Advance marks on both trackers.
+              tracker.markCurrent();
+              otherTracker.markCurrent();
+              trackerWithCompleteSetForCurrentKey = -1;
+            }
+          });
+        }
+      } else {
+        final int trackerToAdvance;
+
+        if (markCmp < 0) {
+          trackerToAdvance = LEFT;
+        } else if (markCmp > 0) {
+          trackerToAdvance = RIGHT;
+        } else {
+          // Key is null on both sides. Note that there is a preference for running through the left side first
+          // on a FULL join. It doesn't really matter which side we run through first, but we do need to be consistent
+          // for the benefit of the logic in "shouldEmitColumnValue".
+          trackerToAdvance = joinType.isLefty() ? LEFT : RIGHT;
+        }
+
+        final Tracker tracker = trackers.get(trackerToAdvance);
+
+        tracker.advance();
+
+        // On next iteration (when we're sure to have data), update mark if the key changed.
+        onNextIteration(() -> {
+          if (!tracker.isCurrentSameKeyAsMark()) {
+            tracker.markCurrent();
+            trackerWithCompleteSetForCurrentKey = -1;
+          }
+        });
+      }
+    }
+
+    if (trackers.get(LEFT).isAtEnd() && trackers.get(RIGHT).isAtEnd()) {
+      // Both channels completely done.
+      flushCurrentFrame();
+      return ReturnOrAwait.returnObject(0L);
+    } else {
+      // Keep reading.
+      return nextAwait();
+    }
+  }
+
+  @Override
+  public void cleanup() throws IOException
+  {
+    FrameProcessors.closeAll(inputChannels(), outputChannels(), frameWriter, () -> trackers.forEach(Tracker::clear));
+  }
+
+  /**
+   * Returns a {@link ReturnOrAwait#awaitAll} for the channel numbers that need more data and have not yet hit their
+   * buffered-bytes limit, {@link Limits#MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN}.
+   *
+   * If all channels have hit their limit, throws {@link MSQException} with {@link TooManyRowsWithSameKeyFault}.
+   */
+  private ReturnOrAwait<Long> nextAwait()
+  {
+    final IntSet awaitSet = new IntOpenHashSet();
+    int trackerAtLimit = -1;
+
+    for (int i = 0; i < inputChannels.size(); i++) {
+      final Tracker tracker = trackers.get(i);
+      if (tracker.needsMoreData()) {
+        if (tracker.totalBytesBuffered() < Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+          awaitSet.add(i);
+        } else if (trackerAtLimit < 0) {
+          trackerAtLimit = i;
+        }
+      }
+    }
+
+    if (awaitSet.isEmpty() && trackerAtLimit > 0) {
+      // All trackers that need more data are at their max buffered bytes limit. Generate a nice exception.
+      final Tracker tracker = trackers.get(trackerAtLimit);
+      if (tracker.totalBytesBuffered() > Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN) {
+        // Generate a nice exception.
+        throw new MSQException(
+            new TooManyRowsWithSameKeyFault(
+                tracker.readMarkKey(),
+                tracker.totalBytesBuffered(),
+                Limits.MAX_BUFFERED_BYTES_FOR_SORT_MERGE_JOIN
+            )
+        );
+      }
+    }
+
+    return ReturnOrAwait.awaitAll(awaitSet);
+  }
+
+  /**
+   * Compares the marked rows of the two {@link #trackers}.
+   *
+   * @throws IllegalStateException if either tracker does not have a marked row and is not completely done
+   */
+  private int compareMarks()
+  {
+    final Tracker leftTracker = trackers.get(LEFT);
+    final Tracker rightTracker = trackers.get(RIGHT);
+
+    Preconditions.checkState(leftTracker.hasMark() || leftTracker.isAtEnd(), "left.hasMark || left.isAtEnd");
+    Preconditions.checkState(rightTracker.hasMark() || rightTracker.isAtEnd(), "right.hasMark || right.isAtEnd");
+
+    if (leftTracker.markFrame < 0) {
+      return rightTracker.markFrame < 0 ? 0 : 1;
+    } else if (rightTracker.markFrame < 0) {
+      return -1;
+    } else {
+      final FrameHolder leftHolder = leftTracker.holders.get(leftTracker.markFrame);
+      final FrameHolder rightHolder = rightTracker.holders.get(rightTracker.markFrame);
+      return leftHolder.comparisonWidget.compare(
+          leftTracker.markRow,
+          rightHolder.comparisonWidget,
+          rightTracker.markRow
+      );
+    }
+  }
+
+  /**
+   * Pushes a frame from the indicated channel into the appropriate tracker. Returns true if a frame was pushed
+   * or if the channel is finished.
+   */
+  private boolean pushNextFrame(final int channelNumber)
+  {
+    final ReadableFrameChannel channel = inputChannels.get(channelNumber);
+    final Tracker tracker = trackers.get(channelNumber);
+
+    if (!channel.isFinished() && !channel.canRead()) {
+      return false;
+    } else if (channel.isFinished()) {
+      tracker.push(null);
+      return true;
+    } else {
+      final Frame frame = channel.read();
+
+      if (frame.numRows() == 0) {
+        // Skip, read next.
+        return false;
+      } else {
+        tracker.push(frame);
+        return true;
+      }
+    }
+  }
+
+  private void onNextIteration(final Runnable runnable)
+  {
+    if (nextIterationRunnable != null) {
+      throw new ISE("postAdvanceRunnable already set");
+    } else {
+      nextIterationRunnable = runnable;
+    }
+  }
+
+  private void startNewFrameIfNeeded()
+  {
+    if (frameWriter == null) {
+      frameWriter = frameWriterFactory.newFrameWriter(joinColumnSelectorFactory);
+    }
+  }
+
+  private void flushCurrentFrame() throws IOException
+  {
+    if (frameWriter != null) {
+      if (frameWriter.getNumRows() > 0) {
+        final Frame frame = Frame.wrap(frameWriter.toByteArray());
+        frameWriter.close();
+        frameWriter = null;
+        outputChannel.write(new FrameWithPartition(frame, FrameWithPartition.NO_PARTITION));
+      }
+    }
+  }
+
+  /**
+   * Tracks the current set of rows that have the same key from a sequence of frames.
+   *
+   * markFrame and markRow are set when we encounter a new key, which enables rewinding and re-reading data with the
+   * same key.
+   */
+  private class Tracker
+  {
+    private final List<FrameHolder> holders = new ArrayList<>();
+    private final int channelNumber;
+
+    // markFrame and markRow are the first frame and row with the current key.
+    private int markFrame = -1;
+    private int markRow = -1;
+
+    // currentFrame is the frame containing the current cursor row.
+    private int currentFrame = -1;
+
+    // done indicates that no more data is available in the channel.
+    private boolean done;
+
+    public Tracker(int channelNumber)
+    {
+      this.channelNumber = channelNumber;
+    }
+
+    /**
+     * Adds a holder for a frame. If this is the first frame, sets the current cursor position and mark to the first
+     * row of the frame. Otherwise, the cursor position and mark are not changed.
+     *
+     * Pushing a null frame indicates no more frames are coming.
+     *
+     * @param frame frame, or null indicating no more frames are coming
+     */
+    public void push(final Frame frame)
+    {
+      if (frame == null) {
+        done = true;
+        return;
+      }
+
+      if (done) {
+        throw new ISE("Cannot push frames when already done");
+      }
+
+      final boolean atEndOfPushedData = isAtEndOfPushedData();
+      final FrameReader frameReader = inputs.get(channelNumber).getChannelFrameReader();
+      final FrameCursor cursor = FrameProcessors.makeCursor(frame, frameReader);
+      final FrameComparisonWidget comparisonWidget =
+          frameReader.makeComparisonWidget(frame, keyColumns.get(channelNumber));
+
+      final RowSignature.Builder keySignatureBuilder = RowSignature.builder();
+      for (final KeyColumn keyColumn : keyColumns.get(channelNumber)) {
+        keySignatureBuilder.add(
+            keyColumn.columnName(),
+            frameReader.signature().getColumnType(keyColumn.columnName()).orElse(null)
+        );
+      }
+
+      holders.add(
+          new FrameHolder(
+              frame,
+              RowKeyReader.create(keySignatureBuilder.build()),
+              cursor,
+              comparisonWidget
+          )
+      );
+
+      if (atEndOfPushedData) {
+        // Move currentFrame so it points at the next row, which we now have, instead of an "isDone" cursor.
+        currentFrame = currentFrame < 0 ? 0 : currentFrame + 1;
+      }
+
+      if (markFrame < 0) {
+        // Cleared mark means we want the current row to be marked.
+        markFrame = currentFrame;
+        markRow = 0;
+      }
+    }
+
+    /**
+     * Number of bytes currently buffered in {@link #holders}.
+     */
+    public long totalBytesBuffered()
+    {
+      long bytes = 0;
+      for (final FrameHolder holder : holders) {
+        bytes += holder.frame.numBytes();
+      }
+      return bytes;
+    }
+
+    /**
+     * Cursor containing the current row.
+     */
+    @Nullable
+    public FrameCursor currentCursor()
+    {
+      if (currentFrame < 0) {
+        return null;
+      } else {
+        return holders.get(currentFrame).cursor;
+      }
+    }
+
+    /**
+     * Advances the current row (the current row of {@link #currentFrame}). After calling this method,
+     * {@link #isAtEndOfPushedData()} may start returning true.
+     */
+    public void advance()
+    {
+      assert !isAtEndOfPushedData();
+
+      final FrameHolder currentHolder = holders.get(currentFrame);
+
+      currentHolder.cursor.advance();
+
+      if (currentHolder.cursor.isDone() && currentFrame + 1 < holders.size()) {
+        currentFrame++;
+        holders.get(currentFrame).cursor.reset();
+      }
+    }
+
+    /**
+     * Whether this tracker has a marked row.
+     */
+    public boolean hasMark()
+    {
+      return markFrame >= 0;
+    }
+
+    /**
+     * Whether this tracker has a marked row that is partially null.
+     */
+    public boolean hasPartiallyNullMark()
+    {
+      return hasMark() && holders.get(markFrame).comparisonWidget.isPartiallyNullKey(markRow);
+    }
+
+    /**
+     * Reads the current marked key.
+     */
+    @Nullable
+    public List<Object> readMarkKey()
+    {
+      if (!hasMark()) {
+        return null;
+      }
+
+      final FrameHolder markHolder = holders.get(markFrame);
+      final RowKey markKey = markHolder.comparisonWidget.readKey(markRow);
+      return markHolder.keyReader.read(markKey);
+    }
+
+    /**
+     * Rewind to the mark row: the first one with the current key.
+     *
+     * @throws IllegalStateException if there is no marked row
+     */
+    public void rewindToMark()
+    {
+      if (markFrame < 0) {
+        throw new ISE("No mark");
+      }
+
+      currentFrame = markFrame;
+      holders.get(currentFrame).cursor.setCurrentRow(markRow);
+    }
+
+    /**
+     * Set the mark row to the current row. Used when data from the old mark to the current row is no longer needed.
+     */
+    public void markCurrent()
+    {
+      if (isAtEndOfPushedData()) {
+        clear();
+      } else {
+        // Remove unnecessary holders.
+        while (currentFrame > 0) {
+          if (currentFrame == holders.size() - 1) {
+            final FrameHolder lastHolder = holders.get(currentFrame);
+            holders.clear();
+            holders.add(lastHolder);
+            currentFrame = 0;
+          } else {
+            holders.remove(0);
+            currentFrame--;
+          }
+        }
+
+        markFrame = 0;
+        markRow = holders.get(currentFrame).cursor.getCurrentRow();
+      }
+    }
+
+    /**
+     * Whether the current cursor is past the end of the last frame for which we have data.
+     */
+    public boolean isAtEndOfPushedData()
+    {
+      return currentFrame < 0 || (currentFrame == holders.size() - 1 && holders.get(currentFrame).cursor.isDone());
+    }
+
+    /**
+     * Whether the current cursor is past the end of all data that will ever be pushed.
+     */
+    public boolean isAtEnd()
+    {
+      return done && isAtEndOfPushedData();
+    }
+
+    /**
+     * Whether this tracker needs more data in order to read the current cursor location or move it forward.
+     */
+    public boolean needsMoreData()
+    {
+      return !done && isAtEndOfPushedData();
+    }
+
+    /**
+     * Whether this tracker contains all rows for the marked key.
+     *
+     * @throws IllegalStateException if there is no marked key
+     */
+    public boolean hasCompleteSetForMark()
+    {
+      if (markFrame < 0) {
+        throw new ISE("No mark");
+      }
+
+      if (done) {
+        return true;
+      }
+
+      final FrameHolder lastHolder = holders.get(holders.size() - 1);
+      return !isSameKeyAsMark(lastHolder, lastHolder.frame.numRows() - 1);
+    }
+
+    /**
+     * Whether the current position (the current row of the {@link #currentFrame}) compares equally to the mark row.
+     * If {@link #isAtEnd()}, returns true iff there is no mark row.
+     */
+    public boolean isCurrentSameKeyAsMark()
+    {
+      if (isAtEnd()) {
+        return markFrame < 0;
+      } else {
+        assert !isAtEndOfPushedData();
+        final FrameHolder headHolder = holders.get(currentFrame);
+        return isSameKeyAsMark(headHolder, headHolder.cursor.getCurrentRow());
+      }
+    }
+
+    /**
+     * Clears the current mark and all buffered frames. Does not change {@link #done}.
+     */
+    public void clear()
+    {
+      holders.clear();
+      markFrame = -1;
+      markRow = -1;
+      currentFrame = -1;
+    }
+
+    /**
+     * Whether the provided frame and row compares equally to the mark row. The provided row must be at, or after,
+     * the mark row.
+     */
+    private boolean isSameKeyAsMark(final FrameHolder holder, final int row)
+    {
+      // Mark row must exist.
+      assert markFrame >= 0;
+
+      // Row must exist.
+      assert row >= 0 && row < holder.frame.numRows();
+
+      final FrameHolder markHolder = holders.get(markFrame);
+      final int cmp = markHolder.comparisonWidget.compare(markRow, holder.comparisonWidget, row);
+
+      assert cmp <= 0;

Review Comment:
   how important is this assert?  If it's truly important, you should perhaps have an if statement (that branch prediction *should* be able to do a really good job with)?



##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/querykit/common/SortMergeJoinFrameProcessorTest.java:
##########
@@ -0,0 +1,987 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.msq.querykit.common;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.MoreExecutors;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.common.guava.FutureUtils;
+import org.apache.druid.frame.Frame;
+import org.apache.druid.frame.FrameType;
+import org.apache.druid.frame.allocation.ArenaMemoryAllocator;
+import org.apache.druid.frame.allocation.SingleMemoryAllocatorFactory;
+import org.apache.druid.frame.channel.BlockingQueueFrameChannel;
+import org.apache.druid.frame.channel.ReadableFrameChannel;
+import org.apache.druid.frame.channel.ReadableNilFrameChannel;
+import org.apache.druid.frame.key.KeyColumn;
+import org.apache.druid.frame.key.KeyOrder;
+import org.apache.druid.frame.processor.FrameProcessorExecutor;
+import org.apache.druid.frame.read.FrameReader;
+import org.apache.druid.frame.segment.FrameStorageAdapter;
+import org.apache.druid.frame.testutil.FrameSequenceBuilder;
+import org.apache.druid.frame.testutil.FrameTestUtil;
+import org.apache.druid.frame.write.FrameWriterFactory;
+import org.apache.druid.frame.write.FrameWriters;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.guava.Sequence;
+import org.apache.druid.java.util.common.guava.Sequences;
+import org.apache.druid.msq.input.ReadableInput;
+import org.apache.druid.msq.kernel.StageId;
+import org.apache.druid.msq.kernel.StagePartition;
+import org.apache.druid.msq.test.LimitedFrameWriterFactory;
+import org.apache.druid.segment.RowBasedSegment;
+import org.apache.druid.segment.StorageAdapter;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.join.JoinTestHelper;
+import org.apache.druid.segment.join.JoinType;
+import org.apache.druid.testing.InitializedNullHandlingTest;
+import org.apache.druid.timeline.SegmentId;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+@RunWith(Parameterized.class)
+public class SortMergeJoinFrameProcessorTest extends InitializedNullHandlingTest
+{
+  private static final StagePartition STAGE_PARTITION = new StagePartition(new StageId("q", 0), 0);
+
+  private final int rowsPerInputFrame;
+  private final int rowsPerOutputFrame;
+
+  private FrameProcessorExecutor exec;
+
+  @Rule
+  public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+  public SortMergeJoinFrameProcessorTest(int rowsPerInputFrame, int rowsPerOutputFrame)
+  {
+    this.rowsPerInputFrame = rowsPerInputFrame;
+    this.rowsPerOutputFrame = rowsPerOutputFrame;
+  }
+
+  @Parameterized.Parameters(name = "rowsPerInputFrame = {0}, rowsPerOutputFrame = {1}")
+  public static Iterable<Object[]> constructorFeeder()
+  {
+    final List<Object[]> constructors = new ArrayList<>();
+
+    for (final int rowsPerInputFrame : new int[]{1, 2, 7, Integer.MAX_VALUE}) {
+      for (final int rowsPerOutputFrame : new int[]{1, 2, 7, Integer.MAX_VALUE}) {
+        constructors.add(new Object[]{rowsPerInputFrame, rowsPerOutputFrame});
+      }
+    }
+
+    return constructors;
+  }
+
+  @Before
+  public void setUp()
+  {
+    exec = new FrameProcessorExecutor(MoreExecutors.listeningDecorator(Execs.singleThreaded("test-exec")));
+  }
+
+  @After
+  public void tearDown() throws Exception
+  {
+    exec.getExecutorService().shutdownNow();
+    exec.getExecutorService().awaitTermination(10, TimeUnit.MINUTES);
+  }
+
+  @Test
+  public void testLeftJoinEmptyLeftSide() throws Exception
+  {
+    final ReadableInput factChannel = ReadableInput.channel(
+        ReadableNilFrameChannel.INSTANCE,
+        FrameReader.create(JoinTestHelper.FACT_SIGNATURE),
+        STAGE_PARTITION
+    );
+
+    final ReadableInput countriesChannel =
+        buildCountriesInput(ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)));
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryName", ColumnType.STRING)
+                    .add("j0.countryNumber", ColumnType.LONG)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        countriesChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.LEFT
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, Collections.emptyList());
+  }
+
+  @Test
+  public void testLeftJoinEmptyRightSide() throws Exception
+  {
+    final ReadableInput factChannel = buildFactInput(
+        ImmutableList.of(
+            new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+            new KeyColumn("page", KeyOrder.ASCENDING)
+        )
+    );
+
+
+    final ReadableInput countriesChannel = ReadableInput.channel(
+        ReadableNilFrameChannel.INSTANCE,
+        FrameReader.create(JoinTestHelper.COUNTRIES_SIGNATURE),
+        STAGE_PARTITION
+    );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryName", ColumnType.STRING)
+                    .add("j0.countryNumber", ColumnType.LONG)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        countriesChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.LEFT
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList("Agama mossambica", null, null, null, null),
+        Arrays.asList("Apamea abruzzorum", null, null, null, null),
+        Arrays.asList("Atractus flammigerus", null, null, null, null),
+        Arrays.asList("Rallicula", null, null, null, null),
+        Arrays.asList("Talk:Oswald Tilghman", null, null, null, null),
+        Arrays.asList("Peremptory norm", "AU", null, null, null),
+        Arrays.asList("Didier Leclair", "CA", null, null, null),
+        Arrays.asList("Les Argonautes", "CA", null, null, null),
+        Arrays.asList("Sarah Michelle Gellar", "CA", null, null, null),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "CL", null, null, null),
+        Arrays.asList("Diskussion:Sebastian Schulz", "DE", null, null, null),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "EC", null, null, null),
+        Arrays.asList("Saison 9 de Secret Story", "FR", null, null, null),
+        Arrays.asList("Glasgow", "GB", null, null, null),
+        Arrays.asList("Giusy Ferreri discography", "IT", null, null, null),
+        Arrays.asList("Roma-Bangkok", "IT", null, null, null),
+        Arrays.asList("青野武", "JP", null, null, null),
+        Arrays.asList("유희왕 GX", "KR", null, null, null),
+        Arrays.asList("History of Fourems", "MMMM", null, null, null),
+        Arrays.asList("Mathis Bolly", "MX", null, null, null),
+        Arrays.asList("Orange Soda", "MatchNothing", null, null, null),
+        Arrays.asList("Алиса в Зазеркалье", "NO", null, null, null),
+        Arrays.asList("Cream Soda", "SU", null, null, null),
+        Arrays.asList("Wendigo", "SV", null, null, null),
+        Arrays.asList("Carlo Curti", "US", null, null, null),
+        Arrays.asList("DirecTV", "US", null, null, null),
+        Arrays.asList("Old Anatolian Turkish", "US", null, null, null),
+        Arrays.asList("Otjiwarongo Airport", "US", null, null, null),
+        Arrays.asList("President of India", "US", null, null, null)
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testInnerJoinEmptyRightSide() throws Exception
+  {
+    final ReadableInput factChannel = buildFactInput(
+        ImmutableList.of(
+            new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+            new KeyColumn("page", KeyOrder.ASCENDING)
+        )
+    );
+
+    final ReadableInput countriesChannel = ReadableInput.channel(
+        ReadableNilFrameChannel.INSTANCE,
+        FrameReader.create(JoinTestHelper.COUNTRIES_SIGNATURE),
+        STAGE_PARTITION
+    );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryName", ColumnType.STRING)
+                    .add("j0.countryNumber", ColumnType.LONG)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        countriesChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.INNER
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, Collections.emptyList());
+  }
+
+  @Test
+  public void testLeftJoinCountryIsoCode() throws Exception
+  {
+    final ReadableInput factChannel = buildFactInput(
+        ImmutableList.of(
+            new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+            new KeyColumn("page", KeyOrder.ASCENDING)
+        )
+    );
+
+    final ReadableInput countriesChannel =
+        buildCountriesInput(ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)));
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryName", ColumnType.STRING)
+                    .add("j0.countryNumber", ColumnType.LONG)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        countriesChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.LEFT
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList("Agama mossambica", null, null, null, null),
+        Arrays.asList("Apamea abruzzorum", null, null, null, null),
+        Arrays.asList("Atractus flammigerus", null, null, null, null),
+        Arrays.asList("Rallicula", null, null, null, null),
+        Arrays.asList("Talk:Oswald Tilghman", null, null, null, null),
+        Arrays.asList("Peremptory norm", "AU", "AU", "Australia", 0L),
+        Arrays.asList("Didier Leclair", "CA", "CA", "Canada", 1L),
+        Arrays.asList("Les Argonautes", "CA", "CA", "Canada", 1L),
+        Arrays.asList("Sarah Michelle Gellar", "CA", "CA", "Canada", 1L),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "CL", "CL", "Chile", 2L),
+        Arrays.asList("Diskussion:Sebastian Schulz", "DE", "DE", "Germany", 3L),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "EC", "EC", "Ecuador", 4L),
+        Arrays.asList("Saison 9 de Secret Story", "FR", "FR", "France", 5L),
+        Arrays.asList("Glasgow", "GB", "GB", "United Kingdom", 6L),
+        Arrays.asList("Giusy Ferreri discography", "IT", "IT", "Italy", 7L),
+        Arrays.asList("Roma-Bangkok", "IT", "IT", "Italy", 7L),
+        Arrays.asList("青野武", "JP", "JP", "Japan", 8L),
+        Arrays.asList("유희왕 GX", "KR", "KR", "Republic of Korea", 9L),
+        Arrays.asList("History of Fourems", "MMMM", "MMMM", "Fourems", 205L),
+        Arrays.asList("Mathis Bolly", "MX", "MX", "Mexico", 10L),
+        Arrays.asList("Orange Soda", "MatchNothing", null, null, null),
+        Arrays.asList("Алиса в Зазеркалье", "NO", "NO", "Norway", 11L),
+        Arrays.asList("Cream Soda", "SU", "SU", "States United", 15L),
+        Arrays.asList("Wendigo", "SV", "SV", "El Salvador", 12L),
+        Arrays.asList("Carlo Curti", "US", "US", "United States", 13L),
+        Arrays.asList("DirecTV", "US", "US", "United States", 13L),
+        Arrays.asList("Old Anatolian Turkish", "US", "US", "United States", 13L),
+        Arrays.asList("Otjiwarongo Airport", "US", "US", "United States", 13L),
+        Arrays.asList("President of India", "US", "US", "United States", 13L)
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testCrossJoin() throws Exception
+  {
+    final ReadableInput factChannel = buildFactInput(
+        ImmutableList.of(
+            new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+            new KeyColumn("page", KeyOrder.ASCENDING)
+        )
+    );
+
+    final ReadableInput countriesChannel = makeChannelFromResourceWithLimit(
+        JoinTestHelper.COUNTRIES_RESOURCE,
+        JoinTestHelper.COUNTRIES_SIGNATURE,
+        ImmutableList.of(new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)),
+        2
+    );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("j0.page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        countriesChannel,
+        factChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(Collections.emptyList(), Collections.emptyList()),
+        JoinType.INNER
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList("Agama mossambica", "AU"),
+        Arrays.asList("Agama mossambica", "CA"),
+        Arrays.asList("Apamea abruzzorum", "AU"),
+        Arrays.asList("Apamea abruzzorum", "CA"),
+        Arrays.asList("Atractus flammigerus", "AU"),
+        Arrays.asList("Atractus flammigerus", "CA"),
+        Arrays.asList("Rallicula", "AU"),
+        Arrays.asList("Rallicula", "CA"),
+        Arrays.asList("Talk:Oswald Tilghman", "AU"),
+        Arrays.asList("Talk:Oswald Tilghman", "CA"),
+        Arrays.asList("Peremptory norm", "AU"),
+        Arrays.asList("Peremptory norm", "CA"),
+        Arrays.asList("Didier Leclair", "AU"),
+        Arrays.asList("Didier Leclair", "CA"),
+        Arrays.asList("Les Argonautes", "AU"),
+        Arrays.asList("Les Argonautes", "CA"),
+        Arrays.asList("Sarah Michelle Gellar", "AU"),
+        Arrays.asList("Sarah Michelle Gellar", "CA"),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "AU"),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "CA"),
+        Arrays.asList("Diskussion:Sebastian Schulz", "AU"),
+        Arrays.asList("Diskussion:Sebastian Schulz", "CA"),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "AU"),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "CA"),
+        Arrays.asList("Saison 9 de Secret Story", "AU"),
+        Arrays.asList("Saison 9 de Secret Story", "CA"),
+        Arrays.asList("Glasgow", "AU"),
+        Arrays.asList("Glasgow", "CA"),
+        Arrays.asList("Giusy Ferreri discography", "AU"),
+        Arrays.asList("Giusy Ferreri discography", "CA"),
+        Arrays.asList("Roma-Bangkok", "AU"),
+        Arrays.asList("Roma-Bangkok", "CA"),
+        Arrays.asList("青野武", "AU"),
+        Arrays.asList("青野武", "CA"),
+        Arrays.asList("유희왕 GX", "AU"),
+        Arrays.asList("유희왕 GX", "CA"),
+        Arrays.asList("History of Fourems", "AU"),
+        Arrays.asList("History of Fourems", "CA"),
+        Arrays.asList("Mathis Bolly", "AU"),
+        Arrays.asList("Mathis Bolly", "CA"),
+        Arrays.asList("Orange Soda", "AU"),
+        Arrays.asList("Orange Soda", "CA"),
+        Arrays.asList("Алиса в Зазеркалье", "AU"),
+        Arrays.asList("Алиса в Зазеркалье", "CA"),
+        Arrays.asList("Cream Soda", "AU"),
+        Arrays.asList("Cream Soda", "CA"),
+        Arrays.asList("Wendigo", "AU"),
+        Arrays.asList("Wendigo", "CA"),
+        Arrays.asList("Carlo Curti", "AU"),
+        Arrays.asList("Carlo Curti", "CA"),
+        Arrays.asList("DirecTV", "AU"),
+        Arrays.asList("DirecTV", "CA"),
+        Arrays.asList("Old Anatolian Turkish", "AU"),
+        Arrays.asList("Old Anatolian Turkish", "CA"),
+        Arrays.asList("Otjiwarongo Airport", "AU"),
+        Arrays.asList("Otjiwarongo Airport", "CA"),
+        Arrays.asList("President of India", "AU"),
+        Arrays.asList("President of India", "CA")
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testLeftJoinRegions() throws Exception
+  {
+    final ReadableInput factChannel =
+        buildFactInput(
+            ImmutableList.of(
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("page", KeyOrder.ASCENDING)
+            )
+        );
+
+    final ReadableInput regionsChannel =
+        buildRegionsInput(
+            ImmutableList.of(
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING)
+            )
+        );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("j0.regionName", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        regionsChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING)
+            ),
+            ImmutableList.of(
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING)
+            )
+        ),
+        JoinType.LEFT
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList("Agama mossambica", null, null),
+        Arrays.asList("Apamea abruzzorum", null, null),
+        Arrays.asList("Atractus flammigerus", null, null),
+        Arrays.asList("Rallicula", null, null),
+        Arrays.asList("Talk:Oswald Tilghman", null, null),
+        Arrays.asList("Peremptory norm", "New South Wales", "AU"),
+        Arrays.asList("Didier Leclair", "Ontario", "CA"),
+        Arrays.asList("Sarah Michelle Gellar", "Ontario", "CA"),
+        Arrays.asList("Les Argonautes", "Quebec", "CA"),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "Santiago Metropolitan", "CL"),
+        Arrays.asList("Diskussion:Sebastian Schulz", "Hesse", "DE"),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "Provincia del Guayas", "EC"),
+        Arrays.asList("Saison 9 de Secret Story", "Val d'Oise", "FR"),
+        Arrays.asList("Glasgow", "Kingston upon Hull", "GB"),
+        Arrays.asList("Giusy Ferreri discography", "Provincia di Varese", "IT"),
+        Arrays.asList("Roma-Bangkok", "Provincia di Varese", "IT"),
+        Arrays.asList("青野武", "Tōkyō", "JP"),
+        Arrays.asList("유희왕 GX", "Seoul", "KR"),
+        Arrays.asList("History of Fourems", "Fourems Province", "MMMM"),
+        Arrays.asList("Mathis Bolly", "Mexico City", "MX"),
+        Arrays.asList("Orange Soda", null, "MatchNothing"),
+        Arrays.asList("Алиса в Зазеркалье", "Finnmark Fylke", "NO"),
+        Arrays.asList("Cream Soda", "Ainigriv", "SU"),
+        Arrays.asList("Wendigo", "Departamento de San Salvador", "SV"),
+        Arrays.asList("Carlo Curti", "California", "US"),
+        Arrays.asList("Otjiwarongo Airport", "California", "US"),
+        Arrays.asList("President of India", "California", "US"),
+        Arrays.asList("DirecTV", "North Carolina", "US"),
+        Arrays.asList("Old Anatolian Turkish", "Virginia", "US")
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testRightJoinRegionCodeOnly() throws Exception
+  {
+    // This join generates duplicates.
+
+    final ReadableInput factChannel =
+        buildFactInput(
+            ImmutableList.of(
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("page", KeyOrder.ASCENDING)
+            )
+        );
+
+    final ReadableInput regionsChannel =
+        buildRegionsInput(
+            ImmutableList.of(
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)
+            )
+        );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("j0.page", ColumnType.STRING)
+                    .add("regionName", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        regionsChannel,
+        factChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("regionIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("regionIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.RIGHT
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList("Agama mossambica", null, null),
+        Arrays.asList("Apamea abruzzorum", null, null),
+        Arrays.asList("Atractus flammigerus", null, null),
+        Arrays.asList("Rallicula", null, null),
+        Arrays.asList("Talk:Oswald Tilghman", null, null),
+        Arrays.asList("유희왕 GX", "Seoul", "KR"),
+        Arrays.asList("青野武", "Tōkyō", "JP"),
+        Arrays.asList("Алиса в Зазеркалье", "Finnmark Fylke", "NO"),
+        Arrays.asList("Saison 9 de Secret Story", "Val d'Oise", "FR"),
+        Arrays.asList("Cream Soda", "Ainigriv", "SU"),
+        Arrays.asList("Carlo Curti", "California", "US"),
+        Arrays.asList("Otjiwarongo Airport", "California", "US"),
+        Arrays.asList("President of India", "California", "US"),
+        Arrays.asList("Mathis Bolly", "Mexico City", "MX"),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "Provincia del Guayas", "EC"),
+        Arrays.asList("Diskussion:Sebastian Schulz", "Hesse", "DE"),
+        Arrays.asList("Glasgow", "Kingston upon Hull", "GB"),
+        Arrays.asList("History of Fourems", "Fourems Province", "MMMM"),
+        Arrays.asList("Orange Soda", null, "MatchNothing"),
+        Arrays.asList("DirecTV", "North Carolina", "US"),
+        Arrays.asList("Peremptory norm", "New South Wales", "AU"),
+        Arrays.asList("Didier Leclair", "Ontario", "CA"),
+        Arrays.asList("Sarah Michelle Gellar", "Ontario", "CA"),
+        Arrays.asList("Les Argonautes", "Quebec", "CA"),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "Santiago Metropolitan", "CL"),
+        Arrays.asList("Wendigo", "Departamento de San Salvador", "SV"),
+        Arrays.asList("Giusy Ferreri discography", "Provincia di Varese", "IT"),
+        Arrays.asList("Giusy Ferreri discography", "Virginia", "IT"),
+        Arrays.asList("Old Anatolian Turkish", "Provincia di Varese", "US"),
+        Arrays.asList("Old Anatolian Turkish", "Virginia", "US"),
+        Arrays.asList("Roma-Bangkok", "Provincia di Varese", "IT"),
+        Arrays.asList("Roma-Bangkok", "Virginia", "IT")
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testFullOuterJoinRegionCodeOnly() throws Exception
+  {
+    // This join generates duplicates.
+
+    final ReadableInput factChannel =
+        buildFactInput(
+            ImmutableList.of(
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("page", KeyOrder.ASCENDING)
+            )
+        );
+
+    final ReadableInput regionsChannel =
+        buildRegionsInput(
+            ImmutableList.of(
+                new KeyColumn("regionIsoCode", KeyOrder.ASCENDING),
+                new KeyColumn("countryIsoCode", KeyOrder.ASCENDING)
+            )
+        );
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("j0.page", ColumnType.STRING)
+                    .add("regionName", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        regionsChannel,
+        factChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("regionIsoCode", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("regionIsoCode", KeyOrder.ASCENDING))
+        ),
+        JoinType.FULL
+    );
+
+    final List<List<Object>> expectedRows = Arrays.asList(
+        Arrays.asList(null, "Nulland", null),
+        Arrays.asList("Agama mossambica", null, null),
+        Arrays.asList("Apamea abruzzorum", null, null),
+        Arrays.asList("Atractus flammigerus", null, null),
+        Arrays.asList("Rallicula", null, null),
+        Arrays.asList("Talk:Oswald Tilghman", null, null),
+        Arrays.asList("유희왕 GX", "Seoul", "KR"),
+        Arrays.asList("青野武", "Tōkyō", "JP"),
+        Arrays.asList("Алиса в Зазеркалье", "Finnmark Fylke", "NO"),
+        Arrays.asList("Saison 9 de Secret Story", "Val d'Oise", "FR"),
+        Arrays.asList(null, "Foureis Province", null),
+        Arrays.asList("Cream Soda", "Ainigriv", "SU"),
+        Arrays.asList("Carlo Curti", "California", "US"),
+        Arrays.asList("Otjiwarongo Airport", "California", "US"),
+        Arrays.asList("President of India", "California", "US"),
+        Arrays.asList("Mathis Bolly", "Mexico City", "MX"),
+        Arrays.asList("Gabinete Ministerial de Rafael Correa", "Provincia del Guayas", "EC"),
+        Arrays.asList("Diskussion:Sebastian Schulz", "Hesse", "DE"),
+        Arrays.asList("Glasgow", "Kingston upon Hull", "GB"),
+        Arrays.asList("History of Fourems", "Fourems Province", "MMMM"),
+        Arrays.asList("Orange Soda", null, "MatchNothing"),
+        Arrays.asList("DirecTV", "North Carolina", "US"),
+        Arrays.asList("Peremptory norm", "New South Wales", "AU"),
+        Arrays.asList("Didier Leclair", "Ontario", "CA"),
+        Arrays.asList("Sarah Michelle Gellar", "Ontario", "CA"),
+        Arrays.asList("Les Argonautes", "Quebec", "CA"),
+        Arrays.asList("Golpe de Estado en Chile de 1973", "Santiago Metropolitan", "CL"),
+        Arrays.asList("Wendigo", "Departamento de San Salvador", "SV"),
+        Arrays.asList("Giusy Ferreri discography", "Provincia di Varese", "IT"),
+        Arrays.asList("Giusy Ferreri discography", "Virginia", "IT"),
+        Arrays.asList("Old Anatolian Turkish", "Provincia di Varese", "US"),
+        Arrays.asList("Old Anatolian Turkish", "Virginia", "US"),
+        Arrays.asList("Roma-Bangkok", "Provincia di Varese", "IT"),
+        Arrays.asList("Roma-Bangkok", "Virginia", "IT"),
+        Arrays.asList(null, "Usca City", null)
+    );
+
+    assertResult(processor, outputChannel.readable(), joinSignature, expectedRows);
+  }
+
+  @Test
+  public void testLeftJoinCountryNumber() throws Exception
+  {
+    final ReadableInput factChannel = buildFactInput(
+        ImmutableList.of(
+            new KeyColumn("countryNumber", KeyOrder.ASCENDING),
+            new KeyColumn("page", KeyOrder.ASCENDING)
+        )
+    );
+
+    final ReadableInput countriesChannel =
+        buildCountriesInput(ImmutableList.of(new KeyColumn("countryNumber", KeyOrder.ASCENDING)));
+
+    final BlockingQueueFrameChannel outputChannel = BlockingQueueFrameChannel.minimal();
+
+    final RowSignature joinSignature =
+        RowSignature.builder()
+                    .add("page", ColumnType.STRING)
+                    .add("countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryIsoCode", ColumnType.STRING)
+                    .add("j0.countryName", ColumnType.STRING)
+                    .add("j0.countryNumber", ColumnType.LONG)
+                    .build();
+
+    final SortMergeJoinFrameProcessor processor = new SortMergeJoinFrameProcessor(
+        factChannel,
+        countriesChannel,
+        outputChannel.writable(),
+        makeFrameWriterFactory(joinSignature),
+        "j0.",
+        ImmutableList.of(
+            ImmutableList.of(new KeyColumn("countryNumber", KeyOrder.ASCENDING)),
+            ImmutableList.of(new KeyColumn("countryNumber", KeyOrder.ASCENDING))
+        ),
+        JoinType.LEFT
+    );
+
+    final String countryCodeForNull;
+    final String countryNameForNull;
+    final Long countryNumberForNull;
+
+    if (NullHandling.sqlCompatible()) {
+      countryCodeForNull = null;
+      countryNameForNull = null;
+      countryNumberForNull = null;
+    } else {
+      // In default-value mode, null country number from the left-hand table converts to zero, which matches Australia.
+      countryCodeForNull = "AU";
+      countryNameForNull = "Australia";
+      countryNumberForNull = 0L;

Review Comment:
   I'm like 80% sure this assumption would be incorrect if the long column actually came from the nested column instead of the "normal" top-level long column.  That is, it's perhaps less a function of the "default-value mode" and more a function of the column implementation that happens to be getting used.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1041845419


##########
docs/multi-stage-query/reference.md:
##########
@@ -198,13 +198,99 @@ The following table lists the context parameters for the MSQ task engine:
 | `maxNumTasks` | SELECT, INSERT, REPLACE<br /><br />The maximum total number of tasks to launch, including the controller task. The lowest possible value for this setting is 2: one controller and one worker. All tasks must be able to launch simultaneously. If they cannot, the query returns a `TaskStartTimeout` error code after approximately 10 minutes.<br /><br />May also be provided as `numTasks`. If both are present, `maxNumTasks` takes priority.| 2 |
 | `taskAssignment` | SELECT, INSERT, REPLACE<br /><br />Determines how many tasks to use. Possible values include: <ul><li>`max`: Uses as many tasks as possible, up to `maxNumTasks`.</li><li>`auto`: When file sizes can be determined through directory listing (for example: local files, S3, GCS, HDFS) uses as few tasks as possible without exceeding 10 GiB or 10,000 files per task, unless exceeding these limits is necessary to stay within `maxNumTasks`. When file sizes cannot be determined through directory listing (for example: http), behaves the same as `max`.</li></ul> | `max` |
 | `finalizeAggregations` | SELECT, INSERT, REPLACE<br /><br />Determines the type of aggregation to return. If true, Druid finalizes the results of complex aggregations that directly appear in query results. If false, Druid returns the aggregation's intermediate type rather than finalized type. This parameter is useful during ingestion, where it enables storing sketches directly in Druid tables. For more information about aggregations, see [SQL aggregation functions](../querying/sql-aggregations.md). | true |
+| `sqlJoinAlgorithm` | SELECT, INSERT, REPLACE<br /><br />Algorithm to use for JOIN. Use `broadcast` (the default) for broadcast hash join or `sortMerge` for sort-merge join. Affects all JOIN operations in the query. See [Joins](#joins) for more details. | `broadcast` |

Review Comment:
   This sort of thing would normally be delivered via a hint in the SQL, perhaps it's not too hard to deliver it that way?



##########
processing/src/main/java/org/apache/druid/frame/key/KeyOrder.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.frame.key;
+
+public enum KeyOrder
+{
+  NONE(false),

Review Comment:
   instead of `NONE` means "hashed", would it make sense to just call it `HASHED`?



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java:
##########
@@ -595,8 +596,8 @@ public void updatePartialKeyStatisticsInformation(int stageNumber, int workerNum
           final StageDefinition stageDef = queryKernel.getStageDefinition(stageId);
           final ObjectMapper mapper = MSQTasks.decorateObjectMapperForKeyCollectorSnapshot(
               context.jsonMapper(),
-              stageDef.getShuffleSpec().get().getClusterBy(),
-              stageDef.getShuffleSpec().get().doesAggregateByClusterKey()
+              stageDef.getShuffleSpec().clusterBy(),
+              stageDef.getShuffleSpec().doesAggregate()

Review Comment:
   You hate the word "get"/"is"?  "getClusterBy()" "isDoesAggregate()"?



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerContext.java:
##########
@@ -73,4 +74,9 @@
   DruidNode selfNode();
 
   Bouncer processorBouncer();
+
+  default File tempDir(int stageNumber, String id)

Review Comment:
   Nit brigade!  I think we tend to use `tmp` in other parts of the code?



##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java:
##########
@@ -1292,6 +853,803 @@ public ReadableFrameChannel openChannel(StageId stageId, int workerNumber, int p
     }
   }
 
+  /**
+   * Main worker logic for executing a {@link WorkOrder}.
+   */
+  private class RunWorkOrder
+  {
+    private final WorkerStageKernel kernel;
+    private final InputChannelFactory inputChannelFactory;
+    private final CounterTracker counterTracker;
+    private final FrameProcessorExecutor exec;
+    private final String cancellationId;
+    private final int parallelism;
+    private final FrameContext frameContext;
+    private final MSQWarningReportPublisher warningPublisher;
+
+    private InputSliceReader inputSliceReader;
+    private OutputChannelFactory workOutputChannelFactory;
+    private OutputChannelFactory shuffleOutputChannelFactory;
+    private ResultAndChannels<?> workResultAndOutputChannels;
+    private SettableFuture<ClusterByPartitions> stagePartitionBoundariesFuture;
+    private ListenableFuture<OutputChannels> shuffleOutputChannelsFuture;
+
+    public RunWorkOrder(
+        final WorkerStageKernel kernel,
+        final InputChannelFactory inputChannelFactory,
+        final CounterTracker counterTracker,
+        final FrameProcessorExecutor exec,
+        final String cancellationId,
+        final int parallelism,
+        final FrameContext frameContext,
+        final MSQWarningReportPublisher warningPublisher
+    )
+    {
+      this.kernel = kernel;
+      this.inputChannelFactory = inputChannelFactory;
+      this.counterTracker = counterTracker;
+      this.exec = exec;
+      this.cancellationId = cancellationId;
+      this.parallelism = parallelism;
+      this.frameContext = frameContext;
+      this.warningPublisher = warningPublisher;
+    }
+
+    private void start() throws IOException
+    {
+      final WorkOrder workOrder = kernel.getWorkOrder();
+      final StageDefinition stageDef = workOrder.getStageDefinition();
+
+      makeInputSliceReader();
+      makeWorkOutputChannelFactory();
+      makeShuffleOutputChannelFactory();
+      makeAndRunWorkProcessors();
+
+      if (stageDef.doesShuffle()) {
+        makeAndRunShuffleProcessors();
+      } else {
+        // No shuffling: work output _is_ shuffle output. Retain read-only versions to reduce memory footprint.
+        shuffleOutputChannelsFuture =
+            Futures.immediateFuture(workResultAndOutputChannels.getOutputChannels().readOnly());
+      }
+
+      setUpCompletionCallbacks();
+    }
+
+    /**
+     * Settable {@link ClusterByPartitions} future for global sort. Necessary because we don't know ahead of time
+     * what the boundaries will be. The controller decides based on statistics from all workers. Once the controller
+     * decides, its decision is written to this future, which allows sorting on workers to proceed.
+     */
+    @Nullable
+    public SettableFuture<ClusterByPartitions> getStagePartitionBoundariesFuture()
+    {
+      return stagePartitionBoundariesFuture;
+    }
+
+    private void makeInputSliceReader()
+    {
+      if (inputSliceReader != null) {
+        throw new ISE("inputSliceReader already created");
+      }
+
+      final WorkOrder workOrder = kernel.getWorkOrder();
+      final String queryId = workOrder.getQueryDefinition().getQueryId();
+
+      final InputChannels inputChannels =
+          new InputChannelsImpl(
+              workOrder.getQueryDefinition(),
+              InputSlices.allReadablePartitions(workOrder.getInputs()),
+              inputChannelFactory,
+              () -> ArenaMemoryAllocator.createOnHeap(frameContext.memoryParameters().getStandardFrameSize()),
+              exec,
+              cancellationId
+          );
+
+      inputSliceReader = new MapInputSliceReader(

Review Comment:
   Is this Map guaranteed to be used to read from all of the different types of slices?  Is there a reason that it needs to be re-instantiated instead of just built once and reused?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #13506:
URL: https://github.com/apache/druid/pull/13506#discussion_r1042621537


##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java:
##########
@@ -1292,6 +853,803 @@ public ReadableFrameChannel openChannel(StageId stageId, int workerNumber, int p
     }
   }
 
+  /**
+   * Main worker logic for executing a {@link WorkOrder}.
+   */
+  private class RunWorkOrder
+  {
+    private final WorkerStageKernel kernel;
+    private final InputChannelFactory inputChannelFactory;
+    private final CounterTracker counterTracker;
+    private final FrameProcessorExecutor exec;
+    private final String cancellationId;
+    private final int parallelism;
+    private final FrameContext frameContext;
+    private final MSQWarningReportPublisher warningPublisher;
+
+    private InputSliceReader inputSliceReader;
+    private OutputChannelFactory workOutputChannelFactory;
+    private OutputChannelFactory shuffleOutputChannelFactory;
+    private ResultAndChannels<?> workResultAndOutputChannels;
+    private SettableFuture<ClusterByPartitions> stagePartitionBoundariesFuture;
+    private ListenableFuture<OutputChannels> shuffleOutputChannelsFuture;
+
+    public RunWorkOrder(
+        final WorkerStageKernel kernel,
+        final InputChannelFactory inputChannelFactory,
+        final CounterTracker counterTracker,
+        final FrameProcessorExecutor exec,
+        final String cancellationId,
+        final int parallelism,
+        final FrameContext frameContext,
+        final MSQWarningReportPublisher warningPublisher
+    )
+    {
+      this.kernel = kernel;
+      this.inputChannelFactory = inputChannelFactory;
+      this.counterTracker = counterTracker;
+      this.exec = exec;
+      this.cancellationId = cancellationId;
+      this.parallelism = parallelism;
+      this.frameContext = frameContext;
+      this.warningPublisher = warningPublisher;
+    }
+
+    private void start() throws IOException
+    {
+      final WorkOrder workOrder = kernel.getWorkOrder();
+      final StageDefinition stageDef = workOrder.getStageDefinition();
+
+      makeInputSliceReader();
+      makeWorkOutputChannelFactory();
+      makeShuffleOutputChannelFactory();
+      makeAndRunWorkProcessors();
+
+      if (stageDef.doesShuffle()) {
+        makeAndRunShuffleProcessors();
+      } else {
+        // No shuffling: work output _is_ shuffle output. Retain read-only versions to reduce memory footprint.
+        shuffleOutputChannelsFuture =
+            Futures.immediateFuture(workResultAndOutputChannels.getOutputChannels().readOnly());
+      }
+
+      setUpCompletionCallbacks();
+    }
+
+    /**
+     * Settable {@link ClusterByPartitions} future for global sort. Necessary because we don't know ahead of time
+     * what the boundaries will be. The controller decides based on statistics from all workers. Once the controller
+     * decides, its decision is written to this future, which allows sorting on workers to proceed.
+     */
+    @Nullable
+    public SettableFuture<ClusterByPartitions> getStagePartitionBoundariesFuture()
+    {
+      return stagePartitionBoundariesFuture;
+    }
+
+    private void makeInputSliceReader()
+    {
+      if (inputSliceReader != null) {
+        throw new ISE("inputSliceReader already created");
+      }
+
+      final WorkOrder workOrder = kernel.getWorkOrder();
+      final String queryId = workOrder.getQueryDefinition().getQueryId();
+
+      final InputChannels inputChannels =
+          new InputChannelsImpl(
+              workOrder.getQueryDefinition(),
+              InputSlices.allReadablePartitions(workOrder.getInputs()),
+              inputChannelFactory,
+              () -> ArenaMemoryAllocator.createOnHeap(frameContext.memoryParameters().getStandardFrameSize()),
+              exec,
+              cancellationId
+          );
+
+      inputSliceReader = new MapInputSliceReader(

Review Comment:
   It's built once per worker. It does have some special pieces that are specific to this worker, so it isn't shared across workers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #13506: Sort-merge join and hash shuffles for MSQ.

Posted by GitBox <gi...@apache.org>.
gianm commented on PR #13506:
URL: https://github.com/apache/druid/pull/13506#issuecomment-1341517191

   > Only just a few nit-level comments so far. I haven't gotten too far, but wanted to leave them as I will need to wander off and come back to this. The last thing I did was read through WorkerImpl, as a newbie to this side of the code, I didn't know if I should expect it to just be a whole bunch of wiring up of objects or if there should be actual logic to it. From what I saw, it seemed like it's all doing just wiring up of things. If that's expected, great. If there was supposed to be business logic inside of it that is essential for the sortMerge algorithm, it got lost on me in the overall scope of what WorkerImpl is doing.
   
   Thanks for taking a look!
   
   WorkerImpl is indeed meant to just write things up and them let them rip.
   
   The join algorithm is entirely contained within SortMergeJoinFrameProcessor. The rest of the changes (the majority of the changes) are about adding hash-based shuffles to MSQ, which previously only included range-based shuffles.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org