[jira] [Created] (DRILL-6498) Support for EMIT outcome in ExternalSortBatch

["Sorabh Hamirwasia (JIRA)" <ji...@apache.org>]

Sorabh Hamirwasia created DRILL-6498:
----------------------------------------

             Summary: Support for EMIT outcome in ExternalSortBatch
                 Key: DRILL-6498
                 URL: https://issues.apache.org/jira/browse/DRILL-6498
             Project: Apache Drill
          Issue Type: Task
          Components: Execution - Relational Operators
            Reporter: Sorabh Hamirwasia
            Assignee: Sorabh Hamirwasia
             Fix For: 1.14.0


With Lateral and Unnest if Sort is present in the sub-query, then it needs to handle the EMIT outcome correctly. This means when a EMIT is received then perform the Sort operation on the records buffered so far and produce output with it. After EMIT Sort should refresh it's state and again work on next batches of incoming record unless an EMIT is seen again.

For first cut Sort will not support spilling in the subquery between Lateral and Unnest since spilling is very unlikely. The worst case that can happen is that Lateral will get a batch with only 1 row of data because of repeated type column data size being too big. In that case Unnest will produce 1 output batch only and Sort or other blocking operators anyways needs enough memory to at least hold 1 incoming batch. So in ideal cases spilling should not happen. But if there is a operator between Sort and Unnest which increases the data size then Sort might be in a situation to spill but thats not a common case for now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)