[jira] [Created] (DRILL-7487) Retire unused OUT_OF_MEMORY iterator status

["Paul Rogers (Jira)" <ji...@apache.org>]

Paul Rogers created DRILL-7487:
----------------------------------

             Summary: Retire unused OUT_OF_MEMORY iterator status
                 Key: DRILL-7487
                 URL: https://issues.apache.org/jira/browse/DRILL-7487
             Project: Apache Drill
          Issue Type: Improvement
            Reporter: Paul Rogers
            Assignee: Paul Rogers


Drill has long supported the {{OUT_OF_MEMORY}} iterator status. The idea is that an operator can realize it has encountered memory pressure and ask its downstream operator to free up some memory. However, an inspection of the code shows that the status is actually sent in only one place ({{UnorderedReceiverBatch}}), and then only in response to the operator hitting its allocator limit (which no other batch can do anything about.)

If an operator did choose to try to use this status, there are two key problems:

1. The operator must be able to suspend itself at any point that it might need memory. For example, an operator that allocates a dozen vectors must be able to stop on, say, the 9th vector, then resume at that point on the subsequent call to `next()`. The complexity of the state machine needed to do this is very high.
2. The *downstream* operators (who may not yet have seen rows) are the least likely to be able to release memory. It is the *upstream* operators (such as spillable operators) that might be able to spill some of the rows they are holding.

Presto suggests a nice alternative:

* An operator which encounters memory pressure asks the fragment executor for more memory.
* The fragment executor asks all *other* operators in that fragment to release memory if possible.

This allows a very simple memory recovery strategy:

{noformat}
  try {
    // allocate something
  } catch (OutOfMemoryException e) {
    context.requestMemory(this);
    // allocate something again, throwing OOM if it fails again
  }
{noformat}

Proposed are two changes:

1. Retire the OUT_OF_MEMORY status. Simply remove all references to it since it is never sent.
2. Create a stub {{requestMemory()}} method in the operator context that does nothing now, but could be expanded to perform the work suggested above.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)