You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@drill.apache.org by abdelhakim deneche <ad...@gmail.com> on 2015/05/05 01:54:11 UTC

Review Request 33829: DRILL-2757: Verify operators correctly handle low memory conditions and cancellations

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33829/
-----------------------------------------------------------

Review request for drill, Hanifi Gunes, Jacques Nadeau, and Steven Phillips.


Bugs: DRILL-2757
    https://issues.apache.org/jira/browse/DRILL-2757


Repository: drill-git


Description
-------

includes:
[DRILL-2893](https://issues.apache.org/jira/browse/DRILL-2893): ScanBatch throws a NullPointerException instead of returning OUT_OF_MEMORY
[DRILL-2894](https://issues.apache.org/jira/browse/DRILL-2894): FixedValueVectors shouldn't set it's data buffer to null when it fails to allocate it
[DRILL-2895](https://issues.apache.org/jira/browse/DRILL-2895): AbstractRecordBatch.buildSchema() should properly handle OUT_OF_MEMORY outcome
[DRILL-2905](https://issues.apache.org/jira/browse/DRILL-2905): RootExec implementations should properly handle IterOutcome.OUT_OF_MEMORY
[DRILL-2920](https://issues.apache.org/jira/browse/DRILL-2920): properly handle OutOfMemoryException
[DRILL-2947](https://issues.apache.org/jira/browse/DRILL-2947): AllocationHelper.allocateNew() doesn't have a consistent behavior when it can't allocate

also:
- improved how system errors are displayed
- added UserException.memoryError() with a pre assigned error message
- injection site in ScanBatch and unit test that runs various tpch queries and injects
  an exception in the ScanBatch that will cause an OUT_OF_MEMORY outcome to be sent


Diffs
-----

  common/src/main/java/org/apache/drill/common/exceptions/ErrorHelper.java 4da4ee8 
  common/src/main/java/org/apache/drill/common/exceptions/UserException.java 9283339 
  common/src/test/java/org/apache/drill/common/exceptions/TestUserException.java a145f95 
  exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 6a924b7 
  exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java 8a4b663 
  exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 09a7568 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java 4700dbd 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScreenCreator.java 5b4d7bd 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java 67062f3 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/TopNBatch.java 9f6bea9 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/WriterRecordBatch.java 15fb7b5 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java b753574 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java 1b90dd8 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java c1c5cb9 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggTemplate.java 86f3100 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java d2282c8 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/FilterTemplate2.java 26f2657 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java dd53477 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatch.java 6466f70 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/NestedLoopJoinBatch.java d20bfa1 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/limit/LimitRecordBatch.java eff9e61 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java c36b0d3 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java cf7ba16 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/producer/ProducerConsumerBatch.java 35bf3cd 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java 7b9fffb 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/sort/SortBatch.java 74b7d85 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/RemovingRecordBatch.java aa9297e 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/trace/TraceRecordBatch.java af45815 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java d7ea3bb 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameRecordBatch.java 86d11d5 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java e88bc67 
  exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java 4e348bb 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java 7c77ca2 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java 2fc5bf3 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 4249cbe 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentExecutor.java ddb828c 
  exec/java-exec/src/test/java/org/apache/drill/TestOutOfMemoryOutcome.java PRE-CREATION 

Diff: https://reviews.apache.org/r/33829/diff/


Testing
-------

added new test class "TestOutOfMemory" that runs several tpch queries and injects an OutOfMemoryException in the ScanBatch. This causes an OUT_OF_MEMORY outcome to be propagated through the execution stack.

All unit tests are passing, still waiting for the results for functional/tpch100


Thanks,

abdelhakim deneche