You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Chris Westin <ch...@gmail.com> on 2015/05/09 00:32:45 UTC
Review Request 34004: DRILL-1942: Improved memory allocator
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34004/
-----------------------------------------------------------
Review request for drill and Jacques Nadeau.
Bugs: DRILL-1942
https://issues.apache.org/jira/browse/DRILL-1942
Repository: drill-git
Description
-------
Rewritten direct memory allocator. Simplified interface, and use, along with a means to support additional allocation policies in the future. There are features in the allocator and in DrillBuf that make finding leaks easier, as well as better enforcement of limits. New features include transfer of buffers, and better slicing support.
This is a preliminary patch to get the review started because it touches a lot of files (readers and record batches were made AutoCloseable in order to cover cleanup). Subsequent reviews can use the differential view to just see additional changes. The new allocator is in BaseAllocator.java (along with derived classes RootAllocator and ChildAllocator); DrillBuf also has significant changes. Most other changes in other files are just to use newer interfaces, or to change cleanup() to close(), or to close subordinate objects that are newly (Auto)Closeable. 1There are still a couple of things to do:
* Some TODO(cwestin)s to clean up tracing and debugging code, as well as adding javadoc
* Using the AllocatorOwner interface to replace the reallocation mechanism for FragmentContext and OperatorContext so that the allocator doesn't know anything about those objects.
Diffs
-----
common/src/main/java/org/apache/drill/common/DrillAutoCloseables.java PRE-CREATION
common/src/main/java/org/apache/drill/common/DrillCloseables.java PRE-CREATION
common/src/main/java/org/apache/drill/common/config/DrillConfig.java 522303f
common/src/main/java/org/apache/drill/common/config/NestedConfig.java 3fd885f
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java 9458db2
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveRecordReader.java 5394ee3
contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoRecordReader.java 182f5a4
exec/java-exec/src/main/codegen/templates/AbstractFieldWriter.java 1b5dad1
exec/java-exec/src/main/codegen/templates/BaseWriter.java ada410d
exec/java-exec/src/main/codegen/templates/ComplexWriters.java 49c75d1
exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 1059bfb
exec/java-exec/src/main/codegen/templates/JsonOutputRecordWriter.java ea643f0
exec/java-exec/src/main/codegen/templates/ListWriters.java 6df4248
exec/java-exec/src/main/codegen/templates/MapWriters.java 6ee8035
exec/java-exec/src/main/codegen/templates/NullableValueVectors.java 9d03efb
exec/java-exec/src/main/codegen/templates/ParquetOutputRecordWriter.java 0d24041
exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java c0fba66
exec/java-exec/src/main/codegen/templates/StringOutputRecordWriter.java f704cca
exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java 7aa7415
exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java 7f80f7a
exec/java-exec/src/main/java/io/netty/buffer/FakeAllocator.java 721aff9
exec/java-exec/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java 2ca79f0
exec/java-exec/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java e332b13
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java be8c7a0
exec/java-exec/src/main/java/org/apache/drill/exec/cache/AbstractStreamSerializable.java ef488d6
exec/java-exec/src/main/java/org/apache/drill/exec/cache/LoopedAbstractDrillSerializable.java d2a7458
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java 8e2ce96
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java 5b28f16
exec/java-exec/src/main/java/org/apache/drill/exec/client/DumpCat.java 55d9cf3
exec/java-exec/src/main/java/org/apache/drill/exec/client/PrintingResultsListener.java f5a119d
exec/java-exec/src/main/java/org/apache/drill/exec/exception/FragmentSetupException.java c276846
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MappifyUtility.java e27234f
exec/java-exec/src/main/java/org/apache/drill/exec/memory/Accountor.java eb932ad
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationPolicy.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationPolicyAgent.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationReservation.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocatorOwner.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocatorsStatsMXBean.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java 057cfa6
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferAllocator.java c233ac5
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferLedger.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/ChainedAllocatorOwner.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/ChildAllocator.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/OutOfMemoryException.java 063f1c1
exec/java-exec/src/main/java/org/apache/drill/exec/memory/OutOfMemoryRuntimeException.java 305eabd
exec/java-exec/src/main/java/org/apache/drill/exec/memory/RootAllocator.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/RootAllocatorStatsMXBean.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/TopLevelAllocator.java a78deb6
exec/java-exec/src/main/java/org/apache/drill/exec/ops/BufferManager.java c953bb3
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java b108924
exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContext.java 7cc52ba
exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContextImpl.java 6dbd880
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 9e2f210
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java f56dae3
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScreenCreator.java 5b4d7bd
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java 67062f3
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java 369c0ec
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/TopNBatch.java 9f6bea9
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/WriterRecordBatch.java 15fb7b5
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java b753574
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java 1b90dd8
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java c1c5cb9
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java d2282c8
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/FilterRecordBatch.java 5eee9df
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java dd53477
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatch.java 6466f70
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatchBuilder.java 1187bd6
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/NestedLoopJoinBatch.java aa4b300
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/limit/LimitRecordBatch.java eff9e61
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java ce683cb
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/orderedpartitioner/OrderedPartitionRecordBatch.java 63b7eba
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java 16d1400
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/producer/ProducerConsumerBatch.java 35bf3cd
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/sort/SortBatch.java 74b7d85
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/sort/SortRecordBatchBuilder.java e559ece
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/RemovingRecordBatch.java aa9297e
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/trace/TraceRecordBatch.java af45815
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java 66bc3e3
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unorderedreceiver/UnorderedReceiverBatch.java 66a2092
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/validate/IteratorValidatorBatchIterator.java efd155e
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameRecordBatch.java 86d11d5
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java e88bc67
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/MSortTemplate.java 9b97e1c
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/PriorityQueueCopier.java 161ca6a
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/PriorityQueueCopierTemplate.java f7786b7
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java 4e348bb
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractSingleRecordBatch.java dd90cab
exec/java-exec/src/main/java/org/apache/drill/exec/record/RawFragmentBatch.java edd79ac
exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java 6f10a1c
exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchLoader.java 1b8b7ce
exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorAccessible.java 9db1681
exec/java-exec/src/main/java/org/apache/drill/exec/record/WritableBatch.java 308a8bc
exec/java-exec/src/main/java/org/apache/drill/exec/record/selection/SelectionVector2.java 7a7c012
exec/java-exec/src/main/java/org/apache/drill/exec/record/selection/SelectionVector4.java 69bc78f
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcDecoder.java 74a4afb
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataServer.java 6f8e20b
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryDataBatch.java 914bd00
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserClient.java a8bad78
exec/java-exec/src/main/java/org/apache/drill/exec/server/BootStrapContext.java d0a998e
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java e7a9a3c
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractRecordReader.java 6e27628
exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordReader.java 61ccac5
exec/java-exec/src/main/java/org/apache/drill/exec/store/avro/AvroRecordReader.java a52fd22
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONRecordReader.java 3d789eb
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JsonProcessor.java 4d8d4ba
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/reader/BaseJsonProcessor.java 7833631
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/compliant/CompliantTextRecordReader.java b2af32d
exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockRecordReader.java fd97c48
exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockScanBatchCreator.java 74423bf
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ParquetRecordReader.java 58cf321
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet2/DrillParquetReader.java 99ac19c
exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/PojoRecordReader.java cf98b83
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/MemoryIterator.java e02b413
exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java 0322f36
exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java 7c77ca2
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseDataValueVector.java 0c6097c
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java 22f0fe7
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java 2fc5bf3
exec/java-exec/src/main/java/org/apache/drill/exec/vector/ValueVector.java e4a0997
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractContainerVector.java d14dca6
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractMapVector.java 78846dc
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java 4138839
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java c061029
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java e5d48dd
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/DrillBufInputStream.java 1061a5c
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java 095d8c6
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/AbstractBaseWriter.java ec8c00b
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/ComplexWriterImpl.java a4a35e2
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/VectorContainerWriter.java 6b6ab46
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/writer/FieldWriter.java 3faa4f7
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/SpoolingRawBatchBuffer.java 2a79e42
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 49d0c94
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/ForemanException.java 32a99ad
exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java f8ec090
exec/java-exec/src/test/java/org/apache/drill/PlanTestBase.java f909681
exec/java-exec/src/test/java/org/apache/drill/PlanningBase.java da033f7
exec/java-exec/src/test/java/org/apache/drill/QueryTestUtil.java e218d6c
exec/java-exec/src/test/java/org/apache/drill/TestTpchPlanning.java 707ea78
exec/java-exec/src/test/java/org/apache/drill/exec/client/DumpCatTest.java 7c58b19
exec/java-exec/src/test/java/org/apache/drill/exec/compile/bytecode/ReplaceMethodInvoke.java bc2d929
exec/java-exec/src/test/java/org/apache/drill/exec/expr/ExpressionTest.java 239a099
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestByteComparisonFunctions.java 1e9a47c
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestCastFunctions.java 3ba8743
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestMathFunctions.java 4f06a9d
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestNewMathFunctions.java 880184e
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestRepeatedFunction.java 73c7508
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestAllocators.java 74ce225
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestBaseAllocator.java PRE-CREATION
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestEndianess.java 48ddada
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/SimpleRootExec.java 42d2193
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestCastFunctions.java ffa8765
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestComparisonFunctions.java c69c6f5
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestImplicitCastFunctions.java 03c6f41
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestOptiqPlans.java dc37071
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestSimpleFragmentRun.java 09ba1a5
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestSimpleFunctions.java d551319
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestStringFunctions.java d72c1e1
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestUnionExchange.java 9c24f79
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/agg/TestAgg.java b39566a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/common/TestHashTable.java b02249d
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/filter/TestSimpleFilter.java a069078
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoin.java 6c067fe
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestMergeJoin.java 18555c7
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/limit/TestSimpleLimit.java 7cdb41a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/mergereceiver/TestMergingReceiver.java 0122c08
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/project/TestSimpleProjection.java 43c430a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/sort/TestSimpleSort.java f37624a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/trace/TestTraceMultiRecordBatch.java b82846e
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/trace/TestTraceOutputDump.java 1cb72ff
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/union/TestSimpleUnion.java 07de27f
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestWriter.java f4d505d
exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestLoad.java f57e765
exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java 1564aea
exec/java-exec/src/test/java/org/apache/drill/exec/server/DrillClientFactory.java 4230518
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestDrillbitResilience.java da69e9e
exec/java-exec/src/test/java/org/apache/drill/exec/store/TestDirectCodecFactory.java 644144e
exec/java-exec/src/test/java/org/apache/drill/exec/store/ischema/TestInfoSchemaFilterPushDown.java b6e789b
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/FieldInfo.java 34f60ba
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 8fdaa72
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 6326478
exec/java-exec/src/test/java/org/apache/drill/exec/vector/TestSplitAndTransfer.java 4b3aa8a
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/TestEmptyPopulator.java 23cc316
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/fn/TestJsonReaderWithSparseFiles.java d674d47
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestComplexTypeReader.java 521a41d
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestComplexTypeWriter.java cb7bef2
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestJsonReader.java dfa89ca
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestRepeated.java 6e2a2b5
exec/java-exec/src/test/resources/logback.xml 2dcfed9
exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillConnectionImpl.java 74c6655
Diff: https://reviews.apache.org/r/34004/diff/
Testing
-------
Many unit tests run by hand. Currently trying to track down a problem where some query tests (currently working with TestJsonReader, but several others exhibit the same behavior) work with assertions enabled, but hang and time out with assertions disabled.
Will begin submitting for automated testing to see what the state of the rest of the world is, but wanted to make this available to look at for now because of its size.
Thanks,
Chris Westin
Re: Review Request 34004: DRILL-1942: Improved memory allocator
Posted by Chris Westin <ch...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34004/
-----------------------------------------------------------
(Updated June 11, 2015, 4:56 p.m.)
Review request for drill and Jacques Nadeau.
Changes
-------
Lots of changes. Mostly fixing many memory leaks, and cleaning up a bunch of TODOs.
Bugs: DRILL-1942
https://issues.apache.org/jira/browse/DRILL-1942
Repository: drill-git
Description
-------
Rewritten direct memory allocator. Simplified interface, and use, along with a means to support additional allocation policies in the future. There are features in the allocator and in DrillBuf that make finding leaks easier, as well as better enforcement of limits. New features include transfer of buffers, and better slicing support.
This is a preliminary patch to get the review started because it touches a lot of files (readers and record batches were made AutoCloseable in order to cover cleanup). Subsequent reviews can use the differential view to just see additional changes. The new allocator is in BaseAllocator.java (along with derived classes RootAllocator and ChildAllocator); DrillBuf also has significant changes. Most other changes in other files are just to use newer interfaces, or to change cleanup() to close(), or to close subordinate objects that are newly (Auto)Closeable. 1There are still a couple of things to do:
* Some TODO(cwestin)s to clean up tracing and debugging code, as well as adding javadoc
* Using the AllocatorOwner interface to replace the reallocation mechanism for FragmentContext and OperatorContext so that the allocator doesn't know anything about those objects.
Diffs (updated)
-----
common/src/main/java/org/apache/drill/common/AutoCloseablePointer.java PRE-CREATION
common/src/main/java/org/apache/drill/common/DrillAutoCloseables.java PRE-CREATION
common/src/main/java/org/apache/drill/common/DrillCloseables.java PRE-CREATION
common/src/main/java/org/apache/drill/common/HistoricalLog.java PRE-CREATION
common/src/main/java/org/apache/drill/common/StackTrace.java 54068ec
common/src/main/java/org/apache/drill/common/config/DrillConfig.java 522303f
common/src/main/java/org/apache/drill/common/config/NestedConfig.java 3fd885f
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java 9458db2
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveRecordReader.java 3c8b9ba
contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoRecordReader.java 182f5a4
exec/java-exec/src/main/codegen/templates/AbstractFieldWriter.java 1b5dad1
exec/java-exec/src/main/codegen/templates/BaseWriter.java ada410d
exec/java-exec/src/main/codegen/templates/ComplexWriters.java 980f9ac
exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 0dffa0b
exec/java-exec/src/main/codegen/templates/JsonOutputRecordWriter.java ea643f0
exec/java-exec/src/main/codegen/templates/ListWriters.java ab78603
exec/java-exec/src/main/codegen/templates/MapWriters.java 06a6813
exec/java-exec/src/main/codegen/templates/NullableValueVectors.java ce6a3a7
exec/java-exec/src/main/codegen/templates/ParquetOutputRecordWriter.java 35777b0
exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java 37b8fac
exec/java-exec/src/main/codegen/templates/StringOutputRecordWriter.java f704cca
exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java 529f21b
exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java 3ec6b3e
exec/java-exec/src/main/java/io/netty/buffer/FakeAllocator.java 721aff9
exec/java-exec/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java 2ca79f0
exec/java-exec/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java e332b13
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java 8a24e8d
exec/java-exec/src/main/java/org/apache/drill/exec/TestMemoryRetention.java 37e5389
exec/java-exec/src/main/java/org/apache/drill/exec/cache/AbstractStreamSerializable.java ef488d6
exec/java-exec/src/main/java/org/apache/drill/exec/cache/LoopedAbstractDrillSerializable.java d2a7458
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java 016cd92
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java c642c4a
exec/java-exec/src/main/java/org/apache/drill/exec/client/DumpCat.java 55d9cf3
exec/java-exec/src/main/java/org/apache/drill/exec/client/PrintingResultsListener.java f5a119d
exec/java-exec/src/main/java/org/apache/drill/exec/exception/FragmentSetupException.java c276846
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MappifyUtility.java e27234f
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctions.java 49f581f
exec/java-exec/src/main/java/org/apache/drill/exec/memory/Accountor.java ad6a787
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationPolicy.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationPolicyAgent.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationReservation.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocatorClosedException.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocatorOwner.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocatorsStatsMXBean.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java 057cfa6
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferAllocator.java 811cceb
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferLedger.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/ChainedAllocatorOwner.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/ChildAllocator.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/OutOfMemoryException.java 063f1c1
exec/java-exec/src/main/java/org/apache/drill/exec/memory/OutOfMemoryRuntimeException.java 305eabd
exec/java-exec/src/main/java/org/apache/drill/exec/memory/RootAllocator.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/RootAllocatorStatsMXBean.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/TopLevelAllocator.java e2d5b18
exec/java-exec/src/main/java/org/apache/drill/exec/ops/BufferManager.java c953bb3
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 1cbe886
exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContext.java 7eb7d8a
exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContextImpl.java ce9f351
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 06f8088
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java 6176f77
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScreenCreator.java 76dc91c
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java 1f6767c
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java 7e22e65
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/TopNBatch.java 516b028
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/WriterRecordBatch.java d5d64a7
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java e1b5909
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java 8af1508
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java b252971
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java c6a07f8
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/FilterRecordBatch.java 5eee9df
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java 5fd866f
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatch.java ee2ce7f
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatchBuilder.java 2798010
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/NestedLoopJoinBatch.java 2d37fa5
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/limit/LimitRecordBatch.java d9330ea
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java baf9bda
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/orderedpartitioner/OrderedPartitionRecordBatch.java 1286fe1
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java 31fc160
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/producer/ProducerConsumerBatch.java b9a1641
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/sort/SortBatch.java dea6ba8
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/sort/SortRecordBatchBuilder.java 00f1992
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/RemovingRecordBatch.java 57e7b55
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/trace/TraceRecordBatch.java 78e83d6
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java 445568b
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unorderedreceiver/UnorderedReceiverBatch.java 684f715
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/validate/IteratorValidatorBatchIterator.java efd155e
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameRecordBatch.java 428632f
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java 8871a5f
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/MSortTemplate.java 9b21ae3
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/PriorityQueueCopier.java 161ca6a
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/PriorityQueueCopierTemplate.java facf192
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java f7cfbf4
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java 330ec79
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractSingleRecordBatch.java dd90cab
exec/java-exec/src/main/java/org/apache/drill/exec/record/RawFragmentBatch.java f2f9450
exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java 6f10a1c
exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchLoader.java de6f665
exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorAccessible.java 9db1681
exec/java-exec/src/main/java/org/apache/drill/exec/record/WritableBatch.java 324829a
exec/java-exec/src/main/java/org/apache/drill/exec/record/selection/SelectionVector2.java 7a7c012
exec/java-exec/src/main/java/org/apache/drill/exec/record/selection/SelectionVector4.java 69bc78f
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcDecoder.java 74a4afb
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataServer.java 80d2d6e
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryDataBatch.java 914bd00
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryResultHandler.java 8443948
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserClient.java b39a103
exec/java-exec/src/main/java/org/apache/drill/exec/server/BootStrapContext.java d0a998e
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java 0640dbb
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractRecordReader.java 6e27628
exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordReader.java 61ccac5
exec/java-exec/src/main/java/org/apache/drill/exec/store/avro/AvroRecordReader.java a52fd22
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONRecordReader.java 0df6227
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JsonProcessor.java 4d8d4ba
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/reader/BaseJsonProcessor.java 7833631
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/compliant/CompliantTextRecordReader.java 254e0d8
exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockRecordReader.java fd97c48
exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockScanBatchCreator.java 74423bf
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ParquetRecordReader.java 0cbd480
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet2/DrillParquetReader.java 4e7d628
exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/PojoRecordReader.java a893da1
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/MemoryIterator.java e02b413
exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java c59ade9
exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExecutionControlsInjector.java e3a4ba6
exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java eddefd0
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseDataValueVector.java 6d356f2
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java 67c489d
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java f88a7bc
exec/java-exec/src/main/java/org/apache/drill/exec/vector/ValueVector.java ab9992e
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractContainerVector.java d14dca6
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractMapVector.java 3c01939
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java d0f38c2
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java b5de8b1
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java a97847b
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/DrillBufInputStream.java 1061a5c
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java 095d8c6
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/AbstractBaseWriter.java ec8c00b
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/ComplexWriterImpl.java a4a35e2
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/VectorContainerWriter.java 6b6ab46
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/writer/FieldWriter.java 3faa4f7
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/AbstractDataCollector.java d52cb5d
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/BaseRawBatchBuffer.java 11b6cc8
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/SpoolingRawBatchBuffer.java cfe5b6b
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 5d07b49
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/ForemanException.java 32a99ad
exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentManager.java ad880da
exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/NonRootFragmentManager.java 77440c5
exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/RootFragmentManager.java b770a33
exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java a07f621
exec/java-exec/src/test/java/org/apache/drill/PlanTestBase.java f909681
exec/java-exec/src/test/java/org/apache/drill/PlanningBase.java da033f7
exec/java-exec/src/test/java/org/apache/drill/QueryTestUtil.java e218d6c
exec/java-exec/src/test/java/org/apache/drill/TestAllocationException.java 051ad4e
exec/java-exec/src/test/java/org/apache/drill/TestTpchLimit0.java 22471c8
exec/java-exec/src/test/java/org/apache/drill/TestTpchPlanning.java 707ea78
exec/java-exec/src/test/java/org/apache/drill/exec/client/DumpCatTest.java 7c58b19
exec/java-exec/src/test/java/org/apache/drill/exec/compile/bytecode/ReplaceMethodInvoke.java bc2d929
exec/java-exec/src/test/java/org/apache/drill/exec/expr/ExpressionTest.java 239a099
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestByteComparisonFunctions.java 1e9a47c
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestCastFunctions.java 3ba8743
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestMathFunctions.java 4f06a9d
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestNewMathFunctions.java 880184e
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestRepeatedFunction.java 73c7508
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestAllocators.java 74ce225
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestBaseAllocator.java PRE-CREATION
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestEndianess.java 48ddada
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/SimpleRootExec.java 42d2193
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestCastFunctions.java ffa8765
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestComparisonFunctions.java c69c6f5
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestImplicitCastFunctions.java 03c6f41
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestOptiqPlans.java dc37071
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestSimpleFragmentRun.java 09ba1a5
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestSimpleFunctions.java d551319
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestStringFunctions.java d72c1e1
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestUnionExchange.java 9c24f79
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/agg/TestAgg.java d2616a8
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/common/TestHashTable.java b02249d
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/filter/TestSimpleFilter.java a069078
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoin.java 6c067fe
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestMergeJoin.java 18555c7
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/limit/TestSimpleLimit.java 7cdb41a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/mergereceiver/TestMergingReceiver.java 0122c08
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/project/TestSimpleProjection.java 43c430a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/sort/TestSimpleSort.java f37624a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/trace/TestTraceMultiRecordBatch.java b82846e
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/trace/TestTraceOutputDump.java 1cb72ff
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/union/TestSimpleUnion.java 07de27f
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestWriter.java f4d505d
exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestLoad.java f57e765
exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java 037c8c6
exec/java-exec/src/test/java/org/apache/drill/exec/server/DrillClientFactory.java 4230518
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestDrillbitResilience.java 696aed8
exec/java-exec/src/test/java/org/apache/drill/exec/store/TestDirectCodecFactory.java 644144e
exec/java-exec/src/test/java/org/apache/drill/exec/store/ischema/TestInfoSchemaFilterPushDown.java b6e789b
exec/java-exec/src/test/java/org/apache/drill/exec/store/json/TestJsonRecordReader.java bb1af9e
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/FieldInfo.java 34f60ba
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 8fdaa72
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 6326478
exec/java-exec/src/test/java/org/apache/drill/exec/testing/TestResourceLeak.java d7e317c
exec/java-exec/src/test/java/org/apache/drill/exec/vector/TestSplitAndTransfer.java 4b3aa8a
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/TestEmptyPopulation.java 06a73e2
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/TestEmptyPopulator.java PRE-CREATION
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/fn/TestJsonReaderWithSparseFiles.java d674d47
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestComplexTypeReader.java 521a41d
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestComplexTypeWriter.java cb7bef2
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestJsonReader.java 912a5f0
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestRepeated.java 6e2a2b5
exec/java-exec/src/test/resources/logback.xml 54ccb42
exec/java-exec/src/test/resources/logback.xml.sav PRE-CREATION
exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillConnectionImpl.java 7c6ef7e
Diff: https://reviews.apache.org/r/34004/diff/
Testing (updated)
-------
mvn install passes
A few tests fail the presubmit suite, but all with the same IndexOutOfBoundsException. I've just gotten a reproducible case to run standalone on my laptop, so I'm debugging that.
Thanks,
Chris Westin
Re: Review Request 34004: DRILL-1942: Improved memory allocator
Posted by Chris Westin <ch...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34004/
-----------------------------------------------------------
(Updated June 11, 2015, 3:58 p.m.)
Review request for drill and Jacques Nadeau.
Bugs: DRILL-1942
https://issues.apache.org/jira/browse/DRILL-1942
Repository: drill-git
Description
-------
Rewritten direct memory allocator. Simplified interface, and use, along with a means to support additional allocation policies in the future. There are features in the allocator and in DrillBuf that make finding leaks easier, as well as better enforcement of limits. New features include transfer of buffers, and better slicing support.
This is a preliminary patch to get the review started because it touches a lot of files (readers and record batches were made AutoCloseable in order to cover cleanup). Subsequent reviews can use the differential view to just see additional changes. The new allocator is in BaseAllocator.java (along with derived classes RootAllocator and ChildAllocator); DrillBuf also has significant changes. Most other changes in other files are just to use newer interfaces, or to change cleanup() to close(), or to close subordinate objects that are newly (Auto)Closeable. 1There are still a couple of things to do:
* Some TODO(cwestin)s to clean up tracing and debugging code, as well as adding javadoc
* Using the AllocatorOwner interface to replace the reallocation mechanism for FragmentContext and OperatorContext so that the allocator doesn't know anything about those objects.
Diffs (updated)
-----
common/src/main/java/org/apache/drill/common/DrillAutoCloseables.java PRE-CREATION
common/src/main/java/org/apache/drill/common/DrillCloseables.java PRE-CREATION
common/src/main/java/org/apache/drill/common/config/DrillConfig.java 522303f
common/src/main/java/org/apache/drill/common/config/NestedConfig.java 3fd885f
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java 9458db2
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveRecordReader.java 5394ee3
contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoRecordReader.java 182f5a4
exec/java-exec/src/main/codegen/templates/AbstractFieldWriter.java 1b5dad1
exec/java-exec/src/main/codegen/templates/BaseWriter.java ada410d
exec/java-exec/src/main/codegen/templates/ComplexWriters.java 49c75d1
exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 1059bfb
exec/java-exec/src/main/codegen/templates/JsonOutputRecordWriter.java ea643f0
exec/java-exec/src/main/codegen/templates/ListWriters.java 6df4248
exec/java-exec/src/main/codegen/templates/MapWriters.java 6ee8035
exec/java-exec/src/main/codegen/templates/NullableValueVectors.java 9d03efb
exec/java-exec/src/main/codegen/templates/ParquetOutputRecordWriter.java 0d24041
exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java c0fba66
exec/java-exec/src/main/codegen/templates/StringOutputRecordWriter.java f704cca
exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java 7aa7415
exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java 7f80f7a
exec/java-exec/src/main/java/io/netty/buffer/FakeAllocator.java 721aff9
exec/java-exec/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java 2ca79f0
exec/java-exec/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java e332b13
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java be8c7a0
exec/java-exec/src/main/java/org/apache/drill/exec/TestMemoryRetention.java 37e5389
exec/java-exec/src/main/java/org/apache/drill/exec/cache/AbstractStreamSerializable.java ef488d6
exec/java-exec/src/main/java/org/apache/drill/exec/cache/LoopedAbstractDrillSerializable.java d2a7458
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java 8e2ce96
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java 5b28f16
exec/java-exec/src/main/java/org/apache/drill/exec/client/DumpCat.java 55d9cf3
exec/java-exec/src/main/java/org/apache/drill/exec/client/PrintingResultsListener.java f5a119d
exec/java-exec/src/main/java/org/apache/drill/exec/exception/FragmentSetupException.java c276846
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MappifyUtility.java e27234f
exec/java-exec/src/main/java/org/apache/drill/exec/memory/Accountor.java eb932ad
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationPolicy.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationPolicyAgent.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocationReservation.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocatorOwner.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AllocatorsStatsMXBean.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java 057cfa6
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BaseAllocator.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferAllocator.java c233ac5
exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferLedger.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/ChainedAllocatorOwner.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/ChildAllocator.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/OutOfMemoryException.java 063f1c1
exec/java-exec/src/main/java/org/apache/drill/exec/memory/OutOfMemoryRuntimeException.java 305eabd
exec/java-exec/src/main/java/org/apache/drill/exec/memory/RootAllocator.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/RootAllocatorStatsMXBean.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/memory/TopLevelAllocator.java a78deb6
exec/java-exec/src/main/java/org/apache/drill/exec/ops/BufferManager.java c953bb3
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java b108924
exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContext.java 7cc52ba
exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContextImpl.java 6dbd880
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 9e2f210
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java f56dae3
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScreenCreator.java 5b4d7bd
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java 67062f3
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java 369c0ec
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/TopNBatch.java 9f6bea9
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/WriterRecordBatch.java 15fb7b5
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java b753574
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java 1b90dd8
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java c1c5cb9
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java d2282c8
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/FilterRecordBatch.java 5eee9df
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java dd53477
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatch.java 6466f70
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatchBuilder.java 1187bd6
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/NestedLoopJoinBatch.java aa4b300
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/limit/LimitRecordBatch.java eff9e61
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java ce683cb
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/orderedpartitioner/OrderedPartitionRecordBatch.java 63b7eba
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java 16d1400
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/producer/ProducerConsumerBatch.java 35bf3cd
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/sort/SortBatch.java 74b7d85
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/sort/SortRecordBatchBuilder.java e559ece
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/RemovingRecordBatch.java aa9297e
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/trace/TraceRecordBatch.java af45815
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java 66bc3e3
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unorderedreceiver/UnorderedReceiverBatch.java 66a2092
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/validate/IteratorValidatorBatchIterator.java efd155e
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameRecordBatch.java 86d11d5
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java e88bc67
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/MSortTemplate.java 9b97e1c
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/PriorityQueueCopier.java 161ca6a
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/PriorityQueueCopierTemplate.java f7786b7
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java 4e348bb
exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractSingleRecordBatch.java dd90cab
exec/java-exec/src/main/java/org/apache/drill/exec/record/RawFragmentBatch.java edd79ac
exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java 6f10a1c
exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchLoader.java 1b8b7ce
exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorAccessible.java 9db1681
exec/java-exec/src/main/java/org/apache/drill/exec/record/WritableBatch.java 308a8bc
exec/java-exec/src/main/java/org/apache/drill/exec/record/selection/SelectionVector2.java 7a7c012
exec/java-exec/src/main/java/org/apache/drill/exec/record/selection/SelectionVector4.java 69bc78f
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcDecoder.java 74a4afb
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataServer.java 6f8e20b
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/QueryDataBatch.java 914bd00
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserClient.java a8bad78
exec/java-exec/src/main/java/org/apache/drill/exec/server/BootStrapContext.java d0a998e
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java e7a9a3c
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractRecordReader.java 6e27628
exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordReader.java 61ccac5
exec/java-exec/src/main/java/org/apache/drill/exec/store/avro/AvroRecordReader.java a52fd22
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONRecordReader.java 3d789eb
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JsonProcessor.java 4d8d4ba
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/reader/BaseJsonProcessor.java 7833631
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/compliant/CompliantTextRecordReader.java b2af32d
exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockRecordReader.java fd97c48
exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockScanBatchCreator.java 74423bf
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ParquetRecordReader.java 58cf321
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet2/DrillParquetReader.java 99ac19c
exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/PojoRecordReader.java cf98b83
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/MemoryIterator.java e02b413
exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java 0322f36
exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java 7c77ca2
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseDataValueVector.java 0c6097c
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java 22f0fe7
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java 2fc5bf3
exec/java-exec/src/main/java/org/apache/drill/exec/vector/ValueVector.java e4a0997
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractContainerVector.java d14dca6
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractMapVector.java 78846dc
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java 4138839
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java c061029
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java e5d48dd
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/DrillBufInputStream.java 1061a5c
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java 095d8c6
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/AbstractBaseWriter.java ec8c00b
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/ComplexWriterImpl.java a4a35e2
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/VectorContainerWriter.java 6b6ab46
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/writer/FieldWriter.java 3faa4f7
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/SpoolingRawBatchBuffer.java 2a79e42
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 49d0c94
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/ForemanException.java 32a99ad
exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java f8ec090
exec/java-exec/src/test/java/org/apache/drill/PlanTestBase.java f909681
exec/java-exec/src/test/java/org/apache/drill/PlanningBase.java da033f7
exec/java-exec/src/test/java/org/apache/drill/QueryTestUtil.java e218d6c
exec/java-exec/src/test/java/org/apache/drill/TestTpchPlanning.java 707ea78
exec/java-exec/src/test/java/org/apache/drill/exec/client/DumpCatTest.java 7c58b19
exec/java-exec/src/test/java/org/apache/drill/exec/compile/bytecode/ReplaceMethodInvoke.java bc2d929
exec/java-exec/src/test/java/org/apache/drill/exec/expr/ExpressionTest.java 239a099
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestByteComparisonFunctions.java 1e9a47c
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestCastFunctions.java 3ba8743
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestMathFunctions.java 4f06a9d
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestNewMathFunctions.java 880184e
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestRepeatedFunction.java 73c7508
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestAllocators.java 74ce225
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestBaseAllocator.java PRE-CREATION
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestEndianess.java 48ddada
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/SimpleRootExec.java 42d2193
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestCastFunctions.java ffa8765
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestComparisonFunctions.java c69c6f5
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestImplicitCastFunctions.java 03c6f41
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestOptiqPlans.java dc37071
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestSimpleFragmentRun.java 09ba1a5
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestSimpleFunctions.java d551319
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestStringFunctions.java d72c1e1
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestUnionExchange.java 9c24f79
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/agg/TestAgg.java b39566a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/common/TestHashTable.java b02249d
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/filter/TestSimpleFilter.java a069078
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoin.java 6c067fe
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestMergeJoin.java 18555c7
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/limit/TestSimpleLimit.java 7cdb41a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/mergereceiver/TestMergingReceiver.java 0122c08
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/project/TestSimpleProjection.java 43c430a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/sort/TestSimpleSort.java f37624a
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/trace/TestTraceMultiRecordBatch.java b82846e
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/trace/TestTraceOutputDump.java 1cb72ff
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/union/TestSimpleUnion.java 07de27f
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestWriter.java f4d505d
exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestLoad.java f57e765
exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java 1564aea
exec/java-exec/src/test/java/org/apache/drill/exec/server/DrillClientFactory.java 4230518
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestDrillbitResilience.java da69e9e
exec/java-exec/src/test/java/org/apache/drill/exec/store/TestDirectCodecFactory.java 644144e
exec/java-exec/src/test/java/org/apache/drill/exec/store/ischema/TestInfoSchemaFilterPushDown.java b6e789b
exec/java-exec/src/test/java/org/apache/drill/exec/store/json/TestJsonRecordReader.java bb1af9e
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/FieldInfo.java 34f60ba
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 8fdaa72
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetResultListener.java 6326478
exec/java-exec/src/test/java/org/apache/drill/exec/vector/TestSplitAndTransfer.java 4b3aa8a
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/TestEmptyPopulator.java 23cc316
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/fn/TestJsonReaderWithSparseFiles.java d674d47
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestComplexTypeReader.java 521a41d
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestComplexTypeWriter.java cb7bef2
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestJsonReader.java dfa89ca
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestRepeated.java 6e2a2b5
exec/java-exec/src/test/resources/logback.xml 2dcfed9
exec/jdbc/src/main/java/org/apache/drill/jdbc/DrillConnectionImpl.java 74c6655
Diff: https://reviews.apache.org/r/34004/diff/
Testing
-------
Many unit tests run by hand. Currently trying to track down a problem where some query tests (currently working with TestJsonReader, but several others exhibit the same behavior) work with assertions enabled, but hang and time out with assertions disabled.
Will begin submitting for automated testing to see what the state of the rest of the world is, but wanted to make this available to look at for now because of its size.
Thanks,
Chris Westin