OOM during flattening

[Alvin Chyan <Al...@turn.com>]

Hi Drill experts,


While attempting to run the query, "select <nestedTableName>.<record>.<column1>, <nestedTableName>.<record>.<column2> from (select flatten(tbl.<nestedTableName>.`array`) <record> from dfs.`/server/apache-drill-1.0.0/sample-data/myfile` tbl ) as <nestedTableName>;"


I run into an OOM error.  It seems to be running out of memory during the flattening process.


java.lang.RuntimeException: java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

Fragment 1:4

[Error Id: 69a33085-d059-4e71-a899-e181ed741715 on localhost:31010]

  (org.apache.drill.exec.memory.OutOfMemoryRuntimeException) Failure while reallocating buffer to 16777216 bytes
    org.apache.drill.exec.vector.Float4Vector.reAlloc():192
    org.apache.drill.exec.vector.Float4Vector.copyFromSafe():292
    org.apache.drill.exec.vector.NullableFloat4Vector.copyFromSafe():301
    org.apache.drill.exec.vector.NullableFloat4Vector$TransferImpl.copyValueSafe():266
    org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapTransferPair.copyValueSafe():355
    org.apache.drill.exec.vector.complex.MapVector$MapTransferPair.copyValueSafe():208
    org.apache.drill.exec.vector.complex.MapVector.copyFromSafe():83
    org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl.copyAsValue():97
    org.apache.drill.exec.test.generated.FlattenerGen5.doEval():30
    org.apache.drill.exec.test.generated.FlattenerGen5.flattenRecords():93
    org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.doWork():153
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93
    org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.innerNext():119
    org.apache.drill.exec.record.AbstractRecordBatch.next():146
    org.apache.drill.exec.record.AbstractRecordBatch.next():105
    org.apache.drill.exec.record.AbstractRecordBatch.next():95
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
    org.apache.drill.exec.record.AbstractRecordBatch.next():146
    org.apache.drill.exec.record.AbstractRecordBatch.next():105
    org.apache.drill.exec.record.AbstractRecordBatch.next():95
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
    org.apache.drill.exec.record.AbstractRecordBatch.next():146
    org.apache.drill.exec.physical.impl.BaseRootExec.next():83
    org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():95
    org.apache.drill.exec.physical.impl.BaseRootExec.next():73
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():259
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():253
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():415
    org.apache.hadoop.security.UserGroupInformation.doAs():1556
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():253
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1145
    java.util.concurrent.ThreadPoolExecutor$Worker.run():615
    java.lang.Thread.run():745

    at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
    at sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:85)
    at sqlline.TableOutputFormat.print(TableOutputFormat.java:116)



I've tried enabling the planner.memory.enable_memory_estimation property, but that didn't help.

I have the following memory settings:

DRILL_MAX_DIRECT_MEMORY="8G"
DRILL_HEAP="4G"

And the file itself is only 1.4 GB.  Is there some other way to trade off performance for reliability?  Also, are there some known limitations with flatten?


Thanks,

Alvin