You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Paul Rogers (JIRA)" <ji...@apache.org> on 2016/12/24 01:02:35 UTC
[jira] [Commented] (DRILL-5160) Memory leak in Parquet async reader
when Snappy fails
[ https://issues.apache.org/jira/browse/DRILL-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15773991#comment-15773991 ]
Paul Rogers commented on DRILL-5160:
------------------------------------
Resulting memory leak from the test described in DRILL-5157:
{code}
java.lang.AssertionError: Query state is incorrect (expected: CANCELED, actual: FAILED) AND/OR
Exception thrown: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalStateException: Allocator[op:3:0:15:ParquetRowGroupScan] closed with outstanding buffers allocated (2).
Allocator(op:3:0:15:ParquetRowGroupScan) 1000000/393216/3162112/10000000000 (res/actual/peak/limit)
child allocators: 0
ledgers: 2
ledger[3407] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true, size: 131072, references: 1, life: 73148192887288..0, allocatorManager: [3027, life: 73148192235794..0] holds 1 buffers.
DrillBuf[4949], udle: [3028 0..131072]
ledger[3471] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true, size: 262144, references: 1, life: 73148451288840..0, allocatorManager: [3091, life: 73148451257480..0] holds 1 buffers.
DrillBuf[5017], udle: [3092 0..262144]
reservations: 0
Fragment 3:0
[Error Id: 8502074b-f488-4a14-bf7d-a2a4480392cd on 172.30.1.67:31016]
at org.apache.drill.exec.server.TestDrillbitResilience.assertStateCompleted(TestDrillbitResilience.java:861)
at org.apache.drill.exec.server.TestDrillbitResilience.assertCancelledWithoutException(TestDrillbitResilience.java:876)
at org.apache.drill.exec.server.TestDrillbitResilience.doMemoryLeaksWhenCancelled(TestDrillbitResilience.java:680)
at org.apache.drill.exec.server.TestDrillbitResilience.memoryLeaksWhenCancelled(TestDrillbitResilience.java:647)
...
{code}
> Memory leak in Parquet async reader when Snappy fails
> -----------------------------------------------------
>
> Key: DRILL-5160
> URL: https://issues.apache.org/jira/browse/DRILL-5160
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.9.0
> Reporter: Paul Rogers
> Assignee: Parth Chandra
>
> See the details in DRILL-5157. The Parquet async reader uses the Snappy library. If the call into Snappy fails (in my case, due to missing dependency management in the Drill pom.xml file), the Parquet reader fails (as it should.)
> Unfortunately, the Parquet reader leaks memory which cause {{IllegalStateException}} errors in production, assertion failures in unit tests.
> To reproduce this the easy way (to avoid the need to undo the fix for DRILL-5157 in Snappy dependencies), use Drill's exception injector in inject an exception at the line where we call into Snappy (See DRILL-5157 for details.)
> Set a breakpoint exception for {{IllegalStateException}} you will see the memory leak.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)