You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Paul Rogers (JIRA)" <ji...@apache.org> on 2016/12/23 03:46:58 UTC
[jira] [Commented] (DRILL-5157)
TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async
Snappy error
[ https://issues.apache.org/jira/browse/DRILL-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771786#comment-15771786 ]
Paul Rogers commented on DRILL-5157:
------------------------------------
There are really two errors here.
First is that the Snappy library is not available on the Mac where I ran the tests. See: http://stackoverflow.com/questions/15158405/unable-to-load-native-hadoop-library-for-mac-os-x
It seems that the Parquet file being read was encoded with Snappy, and so the Snappy library is required.
The solution to this is to describe how to obtain the library, or find some Java alternative when Snappy is not available.
The second error is that the error caused a memory leak. In general, we'd like errors, even those like the above, to clean up memory to avoid putting the Drillbit into a bad state.
> TestDrillbitResilience.doMemoryLeaksWhenCancelled fails with Parquet async Snappy error
> ---------------------------------------------------------------------------------------
>
> Key: DRILL-5157
> URL: https://issues.apache.org/jira/browse/DRILL-5157
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.9.0
> Reporter: Paul Rogers
>
> Run the {{TestDrillbitResilience.doMemoryLeaksWhenCancelled}} unit test. It fails with the following stack trace and the memory leak trace shown second.
> Strangely, this error appears only if the test is run as part of the overall suite. The error does not appear if the test is run individually in the debugger.
> Stack trace that seems to show that the code was trying to find a Snappy native library:
> {code}
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
> at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
> at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
> at org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$DecompressionHelper.decompress(AsyncPageReader.java:402)
> at org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.decompress(AsyncPageReader.java:169)
> at org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.getDecompressedPageData(AsyncPageReader.java:96)
> at org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.nextInternal(AsyncPageReader.java:219)
> at org.apache.drill.exec.store.parquet.columnreaders.PageReader.next(PageReader.java:280)
> at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:250)
> at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:178)
> at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:130)
> at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:485)
> at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:479)
> at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:562)
> at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178)
> at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> …
> {code}
> Resulting memory leak if the test is allowed to complete:
> {code}
> java.lang.AssertionError: Query state is incorrect (expected: CANCELED, actual: FAILED) AND/OR
> Exception thrown: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalStateException: Allocator[op:3:0:15:ParquetRowGroupScan] closed with outstanding buffers allocated (2).
> Allocator(op:3:0:15:ParquetRowGroupScan) 1000000/393216/3162112/10000000000 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 2
> ledger[3407] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true, size: 131072, references: 1, life: 73148192887288..0, allocatorManager: [3027, life: 73148192235794..0] holds 1 buffers.
> DrillBuf[4949], udle: [3028 0..131072]
> ledger[3471] allocator: op:3:0:15:ParquetRowGroupScan), isOwning: true, size: 262144, references: 1, life: 73148451288840..0, allocatorManager: [3091, life: 73148451257480..0] holds 1 buffers.
> DrillBuf[5017], udle: [3092 0..262144]
> reservations: 0
> Fragment 3:0
> [Error Id: 8502074b-f488-4a14-bf7d-a2a4480392cd on 172.30.1.67:31016]
> at org.apache.drill.exec.server.TestDrillbitResilience.assertStateCompleted(TestDrillbitResilience.java:861)
> at org.apache.drill.exec.server.TestDrillbitResilience.assertCancelledWithoutException(TestDrillbitResilience.java:876)
> at org.apache.drill.exec.server.TestDrillbitResilience.doMemoryLeaksWhenCancelled(TestDrillbitResilience.java:680)
> at org.apache.drill.exec.server.TestDrillbitResilience.memoryLeaksWhenCancelled(TestDrillbitResilience.java:647)
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)