You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Joe McDonnell (Jira)" <ji...@apache.org> on 2020/02/14 19:20:00 UTC
[jira] [Commented] (IMPALA-8521) Lots of "unreleased ByteBuffers
allocated by read()" errors from HDFS client
[ https://issues.apache.org/jira/browse/IMPALA-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037227#comment-17037227 ]
Joe McDonnell commented on IMPALA-8521:
---------------------------------------
This reproduces on master (with both USE_CDP_HIVE=false and USE_CDP_HIVE=true). Here is the command to run the test:
{noformat}
run-tests.py --exploration_strategy exhaustive query_test/test_hdfs_caching.py::TestHdfsCachingFallbackPath{noformat}
The test is intentionally creating a scenario where the hadoopReadZero will fail. When I ran the test multiple times without restarting Impala, the count of direct buffers seems to keep going up, but the memory seems to be bounded. From looking at the Impalad's /jmx page:
{noformat}
After first run:
{
"name": "java.nio:type=BufferPool,name=direct",
"modelerType": "sun.management.ManagementFactoryHelper$1",
"Name": "direct",
"Count": 38,
"TotalCapacity": 1245665,
"MemoryUsed": 1245666,
"ObjectName": "java.nio:type=BufferPool,name=direct"
},
After second run:
{
"name": "java.nio:type=BufferPool,name=direct",
"modelerType": "sun.management.ManagementFactoryHelper$1",
"Name": "direct",
"Count": 67,
"TotalCapacity": 2532003,
"MemoryUsed": 2532004,
"ObjectName": "java.nio:type=BufferPool,name=direct"
},
Third run:
{
"name": "java.nio:type=BufferPool,name=direct",
"modelerType": "sun.management.ManagementFactoryHelper$1",
"Name": "direct",
"Count": 108,
"TotalCapacity": 2761671,
"MemoryUsed": 2761672,
"ObjectName": "java.nio:type=BufferPool,name=direct"
},
Fourth run:
{
"name": "java.nio:type=BufferPool,name=direct",
"modelerType": "sun.management.ManagementFactoryHelper$1",
"Name": "direct",
"Count": 117,
"TotalCapacity": 1868479,
"MemoryUsed": 1868480,
"ObjectName": "java.nio:type=BufferPool,name=direct"
},
Fifth run:
{
"name": "java.nio:type=BufferPool,name=direct",
"modelerType": "sun.management.ManagementFactoryHelper$1",
"Name": "direct",
"Count": 157,
"TotalCapacity": 2089955,
"MemoryUsed": 2089956,
"ObjectName": "java.nio:type=BufferPool,name=direct"
},
{noformat}
> Lots of "unreleased ByteBuffers allocated by read()" errors from HDFS client
> ----------------------------------------------------------------------------
>
> Key: IMPALA-8521
> URL: https://issues.apache.org/jira/browse/IMPALA-8521
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Sahil Takiar
> Priority: Critical
>
> I'm looking at some job logs and seeing a bunch of errors like this. I don't know if it's benign or if it's something more serious.
> {noformat}
> I0507 07:34:53.934693 20195 scan-range.cc:607] dd4d6eb8d2ad9587:6b44fe1b00000002] Cache read failed for scan range: file=hdfs://localhost:20500/test-warehouse/f861f1a3/nation.tbl disk_id=0 offset=1024 exclusive_hdfs_fh=0xec09220 num_remote_bytes=0 cancel_status= buffer_queue=0 num_buffers_in_readers=0 unused_iomgr_buffers=0 unused_iomgr_buffer_bytes=0 blocked_on_buffer=0. Switching to disk read path.
> W0507 07:34:53.934787 20195 DFSInputStream.java:668] dd4d6eb8d2ad9587:6b44fe1b00000002] closing file /test-warehouse/f861f1a3/nation.tbl, but there are still unreleased ByteBuffers allocated by read(). Please release java.nio.DirectByteBufferR[pos=1024 lim=2048 cap=2199].
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org