You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Ahshan (JIRA)" <ji...@apache.org> on 2018/09/28 06:09:00 UTC
[jira] [Created] (IMPALA-7641) Memory Limit Exceeded
Ahshan created IMPALA-7641:
------------------------------
Summary: Memory Limit Exceeded
Key: IMPALA-7641
URL: https://issues.apache.org/jira/browse/IMPALA-7641
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.6.4
Reporter: Ahshan
Attachments: profile(8).txt
We are using CDH distribution with impala version -impalad version 2.6.0-cdh5.8.2 RELEASE
As per my understanding memory requirement is of 288 MB and we have an total of 18 Impala Daemons which sum upto 5184MB of total memory consumption
considering the above details, it should not lead to an memory issue when the MEM_LIMIT is set to 20GB
Hence , Could you please let us know the cause of memory limit exceeding
select * from emp_sales where job_id = 55451 and uploaded_month = 201808 limit 1
+-----------------------------------------------------------+
|Explain String|
+-----------------------------------------------------------+
|Estimated Per-Host Requirements: Memory=288.00MB VCores=1|
| |
|01:EXCHANGE [UNPARTITIONED]|
| |limit: 1|
| | |
|00:SCAN HDFS [fenet5.hmig_os_changes_details_malicious]|
|partitions=1/25 files=3118 size=110.01GB|
|predicates: job_id = 55451|
|limit: 1|
+-----------------------------------------------------------+
WARNINGS:
Memory limit exceeded
HdfsParquetScanner::ReadDataPage() failed to allocate 269074889 bytes for dictionary.
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 257.23 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=20.00 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 255.63 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=20.00 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 255.27 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=20.00 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 255.39 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=20.00 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 16.09 KB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.74 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.74 GB
HDFS_SCAN_NODE (id=0): Consumption=19.74 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 15.20 KB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.64 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.64 GB
HDFS_SCAN_NODE (id=0): Consumption=19.64 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 14.61 KB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.64 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.64 GB
HDFS_SCAN_NODE (id=0): Consumption=19.64 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 257.11 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.47 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.47 GB
HDFS_SCAN_NODE (id=0): Consumption=19.47 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 255.51 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.24 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.24 GB
HDFS_SCAN_NODE (id=0): Consumption=19.24 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 255.32 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.24 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.24 GB
HDFS_SCAN_NODE (id=0): Consumption=19.24 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 255.73 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.49 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.49 GB
HDFS_SCAN_NODE (id=0): Consumption=19.49 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 255.29 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.49 GB
Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.49 GB
HDFS_SCAN_NODE (id=0): Consumption=19.49 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 256.61 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: memory limit exceeded. Limit=20.00 GB Consumption=20.00 GB
Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=20.00 GB
HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 256.05 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.69 GB
Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=19.69 GB
HDFS_SCAN_NODE (id=0): Consumption=19.69 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 255.35 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=17.97 GB
Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=17.97 GB
HDFS_SCAN_NODE (id=0): Consumption=17.72 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 1.02 KB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=17.63 GB
Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=17.63 GB
HDFS_SCAN_NODE (id=0): Consumption=17.63 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 1.01 KB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=16.94 GB
Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.94 GB
HDFS_SCAN_NODE (id=0): Consumption=16.68 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 88.00 KB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=16.61 GB
Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.61 GB
HDFS_SCAN_NODE (id=0): Consumption=16.36 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
HDFS_SCAN_NODE (id=0) could not allocate 255.23 MB without exceeding limit.
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=16.30 GB
Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.30 GB
HDFS_SCAN_NODE (id=0): Consumption=16.30 GB
DataStreamSender: Consumption=1.45 KB
Block Manager: Limit=16.00 GB Consumption=0
Memory Limit Exceeded
Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=8.02 GB
Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=8.02 GB
HDFS_SCAN_NODE (id=0): Consumption=8.02 GB
Block Manager: Limit=16.00 GB Consumption=0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org