You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Ahshan (JIRA)" <ji...@apache.org> on 2018/09/28 06:14:00 UTC
[jira] [Updated] (IMPALA-7641) Memory Limit Exceeded
[ https://issues.apache.org/jira/browse/IMPALA-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahshan updated IMPALA-7641:
---------------------------
Epic Color: ghx-label-2 (was: ghx-label-3)
> Memory Limit Exceeded
> ---------------------
>
> Key: IMPALA-7641
> URL: https://issues.apache.org/jira/browse/IMPALA-7641
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.6.4
> Reporter: Ahshan
> Priority: Blocker
> Labels: memory
> Attachments: profile(8).txt
>
>
> We are using CDH distribution with impala version -impalad version 2.6.0-cdh5.8.2 RELEASE
>
> As per my understanding memory requirement is of 288 MB and we have an total of 18 Impala Daemons which sum upto 5184MB of total memory consumption
> considering the above details, it should not lead to an memory issue when the MEM_LIMIT is set to 20GB
> Hence , Could you please let us know the cause of memory limit exceeding
> select * from emp_sales where job_id = 55451 and uploaded_month = 201808 limit 1
> +-----------------------------------------------------------+
> |Explain String|
> +-----------------------------------------------------------+
> |Estimated Per-Host Requirements: Memory=288.00MB VCores=1|
> | |
> |01:EXCHANGE [UNPARTITIONED]|
> | |limit: 1|
> | | |
> |00:SCAN HDFS [fenet5.hmig_os_changes_details_malicious]|
> |partitions=1/25 files=3118 size=110.01GB|
> |predicates: job_id = 55451|
> |limit: 1|
> +-----------------------------------------------------------+
> WARNINGS:
> Memory limit exceeded
> HdfsParquetScanner::ReadDataPage() failed to allocate 269074889 bytes for dictionary.
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 257.23 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=20.00 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
> HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.63 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=20.00 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
> HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.27 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=20.00 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
> HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.39 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=20.00 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=20.00 GB
> HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 16.09 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.74 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.74 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.74 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 15.20 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.64 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.64 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.64 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 14.61 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.64 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.64 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.64 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 257.11 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.47 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.47 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.47 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.51 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.24 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.24 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.24 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.32 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.24 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.24 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.24 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.73 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.49 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.49 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.49 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.29 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.49 GB
> Fragment 294eb435fbf8fc63:f529602818758c8b: Consumption=19.49 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.49 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 256.61 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: memory limit exceeded. Limit=20.00 GB Consumption=20.00 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=20.00 GB
> HDFS_SCAN_NODE (id=0): Consumption=20.00 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 256.05 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=19.69 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=19.69 GB
> HDFS_SCAN_NODE (id=0): Consumption=19.69 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.35 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=17.97 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=17.97 GB
> HDFS_SCAN_NODE (id=0): Consumption=17.72 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 1.02 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=17.63 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=17.63 GB
> HDFS_SCAN_NODE (id=0): Consumption=17.63 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 1.01 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=16.94 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.94 GB
> HDFS_SCAN_NODE (id=0): Consumption=16.68 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 88.00 KB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=16.61 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.61 GB
> HDFS_SCAN_NODE (id=0): Consumption=16.36 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> HDFS_SCAN_NODE (id=0) could not allocate 255.23 MB without exceeding limit.
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=16.30 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=16.30 GB
> HDFS_SCAN_NODE (id=0): Consumption=16.30 GB
> DataStreamSender: Consumption=1.45 KB
> Block Manager: Limit=16.00 GB Consumption=0
> Memory Limit Exceeded
> Query(294eb435fbf8fc63:f529602818758c80) Limit: Limit=20.00 GB Consumption=8.02 GB
> Fragment 294eb435fbf8fc63:f529602818758c8e: Consumption=8.02 GB
> HDFS_SCAN_NODE (id=0): Consumption=8.02 GB
> Block Manager: Limit=16.00 GB Consumption=0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org