You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Dong Bo 董博 <do...@fosun.com> on 2017/09/29 08:11:54 UTC
Impala Memory Leak (version 2.7.0-cdh5.10.2 )
Hi Experts,
We are running queries on impalad version 2.7.0-cdh5.10.2 RELEASE (build 38c989c0330ea952133111e41965ff9af96412d3), and issued with a memory leak problem on the impala deamon .
Once query end with error Memory limit exceeded , the allocated memory won’t be free until impala deamon restarted. Memory limit exceeded is in attachement.
I find jira tickets https://issues.apache.org/jira/browse/IMPALA-3589 , https://issues.apache.org/jira/browse/IMPALA-3582 with same issue as I am facing tagged as “Fixed” in version 2.6.0, but strange I am still facing the issue in a higher version.
Please kindly advise , it will be very helpful.
Thanks & Regards
Carl Dong
Re: 答复: Impala Memory Leak (version 2.7.0-cdh5.10.2 )
Posted by Jeszy <je...@gmail.com>.
Hello Carl,
Impala will not close the query until you explicitly call Close() on
the statement - you can cause queries like this to time out
automatically by setting --idle_session_timeout as a startup option
for Impala.
This behaviour is tracked at https://issues.apache.org/jira/browse/IMPALA-1575.
HTH
On 10 October 2017 at 20:04, Dong Bo 董博 <do...@fosun.com> wrote:
> In our case , the query is called from a tomcat web application with jdbc
> driver , once query end with error Memory Limit Exceeded , it is shown in
> Impala Deamon Web UI hostname:25000/queries in Waiting to be closed part .
>
> I can cancel it throw the Web UI , but is there a solution to cancel it
> automatically in jdbc without locking resources?
>
>
>
> Scenario :
>
>
>
> jdbc query timeout is set to 10 mins , issue query errors in 7 mins , seems
> jdbc is not able to cache the exception from impala , and after 10 mins ,
> the query is still in Waiting to be close .
>
> I test it is able to cancel a query with jdbc if it is still in running
> state when it is timeout . We are using connection pool to handle
> connections.
>
>
>
> Thanks
>
> Carl
>
>
>
> 发件人: Tim Armstrong [mailto:tarmstrong@cloudera.com]
> 发送时间: 2017年9月30日 2:33
> 收件人: user@impala.incubator.apache.org
> 主题: Re: Impala Memory Leak (version 2.7.0-cdh5.10.2 )
>
>
>
> I guess to elaborate further, there are two classes of scenarios:
>
> 1. the query hasn't been cancelled and closed on the coordinator so can't
> release resources.
>
> 2. the query has been closed on the coordinator but a fragment of the query
> is orphaned and still running.
> https://issues.apache.org/jira/browse/IMPALA-3633 is one possible cause of
> that that is fixed in 2.7.0.
> https://issues.apache.org/jira/browse/IMPALA-4925 is another possible case
> if the query had a limit.
>
>
>
> On Fri, Sep 29, 2017 at 11:30 AM, Tim Armstrong <ta...@cloudera.com>
> wrote:
>
> It's not a memory leak, it's a query lifecycle issue. You have fragments
> from two queries 2741ad08c43062bc:608bdb9500000000 and
> 8041bf903a5c7916:2eba50b100000000 that haven't been torn down. There are
> various ways that this could happen but I don't know based on the
> information provided.
>
> I would check if those queries are still running and if you can cancel them
> via Cloudera Manager or the Impala daemon web UI on the coordinator.
>
>
>
> On Fri, Sep 29, 2017 at 1:11 AM, Dong Bo 董博 <do...@fosun.com> wrote:
>
> Hi Experts,
>
>
>
> We are running queries on impalad version 2.7.0-cdh5.10.2 RELEASE (build
> 38c989c0330ea952133111e41965ff9af96412d3), and issued with a memory leak
> problem on the impala deamon .
>
> Once query end with error Memory limit exceeded , the allocated memory won’t
> be free until impala deamon restarted. Memory limit exceeded is in
> attachement.
>
>
>
> I find jira tickets https://issues.apache.org/jira/browse/IMPALA-3589 ,
> https://issues.apache.org/jira/browse/IMPALA-3582 with same issue as I am
> facing tagged as “Fixed” in version 2.6.0, but strange I am still facing the
> issue in a higher version.
>
>
>
> Please kindly advise , it will be very helpful.
>
>
>
> Thanks & Regards
>
> Carl Dong
>
>
>
>
>
>
>
>
>
>
答复: Impala Memory Leak (version 2.7.0-cdh5.10.2 )
Posted by Dong Bo 董博 <do...@fosun.com>.
In our case , the query is called from a tomcat web application with jdbc driver , once query end with error Memory Limit Exceeded , it is shown in Impala Deamon Web UI hostname:25000/queries in Waiting to be closed part .
I can cancel it throw the Web UI , but is there a solution to cancel it automatically in jdbc without locking resources?
Scenario :
jdbc query timeout is set to 10 mins , issue query errors in 7 mins , seems jdbc is not able to cache the exception from impala , and after 10 mins , the query is still in Waiting to be close .
I test it is able to cancel a query with jdbc if it is still in running state when it is timeout . We are using connection pool to handle connections.
Thanks
Carl
发件人: Tim Armstrong [mailto:tarmstrong@cloudera.com]
发送时间: 2017年9月30日 2:33
收件人: user@impala.incubator.apache.org
主题: Re: Impala Memory Leak (version 2.7.0-cdh5.10.2 )
I guess to elaborate further, there are two classes of scenarios:
1. the query hasn't been cancelled and closed on the coordinator so can't release resources.
2. the query has been closed on the coordinator but a fragment of the query is orphaned and still running. https://issues.apache.org/jira/browse/IMPALA-3633 is one possible cause of that that is fixed in 2.7.0. https://issues.apache.org/jira/browse/IMPALA-4925 is another possible case if the query had a limit.
On Fri, Sep 29, 2017 at 11:30 AM, Tim Armstrong <ta...@cloudera.com>> wrote:
It's not a memory leak, it's a query lifecycle issue. You have fragments from two queries 2741ad08c43062bc:608bdb9500000000 and 8041bf903a5c7916:2eba50b100000000 that haven't been torn down. There are various ways that this could happen but I don't know based on the information provided.
I would check if those queries are still running and if you can cancel them via Cloudera Manager or the Impala daemon web UI on the coordinator.
On Fri, Sep 29, 2017 at 1:11 AM, Dong Bo 董博 <do...@fosun.com>> wrote:
Hi Experts,
We are running queries on impalad version 2.7.0-cdh5.10.2 RELEASE (build 38c989c0330ea952133111e41965ff9af96412d3), and issued with a memory leak problem on the impala deamon .
Once query end with error Memory limit exceeded , the allocated memory won’t be free until impala deamon restarted. Memory limit exceeded is in attachement.
I find jira tickets https://issues.apache.org/jira/browse/IMPALA-3589 , https://issues.apache.org/jira/browse/IMPALA-3582 with same issue as I am facing tagged as “Fixed” in version 2.6.0, but strange I am still facing the issue in a higher version.
Please kindly advise , it will be very helpful.
Thanks & Regards
Carl Dong
Re: Impala Memory Leak (version 2.7.0-cdh5.10.2 )
Posted by Tim Armstrong <ta...@cloudera.com>.
I guess to elaborate further, there are two classes of scenarios:
1. the query hasn't been cancelled and closed on the coordinator so can't
release resources.
2. the query has been closed on the coordinator but a fragment of the query
is orphaned and still running.
https://issues.apache.org/jira/browse/IMPALA-3633 is one possible cause of
that that is fixed in 2.7.0.
https://issues.apache.org/jira/browse/IMPALA-4925 is another possible case
if the query had a limit.
On Fri, Sep 29, 2017 at 11:30 AM, Tim Armstrong <ta...@cloudera.com>
wrote:
> It's not a memory leak, it's a query lifecycle issue. You have fragments
> from two queries 2741ad08c43062bc:608bdb9500000000 and 8041bf903a5c7916:2eba50b100000000
> that haven't been torn down. There are various ways that this could happen
> but I don't know based on the information provided.
>
> I would check if those queries are still running and if you can cancel
> them via Cloudera Manager or the Impala daemon web UI on the coordinator.
>
> On Fri, Sep 29, 2017 at 1:11 AM, Dong Bo 董博 <do...@fosun.com> wrote:
>
>> Hi Experts,
>>
>>
>>
>> We are running queries on impalad version 2.7.0-cdh5.10.2 RELEASE (build 38c989c0330ea952133111e41965ff9af96412d3), and issued with a memory leak problem on the impala deamon .
>>
>> Once query end with error Memory limit exceeded , the allocated memory
>> won’t be free until impala deamon restarted. Memory limit exceeded is in
>> attachement.
>>
>>
>>
>> I find jira tickets https://issues.apache.org/jira/browse/IMPALA-3589 ,
>> https://issues.apache.org/jira/browse/IMPALA-3582 with same issue as I
>> am facing tagged as “Fixed” in version 2.6.0, but strange I am still facing
>> the issue in a higher version.
>>
>>
>>
>> Please kindly advise , it will be very helpful.
>>
>>
>>
>> Thanks & Regards
>>
>> Carl Dong
>>
>>
>>
>>
>>
>>
>>
>
>
Re: Impala Memory Leak (version 2.7.0-cdh5.10.2 )
Posted by Tim Armstrong <ta...@cloudera.com>.
It's not a memory leak, it's a query lifecycle issue. You have fragments
from two queries 2741ad08c43062bc:608bdb9500000000 and
8041bf903a5c7916:2eba50b100000000 that haven't been torn down. There are
various ways that this could happen but I don't know based on the
information provided.
I would check if those queries are still running and if you can cancel them
via Cloudera Manager or the Impala daemon web UI on the coordinator.
On Fri, Sep 29, 2017 at 1:11 AM, Dong Bo 董博 <do...@fosun.com> wrote:
> Hi Experts,
>
>
>
> We are running queries on impalad version 2.7.0-cdh5.10.2 RELEASE (build 38c989c0330ea952133111e41965ff9af96412d3), and issued with a memory leak problem on the impala deamon .
>
> Once query end with error Memory limit exceeded , the allocated memory
> won’t be free until impala deamon restarted. Memory limit exceeded is in
> attachement.
>
>
>
> I find jira tickets https://issues.apache.org/jira/browse/IMPALA-3589 ,
> https://issues.apache.org/jira/browse/IMPALA-3582 with same issue as I am
> facing tagged as “Fixed” in version 2.6.0, but strange I am still facing
> the issue in a higher version.
>
>
>
> Please kindly advise , it will be very helpful.
>
>
>
> Thanks & Regards
>
> Carl Dong
>
>
>
>
>
>
>