You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Fawze Abujaber <fa...@gmail.com> on 2018/05/17 17:42:51 UTC
Tracking memory used by impala once a query failed on OOM
Hi Community,
Sometimes when one of my impala queries fail, and i want to investigate and
understand how the memory was distributed at the time that query failed
between the queries and which query consumed most of the memory, i'm
getting stuck, yes i can somehow to look at the queries that ran in the
queries that running at time interval and see which one consumed most of
the query but i cann't corelate all these with the impala daeomon memory
limit and at the time when impala decides to kill the query.
I would appreciate if you have any suggestion how to investigate these
issues and if there is any metric or API that can show me at a given time
the memory used per node for the impala queries, so i can see if it related
to one specific node or a set of nodes which exceeds the memory limit
--
Take Care
Fawze Abujaber
Re: Tracking memory used by impala once a query failed on OOM
Posted by Tim Armstrong <ta...@cloudera.com>.
> —- currently we are in a process of upgrading the CDH to 5.13.0 which
includes impala 2.10.
Great! Impala 2.10 has a lot of big improvements to memory management.
Putting my Cloudera hat on for a sec, I'd recommend 5.13.3 if possible - it
has a few bug fixes.
> — for this i need to enable the heap dump for impala right?
No you shouldn't need to do anything special, it's always included in the
profile.
>— Is there any metric I can follow to understand the memory used by impala
for queries per node at any given time?
There are a lot of metrics :). It depends on what question you're trying to
answer. At the top level the "memory.total-used" metric give you a measure
of how much memory is used by the query execution layer, including various
caches and other overheads. If you're using CM, that's called
impala_memory_total_used.
There are other metrics that break that down further in various ways - e.g.
different subsystems, different resource pools, etc.
On Fri, May 18, 2018 at 10:14 AM, Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Tim,
>
> Please see my response inline.
>
>
>
> On Fri, 18 May 2018 at 19:50 Tim Armstrong <ta...@cloudera.com>
> wrote:
>
>> Hi Fawze,
>> The query profile for the failed query will generally contain an error
>> log with a dump of the memory consumed at the time that the query failed,
>> generally starting with a "memory limit exceeded" message. In some older
>> versions of Impala it only appeared in the impalad.INFO logs
>>
>
> —- currently we are in a process of upgrading the CDH to 5.13.0 which
> includes impala 2.10.
>
> Currently we are with impala 2.3
>
> — for this i need to enable the heap dump for impala right?
>
>>
>>
>> That's been incrementally improved over time, so if it's an older version
>> it may be missing some info. On recent versions of Impala I'd expect that
>> dump to include the information you need to diagnose the problem.
>>
>
> — will it provides me details in the whole impala used memory or only
> related to the query that failed with OOM?
>
> — Is there any metric I can follow to understand the memory used by impala
> for queries per node at any given time?
>
>>
>>
>> Tim
>>
>> On Thu, May 17, 2018 at 10:42 AM, Fawze Abujaber <fa...@gmail.com>
>> wrote:
>>
>>> Hi Community,
>>>
>>> Sometimes when one of my impala queries fail, and i want to investigate
>>> and understand how the memory was distributed at the time that query
>>> failed between the queries and which query consumed most of the memory,
>>> i'm getting stuck, yes i can somehow to look at the queries that ran in the
>>> queries that running at time interval and see which one consumed most of
>>> the query but i cann't corelate all these with the impala daeomon memory
>>> limit and at the time when impala decides to kill the query.
>>>
>>> I would appreciate if you have any suggestion how to investigate these
>>> issues and if there is any metric or API that can show me at a given time
>>> the memory used per node for the impala queries, so i can see if it related
>>> to one specific node or a set of nodes which exceeds the memory limit
>>>
>>> --
>>> Take Care
>>> Fawze Abujaber
>>>
>>
>> --
> Take Care
> Fawze Abujaber
>
Re: Tracking memory used by impala once a query failed on OOM
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Tim,
Please see my response inline.
On Fri, 18 May 2018 at 19:50 Tim Armstrong <ta...@cloudera.com> wrote:
> Hi Fawze,
> The query profile for the failed query will generally contain an error
> log with a dump of the memory consumed at the time that the query failed,
> generally starting with a "memory limit exceeded" message. In some older
> versions of Impala it only appeared in the impalad.INFO logs
>
—- currently we are in a process of upgrading the CDH to 5.13.0 which
includes impala 2.10.
Currently we are with impala 2.3
— for this i need to enable the heap dump for impala right?
>
>
> That's been incrementally improved over time, so if it's an older version
> it may be missing some info. On recent versions of Impala I'd expect that
> dump to include the information you need to diagnose the problem.
>
— will it provides me details in the whole impala used memory or only
related to the query that failed with OOM?
— Is there any metric I can follow to understand the memory used by impala
for queries per node at any given time?
>
>
> Tim
>
> On Thu, May 17, 2018 at 10:42 AM, Fawze Abujaber <fa...@gmail.com>
> wrote:
>
>> Hi Community,
>>
>> Sometimes when one of my impala queries fail, and i want to investigate
>> and understand how the memory was distributed at the time that query
>> failed between the queries and which query consumed most of the memory,
>> i'm getting stuck, yes i can somehow to look at the queries that ran in the
>> queries that running at time interval and see which one consumed most of
>> the query but i cann't corelate all these with the impala daeomon memory
>> limit and at the time when impala decides to kill the query.
>>
>> I would appreciate if you have any suggestion how to investigate these
>> issues and if there is any metric or API that can show me at a given time
>> the memory used per node for the impala queries, so i can see if it related
>> to one specific node or a set of nodes which exceeds the memory limit
>>
>> --
>> Take Care
>> Fawze Abujaber
>>
>
> --
Take Care
Fawze Abujaber
Re: Tracking memory used by impala once a query failed on OOM
Posted by Tim Armstrong <ta...@cloudera.com>.
Hi Fawze,
The query profile for the failed query will generally contain an error
log with a dump of the memory consumed at the time that the query failed,
generally starting with a "memory limit exceeded" message. In some older
versions of Impala it only appeared in the impalad.INFO logs.
That's been incrementally improved over time, so if it's an older version
it may be missing some info. On recent versions of Impala I'd expect that
dump to include the information you need to diagnose the problem.
Tim
On Thu, May 17, 2018 at 10:42 AM, Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Community,
>
> Sometimes when one of my impala queries fail, and i want to investigate
> and understand how the memory was distributed at the time that query
> failed between the queries and which query consumed most of the memory,
> i'm getting stuck, yes i can somehow to look at the queries that ran in the
> queries that running at time interval and see which one consumed most of
> the query but i cann't corelate all these with the impala daeomon memory
> limit and at the time when impala decides to kill the query.
>
> I would appreciate if you have any suggestion how to investigate these
> issues and if there is any metric or API that can show me at a given time
> the memory used per node for the impala queries, so i can see if it related
> to one specific node or a set of nodes which exceeds the memory limit
>
> --
> Take Care
> Fawze Abujaber
>