You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Fawze Abujaber <fa...@gmail.com> on 2018/02/22 12:48:22 UTC
Estimate peak memory VS used peak memory
Hi Guys,
In Impala version 2.10 i see a lot that the Estimate per Mode peak memory
can reach X30 times of the Per Node Peak Memory usage.
For example one of the query the estimate was 9GB while the usage was 30GB.
I have 3 questions:
1- Do you think the compute stats will help here?
2- How i can reduce this gap?
3- Does the resource pools based on the Estimate per Mode peak memory? so
if for such query i put the default max memory limit to 5GB, will this
query fail?
Re: Estimate peak memory VS used peak memory
Posted by Jeszy <je...@gmail.com>.
Looks like a very different question than the original one on this
thread, it would be better to start a new thread for a new question.
Keep in mind that you are likely to get quicker answers (from
yourself) by checking the behaviour against the documentation. If
there is a bug (sounds possible), it might have already been found,
searching around issues.apache.org will tell, along with fix version
(if any).
HTH
On 4 March 2018 at 21:35, Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Mostafa,
>
> Is this expected behavior or a BUG?
>
> On Wed, 28 Feb 2018 at 20:29 Fawze Abujaber <fa...@gmail.com> wrote:
>>
>> Hi Mostafa,
>>
>> I already rollback the version, so i don't know how to get the settings
>> and if i can get the query profile fora finished queries in the rollback
>> version.
>>
>> But for example after the upgrade we started to see the following error
>> which stopped to see after the rollback: IS NOT NULL predicate does not
>> support complex types
>>
>> IllegalStateException: org.apache.impala.common.AnalysisException: IS NOT
>> NULL predicate does not support complex types: participants IS NOT NULL
>> CAUSED BY: AnalysisException: IS NOT NULL predicate does not support complex
>> types: participants IS NOT NULL
>>
>>
>>
>> On Wed, Feb 28, 2018 at 7:56 PM, Mostafa Mokhtar <mm...@cloudera.com>
>> wrote:
>>>
>>> Can you please share the query profiles for the failures you got along
>>> with the admission control setting?
>>>
>>> Thanks
>>> Mostafa
>>>
>>> On Feb 28, 2018, at 9:28 AM, Fawze Abujaber <fa...@gmail.com> wrote:
>>>
>>> Thanks you all for your help and advises.
>>>
>>> Unfortunately i rolled back the upgrade till i understand how to control
>>> impala resources and tackle all the failures that i start to see after the
>>> upgrade.
>>>
>>>
>>>
>>> On Fri, Feb 23, 2018 at 8:22 PM, Fawze Abujaber <fa...@gmail.com>
>>> wrote:
>>>>
>>>> Hi Tim,
>>>>
>>>> My Goal is : queries that their actual memory per node exceeds more than
>>>> what i setup as a default max memory node to fail, despite i have a
>>>> different queries in the pool, in the same pool some business queries can be
>>>> simple as select count(*) and some others can have few joins.
>>>>
>>>> And i think this is the right decision and such query should be
>>>> optimized.
>>>>
>>>> And also if i'm looking in my historical queries, i can know from the
>>>> max used memory per node which queries will fail, and i think this help me
>>>> alot, but i need any other query to queued if it asked actual memory lower
>>>> than what i setup as default max memory per node for a query.
>>>>
>>>> Based on the above i'm looking for the parameters that i need to
>>>> configure.
>>>>
>>>> i don't mind how much time and how much queries will queued, in my case
>>>> i don't have any impala query that running beyond 4-5 minutes and 80% of
>>>> queries below 1 minute.
>>>>
>>>> So i don't mind to setup the queue timeout to 20 minutes and max queued
>>>> to 20-30 queries per pool.
>>>>
>>>> I want to make sure no query will fail if it not exceeding the default
>>>> memory per node that i setup.
>>>>
>>>> should i used only the default max memory per node alone? should i
>>>> combined it with the max running queries or with the memory limit of the
>>>> whole pool?
>>>>
>>>>
>>>> On Fri, Feb 23, 2018 at 8:08 PM, Tim Armstrong <ta...@cloudera.com>
>>>> wrote:
>>>>>
>>>>> I think the previous answers have been good. I wanted to add a couple
>>>>> of side notes for context since I've been doing a lot of work in this area
>>>>> of Impala. I could talk about this stuff for hours.
>>>>>
>>>>> We do have mechanisms, like spilling data to disk or reducing # of
>>>>> threads, that kick in to keep queries under the mem_limit. This has existed
>>>>> in some form since Impala 2.0, but Impala 2.10 included some architectural
>>>>> changes to make this more robust, and we have further improvements in the
>>>>> pipeline. The end goal, which we're getting much closer to, is that queries
>>>>> should reliably run to completion instead of getting killed after they are
>>>>> admitted.
>>>>>
>>>>> That support is going to enable future enhancements to memory-based
>>>>> admission control to make it easier for cluster admins like yourself to
>>>>> configure admission control. It is definitely tricky to pick a good value
>>>>> for mem_limit when pools can contain a mix of queries and I think Impala can
>>>>> do better at making these decisions automatically.
>>>>>
>>>>> - Tim
>>>>>
>>>>> On Fri, Feb 23, 2018 at 9:05 AM, Alexander Behm
>>>>> <al...@cloudera.com> wrote:
>>>>>>
>>>>>> For a given query the logic for determining the memory that will be
>>>>>> required from admission is:
>>>>>> - if the query has mem_limit use that
>>>>>> - otherwise, use memory estimates from the planner
>>>>>>
>>>>>> A query may be assigned a mem_limit by:
>>>>>> - taking the default mem_limit from the pool it was submitted to (this
>>>>>> is the recommended practice)
>>>>>> - manually setting one for the query (in case you want to override the
>>>>>> pool default for a single query)
>>>>>>
>>>>>> In that setup, the memory estimates from the planner are irrelevant
>>>>>> for admission decisions and only serve for informational purposes.
>>>>>> Please do not read too much into the memory estimates from the
>>>>>> planner. They can be totally wrong (like your 8TB example).
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 23, 2018 at 3:47 AM, Jeszy <je...@gmail.com> wrote:
>>>>>>>
>>>>>>> Again, the 8TB estimate would not be relevant if the query had a
>>>>>>> mem_limit set.
>>>>>>> I think all that we discussed is covered in the docs, but if you feel
>>>>>>> like specific parts need clarification, please file a jira.
>>>>>>>
>>>>>>> On 23 February 2018 at 11:51, Fawze Abujaber <fa...@gmail.com>
>>>>>>> wrote:
>>>>>>> > Sorry for asking many questions, but i see your answers are
>>>>>>> > closing the
>>>>>>> > gaps that i cannot find in the documentation.
>>>>>>> >
>>>>>>> > So how we can explain that there was an estimate for 8T per node
>>>>>>> > and impala
>>>>>>> > decided to submit this query?
>>>>>>> >
>>>>>>> > My goal that each query running beyond the actual limit per node to
>>>>>>> > fail (
>>>>>>> > and this is what i setup in the default memory per node per pool)
>>>>>>> > an want
>>>>>>> > all other queries to be queue and not killed, so what i understand
>>>>>>> > that i
>>>>>>> > need to setup the max queue query to unlimited and the queue
>>>>>>> > timeout to
>>>>>>> > hours.
>>>>>>> >
>>>>>>> > And in order to reach that i need to setup the default memory per
>>>>>>> > node for
>>>>>>> > each pool and setting either max concurrency or the max memory per
>>>>>>> > pool that
>>>>>>> > will help to measure the max concurrent queries that can run in
>>>>>>> > specific
>>>>>>> > pool.
>>>>>>> >
>>>>>>> > I think reaching this goal will close all my gaps.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Fri, Feb 23, 2018 at 11:49 AM, Jeszy <je...@gmail.com> wrote:
>>>>>>> >>
>>>>>>> >> > Do queuing query or not is based on the prediction which based
>>>>>>> >> > on the
>>>>>>> >> > estimate and of course the concurrency that can run in a pool.
>>>>>>> >>
>>>>>>> >> Yes, it is.
>>>>>>> >>
>>>>>>> >> > If I have memory limit per pool and memory limit per node for a
>>>>>>> >> > pool, so
>>>>>>> >> > it
>>>>>>> >> > can be used to estimate number of queries that can run
>>>>>>> >> > concurrently, is
>>>>>>> >> > this
>>>>>>> >> > also based on the prediction and not the actual use.
>>>>>>> >>
>>>>>>> >> Also on prediction.
>>>>>>> >
>>>>>>> >
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Estimate peak memory VS used peak memory
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Mostafa,
Is this expected behavior or a BUG?
On Wed, 28 Feb 2018 at 20:29 Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Mostafa,
>
> I already rollback the version, so i don't know how to get the settings
> and if i can get the query profile fora finished queries in the rollback
> version.
>
> But for example after the upgrade we started to see the following error
> which stopped to see after the rollback: IS NOT NULL predicate does not
> support complex types
>
>
> - IllegalStateException: org.apache.impala.common.AnalysisException:
> IS NOT NULL predicate does not support complex types: participants IS NOT
> NULL CAUSED BY: AnalysisException: IS NOT NULL predicate does not support
> complex types: participants IS NOT NULL
>
>
>
> On Wed, Feb 28, 2018 at 7:56 PM, Mostafa Mokhtar <mm...@cloudera.com>
> wrote:
>
>> Can you please share the query profiles for the failures you got along
>> with the admission control setting?
>>
>> Thanks
>> Mostafa
>>
>> On Feb 28, 2018, at 9:28 AM, Fawze Abujaber <fa...@gmail.com> wrote:
>>
>> Thanks you all for your help and advises.
>>
>> Unfortunately i rolled back the upgrade till i understand how to control
>> impala resources and tackle all the failures that i start to see after the
>> upgrade.
>>
>>
>>
>> On Fri, Feb 23, 2018 at 8:22 PM, Fawze Abujaber <fa...@gmail.com>
>> wrote:
>>
>>> Hi Tim,
>>>
>>> My Goal is : queries that their actual memory per node exceeds more than
>>> what i setup as a default max memory node to fail, despite i have a
>>> different queries in the pool, in the same pool some business queries can
>>> be simple as select count(*) and some others can have few joins.
>>>
>>> And i think this is the right decision and such query should be
>>> optimized.
>>>
>>> And also if i'm looking in my historical queries, i can know from the
>>> max used memory per node which queries will fail, and i think this help me
>>> alot, but i need any other query to queued if it asked actual memory lower
>>> than what i setup as default max memory per node for a query.
>>>
>>> Based on the above i'm looking for the parameters that i need to
>>> configure.
>>>
>>> i don't mind how much time and how much queries will queued, in my case
>>> i don't have any impala query that running beyond 4-5 minutes and 80% of
>>> queries below 1 minute.
>>>
>>> So i don't mind to setup the queue timeout to 20 minutes and max queued
>>> to 20-30 queries per pool.
>>>
>>> I want to make sure no query will fail if it not exceeding the default
>>> memory per node that i setup.
>>>
>>> should i used only the default max memory per node alone? should i
>>> combined it with the max running queries or with the memory limit of the
>>> whole pool?
>>>
>>>
>>> On Fri, Feb 23, 2018 at 8:08 PM, Tim Armstrong <ta...@cloudera.com>
>>> wrote:
>>>
>>>> I think the previous answers have been good. I wanted to add a couple
>>>> of side notes for context since I've been doing a lot of work in this area
>>>> of Impala. I could talk about this stuff for hours.
>>>>
>>>> We do have mechanisms, like spilling data to disk or reducing # of
>>>> threads, that kick in to keep queries under the mem_limit. This has existed
>>>> in some form since Impala 2.0, but Impala 2.10 included some architectural
>>>> changes to make this more robust, and we have further improvements in the
>>>> pipeline. The end goal, which we're getting much closer to, is that queries
>>>> should reliably run to completion instead of getting killed after they are
>>>> admitted.
>>>>
>>>> That support is going to enable future enhancements to memory-based
>>>> admission control to make it easier for cluster admins like yourself to
>>>> configure admission control. It is definitely tricky to pick a good value
>>>> for mem_limit when pools can contain a mix of queries and I think Impala
>>>> can do better at making these decisions automatically.
>>>>
>>>> - Tim
>>>>
>>>> On Fri, Feb 23, 2018 at 9:05 AM, Alexander Behm <alex.behm@cloudera.com
>>>> > wrote:
>>>>
>>>>> For a given query the logic for determining the memory that will be
>>>>> required from admission is:
>>>>> - if the query has mem_limit use that
>>>>> - otherwise, use memory estimates from the planner
>>>>>
>>>>> A query may be assigned a mem_limit by:
>>>>> - taking the default mem_limit from the pool it was submitted to (this
>>>>> is the recommended practice)
>>>>> - manually setting one for the query (in case you want to override the
>>>>> pool default for a single query)
>>>>>
>>>>> In that setup, the memory estimates from the planner are irrelevant
>>>>> for admission decisions and only serve for informational purposes.
>>>>> Please do not read too much into the memory estimates from the
>>>>> planner. They can be totally wrong (like your 8TB example).
>>>>>
>>>>>
>>>>> On Fri, Feb 23, 2018 at 3:47 AM, Jeszy <je...@gmail.com> wrote:
>>>>>
>>>>>> Again, the 8TB estimate would not be relevant if the query had a
>>>>>> mem_limit set.
>>>>>> I think all that we discussed is covered in the docs, but if you feel
>>>>>> like specific parts need clarification, please file a jira.
>>>>>>
>>>>>> On 23 February 2018 at 11:51, Fawze Abujaber <fa...@gmail.com>
>>>>>> wrote:
>>>>>> > Sorry for asking many questions, but i see your answers are
>>>>>> closing the
>>>>>> > gaps that i cannot find in the documentation.
>>>>>> >
>>>>>> > So how we can explain that there was an estimate for 8T per node
>>>>>> and impala
>>>>>> > decided to submit this query?
>>>>>> >
>>>>>> > My goal that each query running beyond the actual limit per node to
>>>>>> fail (
>>>>>> > and this is what i setup in the default memory per node per pool)
>>>>>> an want
>>>>>> > all other queries to be queue and not killed, so what i understand
>>>>>> that i
>>>>>> > need to setup the max queue query to unlimited and the queue
>>>>>> timeout to
>>>>>> > hours.
>>>>>> >
>>>>>> > And in order to reach that i need to setup the default memory per
>>>>>> node for
>>>>>> > each pool and setting either max concurrency or the max memory per
>>>>>> pool that
>>>>>> > will help to measure the max concurrent queries that can run in
>>>>>> specific
>>>>>> > pool.
>>>>>> >
>>>>>> > I think reaching this goal will close all my gaps.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Fri, Feb 23, 2018 at 11:49 AM, Jeszy <je...@gmail.com> wrote:
>>>>>> >>
>>>>>> >> > Do queuing query or not is based on the prediction which based
>>>>>> on the
>>>>>> >> > estimate and of course the concurrency that can run in a pool.
>>>>>> >>
>>>>>> >> Yes, it is.
>>>>>> >>
>>>>>> >> > If I have memory limit per pool and memory limit per node for a
>>>>>> pool, so
>>>>>> >> > it
>>>>>> >> > can be used to estimate number of queries that can run
>>>>>> concurrently, is
>>>>>> >> > this
>>>>>> >> > also based on the prediction and not the actual use.
>>>>>> >>
>>>>>> >> Also on prediction.
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Estimate peak memory VS used peak memory
Posted by Alexander Behm <al...@cloudera.com>.
Sounds like you either onboarded a new workload with nested types or an
existing workload with nested types somehow got broken in the upgrade.
That error message is quite accurate: Impala does not support IS [NOT] NULL
predicates on complex types, but it sounds like that same query used to
work before.
I'm happy to help figure out what happened, but I'll need the SQL of the
query and the CREATE TABLE/VIEW statements of the tables/views involved in
the query. Sounds like there might be a bug here.
On Wed, Feb 28, 2018 at 10:29 AM, Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Mostafa,
>
> I already rollback the version, so i don't know how to get the settings
> and if i can get the query profile fora finished queries in the rollback
> version.
>
> But for example after the upgrade we started to see the following error
> which stopped to see after the rollback: IS NOT NULL predicate does not
> support complex types
>
>
> - IllegalStateException: org.apache.impala.common.AnalysisException:
> IS NOT NULL predicate does not support complex types: participants IS NOT
> NULL CAUSED BY: AnalysisException: IS NOT NULL predicate does not support
> complex types: participants IS NOT NULL
>
>
>
> On Wed, Feb 28, 2018 at 7:56 PM, Mostafa Mokhtar <mm...@cloudera.com>
> wrote:
>
>> Can you please share the query profiles for the failures you got along
>> with the admission control setting?
>>
>> Thanks
>> Mostafa
>>
>> On Feb 28, 2018, at 9:28 AM, Fawze Abujaber <fa...@gmail.com> wrote:
>>
>> Thanks you all for your help and advises.
>>
>> Unfortunately i rolled back the upgrade till i understand how to control
>> impala resources and tackle all the failures that i start to see after the
>> upgrade.
>>
>>
>>
>> On Fri, Feb 23, 2018 at 8:22 PM, Fawze Abujaber <fa...@gmail.com>
>> wrote:
>>
>>> Hi Tim,
>>>
>>> My Goal is : queries that their actual memory per node exceeds more than
>>> what i setup as a default max memory node to fail, despite i have a
>>> different queries in the pool, in the same pool some business queries can
>>> be simple as select count(*) and some others can have few joins.
>>>
>>> And i think this is the right decision and such query should be
>>> optimized.
>>>
>>> And also if i'm looking in my historical queries, i can know from the
>>> max used memory per node which queries will fail, and i think this help me
>>> alot, but i need any other query to queued if it asked actual memory lower
>>> than what i setup as default max memory per node for a query.
>>>
>>> Based on the above i'm looking for the parameters that i need to
>>> configure.
>>>
>>> i don't mind how much time and how much queries will queued, in my case
>>> i don't have any impala query that running beyond 4-5 minutes and 80% of
>>> queries below 1 minute.
>>>
>>> So i don't mind to setup the queue timeout to 20 minutes and max queued
>>> to 20-30 queries per pool.
>>>
>>> I want to make sure no query will fail if it not exceeding the default
>>> memory per node that i setup.
>>>
>>> should i used only the default max memory per node alone? should i
>>> combined it with the max running queries or with the memory limit of the
>>> whole pool?
>>>
>>>
>>> On Fri, Feb 23, 2018 at 8:08 PM, Tim Armstrong <ta...@cloudera.com>
>>> wrote:
>>>
>>>> I think the previous answers have been good. I wanted to add a couple
>>>> of side notes for context since I've been doing a lot of work in this area
>>>> of Impala. I could talk about this stuff for hours.
>>>>
>>>> We do have mechanisms, like spilling data to disk or reducing # of
>>>> threads, that kick in to keep queries under the mem_limit. This has existed
>>>> in some form since Impala 2.0, but Impala 2.10 included some architectural
>>>> changes to make this more robust, and we have further improvements in the
>>>> pipeline. The end goal, which we're getting much closer to, is that queries
>>>> should reliably run to completion instead of getting killed after they are
>>>> admitted.
>>>>
>>>> That support is going to enable future enhancements to memory-based
>>>> admission control to make it easier for cluster admins like yourself to
>>>> configure admission control. It is definitely tricky to pick a good value
>>>> for mem_limit when pools can contain a mix of queries and I think Impala
>>>> can do better at making these decisions automatically.
>>>>
>>>> - Tim
>>>>
>>>> On Fri, Feb 23, 2018 at 9:05 AM, Alexander Behm <alex.behm@cloudera.com
>>>> > wrote:
>>>>
>>>>> For a given query the logic for determining the memory that will be
>>>>> required from admission is:
>>>>> - if the query has mem_limit use that
>>>>> - otherwise, use memory estimates from the planner
>>>>>
>>>>> A query may be assigned a mem_limit by:
>>>>> - taking the default mem_limit from the pool it was submitted to (this
>>>>> is the recommended practice)
>>>>> - manually setting one for the query (in case you want to override the
>>>>> pool default for a single query)
>>>>>
>>>>> In that setup, the memory estimates from the planner are irrelevant
>>>>> for admission decisions and only serve for informational purposes.
>>>>> Please do not read too much into the memory estimates from the
>>>>> planner. They can be totally wrong (like your 8TB example).
>>>>>
>>>>>
>>>>> On Fri, Feb 23, 2018 at 3:47 AM, Jeszy <je...@gmail.com> wrote:
>>>>>
>>>>>> Again, the 8TB estimate would not be relevant if the query had a
>>>>>> mem_limit set.
>>>>>> I think all that we discussed is covered in the docs, but if you feel
>>>>>> like specific parts need clarification, please file a jira.
>>>>>>
>>>>>> On 23 February 2018 at 11:51, Fawze Abujaber <fa...@gmail.com>
>>>>>> wrote:
>>>>>> > Sorry for asking many questions, but i see your answers are
>>>>>> closing the
>>>>>> > gaps that i cannot find in the documentation.
>>>>>> >
>>>>>> > So how we can explain that there was an estimate for 8T per node
>>>>>> and impala
>>>>>> > decided to submit this query?
>>>>>> >
>>>>>> > My goal that each query running beyond the actual limit per node to
>>>>>> fail (
>>>>>> > and this is what i setup in the default memory per node per pool)
>>>>>> an want
>>>>>> > all other queries to be queue and not killed, so what i understand
>>>>>> that i
>>>>>> > need to setup the max queue query to unlimited and the queue
>>>>>> timeout to
>>>>>> > hours.
>>>>>> >
>>>>>> > And in order to reach that i need to setup the default memory per
>>>>>> node for
>>>>>> > each pool and setting either max concurrency or the max memory per
>>>>>> pool that
>>>>>> > will help to measure the max concurrent queries that can run in
>>>>>> specific
>>>>>> > pool.
>>>>>> >
>>>>>> > I think reaching this goal will close all my gaps.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Fri, Feb 23, 2018 at 11:49 AM, Jeszy <je...@gmail.com> wrote:
>>>>>> >>
>>>>>> >> > Do queuing query or not is based on the prediction which based
>>>>>> on the
>>>>>> >> > estimate and of course the concurrency that can run in a pool.
>>>>>> >>
>>>>>> >> Yes, it is.
>>>>>> >>
>>>>>> >> > If I have memory limit per pool and memory limit per node for a
>>>>>> pool, so
>>>>>> >> > it
>>>>>> >> > can be used to estimate number of queries that can run
>>>>>> concurrently, is
>>>>>> >> > this
>>>>>> >> > also based on the prediction and not the actual use.
>>>>>> >>
>>>>>> >> Also on prediction.
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Estimate peak memory VS used peak memory
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Mostafa,
I already rollback the version, so i don't know how to get the settings and
if i can get the query profile fora finished queries in the rollback
version.
But for example after the upgrade we started to see the following error
which stopped to see after the rollback: IS NOT NULL predicate does not
support complex types
- IllegalStateException: org.apache.impala.common.AnalysisException: IS
NOT NULL predicate does not support complex types: participants IS NOT NULL
CAUSED BY: AnalysisException: IS NOT NULL predicate does not support
complex types: participants IS NOT NULL
On Wed, Feb 28, 2018 at 7:56 PM, Mostafa Mokhtar <mm...@cloudera.com>
wrote:
> Can you please share the query profiles for the failures you got along
> with the admission control setting?
>
> Thanks
> Mostafa
>
> On Feb 28, 2018, at 9:28 AM, Fawze Abujaber <fa...@gmail.com> wrote:
>
> Thanks you all for your help and advises.
>
> Unfortunately i rolled back the upgrade till i understand how to control
> impala resources and tackle all the failures that i start to see after the
> upgrade.
>
>
>
> On Fri, Feb 23, 2018 at 8:22 PM, Fawze Abujaber <fa...@gmail.com> wrote:
>
>> Hi Tim,
>>
>> My Goal is : queries that their actual memory per node exceeds more than
>> what i setup as a default max memory node to fail, despite i have a
>> different queries in the pool, in the same pool some business queries can
>> be simple as select count(*) and some others can have few joins.
>>
>> And i think this is the right decision and such query should be optimized.
>>
>> And also if i'm looking in my historical queries, i can know from the max
>> used memory per node which queries will fail, and i think this help me
>> alot, but i need any other query to queued if it asked actual memory lower
>> than what i setup as default max memory per node for a query.
>>
>> Based on the above i'm looking for the parameters that i need to
>> configure.
>>
>> i don't mind how much time and how much queries will queued, in my case i
>> don't have any impala query that running beyond 4-5 minutes and 80% of
>> queries below 1 minute.
>>
>> So i don't mind to setup the queue timeout to 20 minutes and max queued
>> to 20-30 queries per pool.
>>
>> I want to make sure no query will fail if it not exceeding the default
>> memory per node that i setup.
>>
>> should i used only the default max memory per node alone? should i
>> combined it with the max running queries or with the memory limit of the
>> whole pool?
>>
>>
>> On Fri, Feb 23, 2018 at 8:08 PM, Tim Armstrong <ta...@cloudera.com>
>> wrote:
>>
>>> I think the previous answers have been good. I wanted to add a couple of
>>> side notes for context since I've been doing a lot of work in this area of
>>> Impala. I could talk about this stuff for hours.
>>>
>>> We do have mechanisms, like spilling data to disk or reducing # of
>>> threads, that kick in to keep queries under the mem_limit. This has existed
>>> in some form since Impala 2.0, but Impala 2.10 included some architectural
>>> changes to make this more robust, and we have further improvements in the
>>> pipeline. The end goal, which we're getting much closer to, is that queries
>>> should reliably run to completion instead of getting killed after they are
>>> admitted.
>>>
>>> That support is going to enable future enhancements to memory-based
>>> admission control to make it easier for cluster admins like yourself to
>>> configure admission control. It is definitely tricky to pick a good value
>>> for mem_limit when pools can contain a mix of queries and I think Impala
>>> can do better at making these decisions automatically.
>>>
>>> - Tim
>>>
>>> On Fri, Feb 23, 2018 at 9:05 AM, Alexander Behm <al...@cloudera.com>
>>> wrote:
>>>
>>>> For a given query the logic for determining the memory that will be
>>>> required from admission is:
>>>> - if the query has mem_limit use that
>>>> - otherwise, use memory estimates from the planner
>>>>
>>>> A query may be assigned a mem_limit by:
>>>> - taking the default mem_limit from the pool it was submitted to (this
>>>> is the recommended practice)
>>>> - manually setting one for the query (in case you want to override the
>>>> pool default for a single query)
>>>>
>>>> In that setup, the memory estimates from the planner are irrelevant for
>>>> admission decisions and only serve for informational purposes.
>>>> Please do not read too much into the memory estimates from the planner.
>>>> They can be totally wrong (like your 8TB example).
>>>>
>>>>
>>>> On Fri, Feb 23, 2018 at 3:47 AM, Jeszy <je...@gmail.com> wrote:
>>>>
>>>>> Again, the 8TB estimate would not be relevant if the query had a
>>>>> mem_limit set.
>>>>> I think all that we discussed is covered in the docs, but if you feel
>>>>> like specific parts need clarification, please file a jira.
>>>>>
>>>>> On 23 February 2018 at 11:51, Fawze Abujaber <fa...@gmail.com>
>>>>> wrote:
>>>>> > Sorry for asking many questions, but i see your answers are closing
>>>>> the
>>>>> > gaps that i cannot find in the documentation.
>>>>> >
>>>>> > So how we can explain that there was an estimate for 8T per node and
>>>>> impala
>>>>> > decided to submit this query?
>>>>> >
>>>>> > My goal that each query running beyond the actual limit per node to
>>>>> fail (
>>>>> > and this is what i setup in the default memory per node per pool) an
>>>>> want
>>>>> > all other queries to be queue and not killed, so what i understand
>>>>> that i
>>>>> > need to setup the max queue query to unlimited and the queue timeout
>>>>> to
>>>>> > hours.
>>>>> >
>>>>> > And in order to reach that i need to setup the default memory per
>>>>> node for
>>>>> > each pool and setting either max concurrency or the max memory per
>>>>> pool that
>>>>> > will help to measure the max concurrent queries that can run in
>>>>> specific
>>>>> > pool.
>>>>> >
>>>>> > I think reaching this goal will close all my gaps.
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Fri, Feb 23, 2018 at 11:49 AM, Jeszy <je...@gmail.com> wrote:
>>>>> >>
>>>>> >> > Do queuing query or not is based on the prediction which based on
>>>>> the
>>>>> >> > estimate and of course the concurrency that can run in a pool.
>>>>> >>
>>>>> >> Yes, it is.
>>>>> >>
>>>>> >> > If I have memory limit per pool and memory limit per node for a
>>>>> pool, so
>>>>> >> > it
>>>>> >> > can be used to estimate number of queries that can run
>>>>> concurrently, is
>>>>> >> > this
>>>>> >> > also based on the prediction and not the actual use.
>>>>> >>
>>>>> >> Also on prediction.
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Estimate peak memory VS used peak memory
Posted by Mostafa Mokhtar <mm...@cloudera.com>.
Can you please share the query profiles for the failures you got along with the admission control setting?
Thanks
Mostafa
> On Feb 28, 2018, at 9:28 AM, Fawze Abujaber <fa...@gmail.com> wrote:
>
> Thanks you all for your help and advises.
>
> Unfortunately i rolled back the upgrade till i understand how to control impala resources and tackle all the failures that i start to see after the upgrade.
>
>
>
>> On Fri, Feb 23, 2018 at 8:22 PM, Fawze Abujaber <fa...@gmail.com> wrote:
>> Hi Tim,
>>
>> My Goal is : queries that their actual memory per node exceeds more than what i setup as a default max memory node to fail, despite i have a different queries in the pool, in the same pool some business queries can be simple as select count(*) and some others can have few joins.
>>
>> And i think this is the right decision and such query should be optimized.
>>
>> And also if i'm looking in my historical queries, i can know from the max used memory per node which queries will fail, and i think this help me alot, but i need any other query to queued if it asked actual memory lower than what i setup as default max memory per node for a query.
>>
>> Based on the above i'm looking for the parameters that i need to configure.
>>
>> i don't mind how much time and how much queries will queued, in my case i don't have any impala query that running beyond 4-5 minutes and 80% of queries below 1 minute.
>>
>> So i don't mind to setup the queue timeout to 20 minutes and max queued to 20-30 queries per pool.
>>
>> I want to make sure no query will fail if it not exceeding the default memory per node that i setup.
>>
>> should i used only the default max memory per node alone? should i combined it with the max running queries or with the memory limit of the whole pool?
>>
>>
>>> On Fri, Feb 23, 2018 at 8:08 PM, Tim Armstrong <ta...@cloudera.com> wrote:
>>> I think the previous answers have been good. I wanted to add a couple of side notes for context since I've been doing a lot of work in this area of Impala. I could talk about this stuff for hours.
>>>
>>> We do have mechanisms, like spilling data to disk or reducing # of threads, that kick in to keep queries under the mem_limit. This has existed in some form since Impala 2.0, but Impala 2.10 included some architectural changes to make this more robust, and we have further improvements in the pipeline. The end goal, which we're getting much closer to, is that queries should reliably run to completion instead of getting killed after they are admitted.
>>>
>>> That support is going to enable future enhancements to memory-based admission control to make it easier for cluster admins like yourself to configure admission control. It is definitely tricky to pick a good value for mem_limit when pools can contain a mix of queries and I think Impala can do better at making these decisions automatically.
>>>
>>> - Tim
>>>
>>>> On Fri, Feb 23, 2018 at 9:05 AM, Alexander Behm <al...@cloudera.com> wrote:
>>>> For a given query the logic for determining the memory that will be required from admission is:
>>>> - if the query has mem_limit use that
>>>> - otherwise, use memory estimates from the planner
>>>>
>>>> A query may be assigned a mem_limit by:
>>>> - taking the default mem_limit from the pool it was submitted to (this is the recommended practice)
>>>> - manually setting one for the query (in case you want to override the pool default for a single query)
>>>>
>>>> In that setup, the memory estimates from the planner are irrelevant for admission decisions and only serve for informational purposes.
>>>> Please do not read too much into the memory estimates from the planner. They can be totally wrong (like your 8TB example).
>>>>
>>>>
>>>>> On Fri, Feb 23, 2018 at 3:47 AM, Jeszy <je...@gmail.com> wrote:
>>>>> Again, the 8TB estimate would not be relevant if the query had a mem_limit set.
>>>>> I think all that we discussed is covered in the docs, but if you feel
>>>>> like specific parts need clarification, please file a jira.
>>>>>
>>>>> On 23 February 2018 at 11:51, Fawze Abujaber <fa...@gmail.com> wrote:
>>>>> > Sorry for asking many questions, but i see your answers are closing the
>>>>> > gaps that i cannot find in the documentation.
>>>>> >
>>>>> > So how we can explain that there was an estimate for 8T per node and impala
>>>>> > decided to submit this query?
>>>>> >
>>>>> > My goal that each query running beyond the actual limit per node to fail (
>>>>> > and this is what i setup in the default memory per node per pool) an want
>>>>> > all other queries to be queue and not killed, so what i understand that i
>>>>> > need to setup the max queue query to unlimited and the queue timeout to
>>>>> > hours.
>>>>> >
>>>>> > And in order to reach that i need to setup the default memory per node for
>>>>> > each pool and setting either max concurrency or the max memory per pool that
>>>>> > will help to measure the max concurrent queries that can run in specific
>>>>> > pool.
>>>>> >
>>>>> > I think reaching this goal will close all my gaps.
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Fri, Feb 23, 2018 at 11:49 AM, Jeszy <je...@gmail.com> wrote:
>>>>> >>
>>>>> >> > Do queuing query or not is based on the prediction which based on the
>>>>> >> > estimate and of course the concurrency that can run in a pool.
>>>>> >>
>>>>> >> Yes, it is.
>>>>> >>
>>>>> >> > If I have memory limit per pool and memory limit per node for a pool, so
>>>>> >> > it
>>>>> >> > can be used to estimate number of queries that can run concurrently, is
>>>>> >> > this
>>>>> >> > also based on the prediction and not the actual use.
>>>>> >>
>>>>> >> Also on prediction.
>>>>> >
>>>>> >
>>>>
>>>
>>
>
Re: Estimate peak memory VS used peak memory
Posted by Fawze Abujaber <fa...@gmail.com>.
Thanks you all for your help and advises.
Unfortunately i rolled back the upgrade till i understand how to control
impala resources and tackle all the failures that i start to see after the
upgrade.
On Fri, Feb 23, 2018 at 8:22 PM, Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Tim,
>
> My Goal is : queries that their actual memory per node exceeds more than
> what i setup as a default max memory node to fail, despite i have a
> different queries in the pool, in the same pool some business queries can
> be simple as select count(*) and some others can have few joins.
>
> And i think this is the right decision and such query should be optimized.
>
> And also if i'm looking in my historical queries, i can know from the max
> used memory per node which queries will fail, and i think this help me
> alot, but i need any other query to queued if it asked actual memory lower
> than what i setup as default max memory per node for a query.
>
> Based on the above i'm looking for the parameters that i need to configure.
>
> i don't mind how much time and how much queries will queued, in my case i
> don't have any impala query that running beyond 4-5 minutes and 80% of
> queries below 1 minute.
>
> So i don't mind to setup the queue timeout to 20 minutes and max queued to
> 20-30 queries per pool.
>
> I want to make sure no query will fail if it not exceeding the default
> memory per node that i setup.
>
> should i used only the default max memory per node alone? should i
> combined it with the max running queries or with the memory limit of the
> whole pool?
>
>
> On Fri, Feb 23, 2018 at 8:08 PM, Tim Armstrong <ta...@cloudera.com>
> wrote:
>
>> I think the previous answers have been good. I wanted to add a couple of
>> side notes for context since I've been doing a lot of work in this area of
>> Impala. I could talk about this stuff for hours.
>>
>> We do have mechanisms, like spilling data to disk or reducing # of
>> threads, that kick in to keep queries under the mem_limit. This has existed
>> in some form since Impala 2.0, but Impala 2.10 included some architectural
>> changes to make this more robust, and we have further improvements in the
>> pipeline. The end goal, which we're getting much closer to, is that queries
>> should reliably run to completion instead of getting killed after they are
>> admitted.
>>
>> That support is going to enable future enhancements to memory-based
>> admission control to make it easier for cluster admins like yourself to
>> configure admission control. It is definitely tricky to pick a good value
>> for mem_limit when pools can contain a mix of queries and I think Impala
>> can do better at making these decisions automatically.
>>
>> - Tim
>>
>> On Fri, Feb 23, 2018 at 9:05 AM, Alexander Behm <al...@cloudera.com>
>> wrote:
>>
>>> For a given query the logic for determining the memory that will be
>>> required from admission is:
>>> - if the query has mem_limit use that
>>> - otherwise, use memory estimates from the planner
>>>
>>> A query may be assigned a mem_limit by:
>>> - taking the default mem_limit from the pool it was submitted to (this
>>> is the recommended practice)
>>> - manually setting one for the query (in case you want to override the
>>> pool default for a single query)
>>>
>>> In that setup, the memory estimates from the planner are irrelevant for
>>> admission decisions and only serve for informational purposes.
>>> Please do not read too much into the memory estimates from the planner.
>>> They can be totally wrong (like your 8TB example).
>>>
>>>
>>> On Fri, Feb 23, 2018 at 3:47 AM, Jeszy <je...@gmail.com> wrote:
>>>
>>>> Again, the 8TB estimate would not be relevant if the query had a
>>>> mem_limit set.
>>>> I think all that we discussed is covered in the docs, but if you feel
>>>> like specific parts need clarification, please file a jira.
>>>>
>>>> On 23 February 2018 at 11:51, Fawze Abujaber <fa...@gmail.com> wrote:
>>>> > Sorry for asking many questions, but i see your answers are closing
>>>> the
>>>> > gaps that i cannot find in the documentation.
>>>> >
>>>> > So how we can explain that there was an estimate for 8T per node and
>>>> impala
>>>> > decided to submit this query?
>>>> >
>>>> > My goal that each query running beyond the actual limit per node to
>>>> fail (
>>>> > and this is what i setup in the default memory per node per pool) an
>>>> want
>>>> > all other queries to be queue and not killed, so what i understand
>>>> that i
>>>> > need to setup the max queue query to unlimited and the queue timeout
>>>> to
>>>> > hours.
>>>> >
>>>> > And in order to reach that i need to setup the default memory per
>>>> node for
>>>> > each pool and setting either max concurrency or the max memory per
>>>> pool that
>>>> > will help to measure the max concurrent queries that can run in
>>>> specific
>>>> > pool.
>>>> >
>>>> > I think reaching this goal will close all my gaps.
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Feb 23, 2018 at 11:49 AM, Jeszy <je...@gmail.com> wrote:
>>>> >>
>>>> >> > Do queuing query or not is based on the prediction which based on
>>>> the
>>>> >> > estimate and of course the concurrency that can run in a pool.
>>>> >>
>>>> >> Yes, it is.
>>>> >>
>>>> >> > If I have memory limit per pool and memory limit per node for a
>>>> pool, so
>>>> >> > it
>>>> >> > can be used to estimate number of queries that can run
>>>> concurrently, is
>>>> >> > this
>>>> >> > also based on the prediction and not the actual use.
>>>> >>
>>>> >> Also on prediction.
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
Re: Estimate peak memory VS used peak memory
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Tim,
My Goal is : queries that their actual memory per node exceeds more than
what i setup as a default max memory node to fail, despite i have a
different queries in the pool, in the same pool some business queries can
be simple as select count(*) and some others can have few joins.
And i think this is the right decision and such query should be optimized.
And also if i'm looking in my historical queries, i can know from the max
used memory per node which queries will fail, and i think this help me
alot, but i need any other query to queued if it asked actual memory lower
than what i setup as default max memory per node for a query.
Based on the above i'm looking for the parameters that i need to configure.
i don't mind how much time and how much queries will queued, in my case i
don't have any impala query that running beyond 4-5 minutes and 80% of
queries below 1 minute.
So i don't mind to setup the queue timeout to 20 minutes and max queued to
20-30 queries per pool.
I want to make sure no query will fail if it not exceeding the default
memory per node that i setup.
should i used only the default max memory per node alone? should i combined
it with the max running queries or with the memory limit of the whole pool?
On Fri, Feb 23, 2018 at 8:08 PM, Tim Armstrong <ta...@cloudera.com>
wrote:
> I think the previous answers have been good. I wanted to add a couple of
> side notes for context since I've been doing a lot of work in this area of
> Impala. I could talk about this stuff for hours.
>
> We do have mechanisms, like spilling data to disk or reducing # of
> threads, that kick in to keep queries under the mem_limit. This has existed
> in some form since Impala 2.0, but Impala 2.10 included some architectural
> changes to make this more robust, and we have further improvements in the
> pipeline. The end goal, which we're getting much closer to, is that queries
> should reliably run to completion instead of getting killed after they are
> admitted.
>
> That support is going to enable future enhancements to memory-based
> admission control to make it easier for cluster admins like yourself to
> configure admission control. It is definitely tricky to pick a good value
> for mem_limit when pools can contain a mix of queries and I think Impala
> can do better at making these decisions automatically.
>
> - Tim
>
> On Fri, Feb 23, 2018 at 9:05 AM, Alexander Behm <al...@cloudera.com>
> wrote:
>
>> For a given query the logic for determining the memory that will be
>> required from admission is:
>> - if the query has mem_limit use that
>> - otherwise, use memory estimates from the planner
>>
>> A query may be assigned a mem_limit by:
>> - taking the default mem_limit from the pool it was submitted to (this is
>> the recommended practice)
>> - manually setting one for the query (in case you want to override the
>> pool default for a single query)
>>
>> In that setup, the memory estimates from the planner are irrelevant for
>> admission decisions and only serve for informational purposes.
>> Please do not read too much into the memory estimates from the planner.
>> They can be totally wrong (like your 8TB example).
>>
>>
>> On Fri, Feb 23, 2018 at 3:47 AM, Jeszy <je...@gmail.com> wrote:
>>
>>> Again, the 8TB estimate would not be relevant if the query had a
>>> mem_limit set.
>>> I think all that we discussed is covered in the docs, but if you feel
>>> like specific parts need clarification, please file a jira.
>>>
>>> On 23 February 2018 at 11:51, Fawze Abujaber <fa...@gmail.com> wrote:
>>> > Sorry for asking many questions, but i see your answers are closing
>>> the
>>> > gaps that i cannot find in the documentation.
>>> >
>>> > So how we can explain that there was an estimate for 8T per node and
>>> impala
>>> > decided to submit this query?
>>> >
>>> > My goal that each query running beyond the actual limit per node to
>>> fail (
>>> > and this is what i setup in the default memory per node per pool) an
>>> want
>>> > all other queries to be queue and not killed, so what i understand
>>> that i
>>> > need to setup the max queue query to unlimited and the queue timeout to
>>> > hours.
>>> >
>>> > And in order to reach that i need to setup the default memory per node
>>> for
>>> > each pool and setting either max concurrency or the max memory per
>>> pool that
>>> > will help to measure the max concurrent queries that can run in
>>> specific
>>> > pool.
>>> >
>>> > I think reaching this goal will close all my gaps.
>>> >
>>> >
>>> >
>>> > On Fri, Feb 23, 2018 at 11:49 AM, Jeszy <je...@gmail.com> wrote:
>>> >>
>>> >> > Do queuing query or not is based on the prediction which based on
>>> the
>>> >> > estimate and of course the concurrency that can run in a pool.
>>> >>
>>> >> Yes, it is.
>>> >>
>>> >> > If I have memory limit per pool and memory limit per node for a
>>> pool, so
>>> >> > it
>>> >> > can be used to estimate number of queries that can run
>>> concurrently, is
>>> >> > this
>>> >> > also based on the prediction and not the actual use.
>>> >>
>>> >> Also on prediction.
>>> >
>>> >
>>>
>>
>>
>
Re: Estimate peak memory VS used peak memory
Posted by Tim Armstrong <ta...@cloudera.com>.
I think the previous answers have been good. I wanted to add a couple of
side notes for context since I've been doing a lot of work in this area of
Impala. I could talk about this stuff for hours.
We do have mechanisms, like spilling data to disk or reducing # of threads,
that kick in to keep queries under the mem_limit. This has existed in some
form since Impala 2.0, but Impala 2.10 included some architectural changes
to make this more robust, and we have further improvements in the pipeline.
The end goal, which we're getting much closer to, is that queries should
reliably run to completion instead of getting killed after they are
admitted.
That support is going to enable future enhancements to memory-based
admission control to make it easier for cluster admins like yourself to
configure admission control. It is definitely tricky to pick a good value
for mem_limit when pools can contain a mix of queries and I think Impala
can do better at making these decisions automatically.
- Tim
On Fri, Feb 23, 2018 at 9:05 AM, Alexander Behm <al...@cloudera.com>
wrote:
> For a given query the logic for determining the memory that will be
> required from admission is:
> - if the query has mem_limit use that
> - otherwise, use memory estimates from the planner
>
> A query may be assigned a mem_limit by:
> - taking the default mem_limit from the pool it was submitted to (this is
> the recommended practice)
> - manually setting one for the query (in case you want to override the
> pool default for a single query)
>
> In that setup, the memory estimates from the planner are irrelevant for
> admission decisions and only serve for informational purposes.
> Please do not read too much into the memory estimates from the planner.
> They can be totally wrong (like your 8TB example).
>
>
> On Fri, Feb 23, 2018 at 3:47 AM, Jeszy <je...@gmail.com> wrote:
>
>> Again, the 8TB estimate would not be relevant if the query had a
>> mem_limit set.
>> I think all that we discussed is covered in the docs, but if you feel
>> like specific parts need clarification, please file a jira.
>>
>> On 23 February 2018 at 11:51, Fawze Abujaber <fa...@gmail.com> wrote:
>> > Sorry for asking many questions, but i see your answers are closing the
>> > gaps that i cannot find in the documentation.
>> >
>> > So how we can explain that there was an estimate for 8T per node and
>> impala
>> > decided to submit this query?
>> >
>> > My goal that each query running beyond the actual limit per node to
>> fail (
>> > and this is what i setup in the default memory per node per pool) an
>> want
>> > all other queries to be queue and not killed, so what i understand that
>> i
>> > need to setup the max queue query to unlimited and the queue timeout to
>> > hours.
>> >
>> > And in order to reach that i need to setup the default memory per node
>> for
>> > each pool and setting either max concurrency or the max memory per pool
>> that
>> > will help to measure the max concurrent queries that can run in specific
>> > pool.
>> >
>> > I think reaching this goal will close all my gaps.
>> >
>> >
>> >
>> > On Fri, Feb 23, 2018 at 11:49 AM, Jeszy <je...@gmail.com> wrote:
>> >>
>> >> > Do queuing query or not is based on the prediction which based on the
>> >> > estimate and of course the concurrency that can run in a pool.
>> >>
>> >> Yes, it is.
>> >>
>> >> > If I have memory limit per pool and memory limit per node for a
>> pool, so
>> >> > it
>> >> > can be used to estimate number of queries that can run concurrently,
>> is
>> >> > this
>> >> > also based on the prediction and not the actual use.
>> >>
>> >> Also on prediction.
>> >
>> >
>>
>
>
Re: Estimate peak memory VS used peak memory
Posted by Alexander Behm <al...@cloudera.com>.
For a given query the logic for determining the memory that will be
required from admission is:
- if the query has mem_limit use that
- otherwise, use memory estimates from the planner
A query may be assigned a mem_limit by:
- taking the default mem_limit from the pool it was submitted to (this is
the recommended practice)
- manually setting one for the query (in case you want to override the pool
default for a single query)
In that setup, the memory estimates from the planner are irrelevant for
admission decisions and only serve for informational purposes.
Please do not read too much into the memory estimates from the planner.
They can be totally wrong (like your 8TB example).
On Fri, Feb 23, 2018 at 3:47 AM, Jeszy <je...@gmail.com> wrote:
> Again, the 8TB estimate would not be relevant if the query had a mem_limit
> set.
> I think all that we discussed is covered in the docs, but if you feel
> like specific parts need clarification, please file a jira.
>
> On 23 February 2018 at 11:51, Fawze Abujaber <fa...@gmail.com> wrote:
> > Sorry for asking many questions, but i see your answers are closing the
> > gaps that i cannot find in the documentation.
> >
> > So how we can explain that there was an estimate for 8T per node and
> impala
> > decided to submit this query?
> >
> > My goal that each query running beyond the actual limit per node to fail
> (
> > and this is what i setup in the default memory per node per pool) an want
> > all other queries to be queue and not killed, so what i understand that i
> > need to setup the max queue query to unlimited and the queue timeout to
> > hours.
> >
> > And in order to reach that i need to setup the default memory per node
> for
> > each pool and setting either max concurrency or the max memory per pool
> that
> > will help to measure the max concurrent queries that can run in specific
> > pool.
> >
> > I think reaching this goal will close all my gaps.
> >
> >
> >
> > On Fri, Feb 23, 2018 at 11:49 AM, Jeszy <je...@gmail.com> wrote:
> >>
> >> > Do queuing query or not is based on the prediction which based on the
> >> > estimate and of course the concurrency that can run in a pool.
> >>
> >> Yes, it is.
> >>
> >> > If I have memory limit per pool and memory limit per node for a pool,
> so
> >> > it
> >> > can be used to estimate number of queries that can run concurrently,
> is
> >> > this
> >> > also based on the prediction and not the actual use.
> >>
> >> Also on prediction.
> >
> >
>
Re: Estimate peak memory VS used peak memory
Posted by Jeszy <je...@gmail.com>.
Again, the 8TB estimate would not be relevant if the query had a mem_limit set.
I think all that we discussed is covered in the docs, but if you feel
like specific parts need clarification, please file a jira.
On 23 February 2018 at 11:51, Fawze Abujaber <fa...@gmail.com> wrote:
> Sorry for asking many questions, but i see your answers are closing the
> gaps that i cannot find in the documentation.
>
> So how we can explain that there was an estimate for 8T per node and impala
> decided to submit this query?
>
> My goal that each query running beyond the actual limit per node to fail (
> and this is what i setup in the default memory per node per pool) an want
> all other queries to be queue and not killed, so what i understand that i
> need to setup the max queue query to unlimited and the queue timeout to
> hours.
>
> And in order to reach that i need to setup the default memory per node for
> each pool and setting either max concurrency or the max memory per pool that
> will help to measure the max concurrent queries that can run in specific
> pool.
>
> I think reaching this goal will close all my gaps.
>
>
>
> On Fri, Feb 23, 2018 at 11:49 AM, Jeszy <je...@gmail.com> wrote:
>>
>> > Do queuing query or not is based on the prediction which based on the
>> > estimate and of course the concurrency that can run in a pool.
>>
>> Yes, it is.
>>
>> > If I have memory limit per pool and memory limit per node for a pool, so
>> > it
>> > can be used to estimate number of queries that can run concurrently, is
>> > this
>> > also based on the prediction and not the actual use.
>>
>> Also on prediction.
>
>
Re: Estimate peak memory VS used peak memory
Posted by Fawze Abujaber <fa...@gmail.com>.
Sorry for asking many questions, but i see your answers are closing the
gaps that i cannot find in the documentation.
So how we can explain that there was an estimate for 8T per node and impala
decided to submit this query?
My goal that each query running beyond the actual limit per node to fail (
and this is what i setup in the default memory per node per pool) an want
all other queries to be queue and not killed, so what i understand that i
need to setup the max queue query to unlimited and the queue timeout to
hours.
And in order to reach that i need to setup the default memory per node for
each pool and setting either max concurrency or the max memory per pool
that will help to measure the max concurrent queries that can run in
specific pool.
I think reaching this goal will close all my gaps.
On Fri, Feb 23, 2018 at 11:49 AM, Jeszy <je...@gmail.com> wrote:
> > Do queuing query or not is based on the prediction which based on the
> > estimate and of course the concurrency that can run in a pool.
>
> Yes, it is.
>
> > If I have memory limit per pool and memory limit per node for a pool, so
> it
> > can be used to estimate number of queries that can run concurrently, is
> this
> > also based on the prediction and not the actual use.
>
> Also on prediction.
>
Re: Estimate peak memory VS used peak memory
Posted by Jeszy <je...@gmail.com>.
> Do queuing query or not is based on the prediction which based on the
> estimate and of course the concurrency that can run in a pool.
Yes, it is.
> If I have memory limit per pool and memory limit per node for a pool, so it
> can be used to estimate number of queries that can run concurrently, is this
> also based on the prediction and not the actual use.
Also on prediction.
Re: Estimate peak memory VS used peak memory
Posted by Fawze Abujaber <fa...@gmail.com>.
Do queuing query or not is based on the prediction which based on the
estimate and of course the concurrency that can run in a pool.
If I have memory limit per pool and memory limit per node for a pool, so it
can be used to estimate number of queries that can run concurrently, is
this also based on the prediction and not the actual use.
I believe with the time and trends we can learn a lot from the admission
control but trying to minimize the impact the business at this period of
learning ...
On Fri, 23 Feb 2018 at 11:26 Jeszy <je...@gmail.com> wrote:
> Queries will be killed based on actual usage (peak memory usage across
> hosts), so the 200mb is the interesting value in your example.
>
> Compare the pool's available memory to the query's mem requirement
> (based on estimate or mem_limit, as discussed) to predict admission.
>
> On 23 February 2018 at 10:06, Fawze Abujaber <fa...@gmail.com> wrote:
> > Thanks jezy for your detailed response.
> >
> > Yes I read the documentation.
> >
> > Let simplify my question:
> >
> > I have pools set up with memory limit per node and concurrency.
> >
> > If I’m looking on the historical impala queries that I have and the
> metrics
> > I have per query, on which metrics I can understand that impala will kill
> > the query, for example if I have a query with estimate of 2GB and the
> used
> > per node is 200mb, what is the default memory values that i need to
> setup so
> > the query will not fail.
> >
> > The second one is the distribution between pools, if one query is running
> > which metrics o have to look into to know if I submit a query it fail or
> > not.
> >
> > On Fri, 23 Feb 2018 at 10:48 Jeszy <je...@gmail.com> wrote:
> >>
> >> Hey Fawze,
> >>
> >> Answers inline.
> >>
> >> On 23 February 2018 at 01:23, Fawze Abujaber <fa...@gmail.com> wrote:
> >> > There is no option in the admission control to setup memory limit per
> >> > query,
> >> > the memory limit is per pool and there is a default memory per node
> for
> >> > query.
> >>
> >> per node for query memory limit multiplied by number of nodes gives
> >> you a per query memory limit. I agree its confusing that the
> >> configurations mix and match between per-node and aggregated values.
> >> In this case there's a good reason though, as a single node running
> >> out of memory will lead to query failure, meaning that in addition to
> >> total memory used, distribution of memory usage between hosts also
> >> matters.
> >>
> >> > I have hundreds of impala queries and more add hoc queries, making a
> >> > pool
> >> > for each query is not a visible solution.
> >> >
> >> > still waiting to understand how the estimate per node related to the
> >> > default
> >> > memory per node I set up per pool, is it used in the decision of
> queuing
> >> > and
> >> > killing the query? and if this is true how it was not kill a query
> that
> >> > was
> >> > estimated it needs 8.2TB memory per node.
> >> >
> >> > Understanding on which parameters impala decides to kill a query can
> >> > help
> >> > understand to define and divide the memory between the pools.
> >>
> >> If you set mem_limit at any level (service level, pool level, or query
> >> level), it will be used for admission control purposes instead of
> >> estimates. So a 8.2TB estimate would not be a problem, if impala can
> >> reserve mem_limit amount on each host, it will start running the
> >> query.
> >>
> >> > Passing memory limit per query manually is also not visible and such
> >> > settings not needs admission control.
> >> >
> >> > I have support pool that runs ad hoc query and I can not ask them to
> use
> >> > memory limit per query, and I have analytics pool which is fully
> >> > business
> >> > and I can rely on admission control if it extremely in accurate.
> >>
> >> It's a bit tricky to use memory-based admission control with
> >> non-trivial ad hoc queries. For simple ad-hoc queries, you can try to
> >> come up with a 'good enough' mem_limit, or omit mem_limit and trust
> >> impala's estimations. You can check the estimated vs. actual values
> >> for a representative set of ad hoc queries to see what would work in
> >> your case. I've found that people tend to go with a large enough
> >> mem_limit for the ad hoc pool.
> >>
> >> > Can someone explain me exactly which recommended setting to use per
> pool
> >> > and
> >> > which of them rely on impala memory estimates?
> >>
> >> The documentation of admission control
> >> (https://impala.apache.org/docs/build/html/topics/impala_admission.html
> )
> >> gives you a good view on how stuff works, but you will have to figure
> >> out how to use these features for your specific use case. That said,
> >> when using memory based admission control, it is best practice to
> >> always use a mem_limit due to potential inaccuracy of estimates as
> >> well as potential variance of estimates between Impala releases. Keep
> >> in mind that you can opt to set a default mem_limit for one pool and
> >> leave it unset for another.
> >>
> >> > So my conclusion right now to avoid using any settings rely on the
> >> > estimates
> >> > and to ignore the estimates when I want to evaluate query.
> >>
> >> Sounds good.
> >>
> >> > @mostafa, since my issue with all the query, I think the profile will
> >> > not
> >> > help me to solve such huge issue.
> >> >
> >> > I’m planning to move a way from Vertica and rely on impala as a sql
> >> > engine
> >> > and now fully confused how I can do this if I can’t use the admission
> >> > control.
> >> >
> >> > Last think, is it recommend to use the impala admission control?
> >>
> >> Yes. Admission control can take a while to understand, but if done
> >> right, it works.
> >>
> >> HTH
> >>
> >> > On Fri, 23 Feb 2018 at 1:56 Alexander Behm <al...@cloudera.com>
> >> > wrote:
> >> >>
> >> >> The planner memory estimates are conservative and sometimes extremely
> >> >> inaccurate. In their current form, they are rarely appropriate for
> >> >> admission
> >> >> decisions.
> >> >>
> >> >> The recommended practice for memory-based admission control it to
> set a
> >> >> mem_limit for every query. You can make this easier by setting up
> >> >> different
> >> >> pools with different mem_limits, e.g. a small/medium/big queries pool
> >> >> or
> >> >> similar.
> >> >>
> >> >> On Thu, Feb 22, 2018 at 3:00 PM, Mostafa Mokhtar
> >> >> <mm...@cloudera.com>
> >> >> wrote:
> >> >>>
> >> >>> It is recommended to set a per query memory limit as part of
> admission
> >> >>> and not rely on estimates as they are sometimes inaccurate.
> >> >>> Can you please include the full query profile?
> >> >>>
> >> >>>
> >> >>> On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <fawzeaj@gmail.com
> >
> >> >>> wrote:
> >> >>>>
> >> >>>> Hi Mostafa,
> >> >>>>
> >> >>>> It's not a specific query, almost all the query has such differene
> >> >>>> between the 2 values.
> >> >>>>
> >> >>>> I can see even queries showing the estimate per node is 8.2 Tib
> >> >>>>
> >> >>>> User: psanalytics
> >> >>>>
> >> >>>> Database: default
> >> >>>>
> >> >>>> Query Type: QUERY
> >> >>>> Coordinator: slpr-dhc014.lpdomain.com
> >> >>>>
> >> >>>> Duration: 6.48s
> >> >>>>
> >> >>>> Rows Produced: 708
> >> >>>> Estimated per Node Peak Memory: 8.2 TiB
> >> >>>>
> >> >>>> Per Node Peak Memory Usage: 1.1 GiB
> >> >>>>
> >> >>>> Pool: root.impanalytics
> >> >>>> Threads: CPU Time: 20.1m
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> How you can explain this behavior, and for sure i don't have 8.2
> Tib
> >> >>>> memory per node to give neither you.
> >> >>>>
> >> >>>> Can you please explain me how i should treat Estimated per Node
> Peak
> >> >>>> Memory and if it used by impala for the resource pool and admission
> >> >>>> control
> >> >>>> and what is the relation of this value to the default memory per
> node
> >> >>>> that i
> >> >>>> setup for each resource pool?
> >> >>>>
> >> >>>> Below is part of one of the queries profile which the estimate per
> >> >>>> node
> >> >>>> was ~ @GB and the used was 200MB per node.
> >> >>>>
> >> >>>>
> >> >>>> Instance 744de1b6228736fa:b54bfaa70000000f
> >> >>>> (host=slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child:
> >> >>>> 1s292ms, %
> >> >>>> non-child: 88.82%)
> >> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB
> >> >>>> 9:1/191.64
> >> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB
> >> >>>> MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB
> >> >>>> ThreadUsage(500.000ms): 1, 1, 1
> >> >>>> - AverageThreadTokens: 1.00
> >> >>>> - BloomFilterBytes: 0
> >> >>>> - PeakMemoryUsage: 7.17 MB (7521751)
> >> >>>> - PeakReservation: 0
> >> >>>> - PeakUsedReservation: 0
> >> >>>> - PerHostPeakMemUsage: 106.53 MB (111709581)
> >> >>>> - RowsProduced: 32.83K (32826)
> >> >>>> - TotalNetworkReceiveTime: 0.000ns
> >> >>>> - TotalNetworkSendTime: 1s297ms
> >> >>>> - TotalStorageWaitTime: 234.356ms
> >> >>>> - TotalThreadsInvoluntaryContextSwitches: 66 (66)
> >> >>>> - TotalThreadsTotalWallClockTime: 1s715ms
> >> >>>> - TotalThreadsSysTime: 5.998ms
> >> >>>> - TotalThreadsUserTime: 124.975ms
> >> >>>> - TotalThreadsVoluntaryContextSwitches: 303 (303)
> >> >>>> Fragment Instance Lifecycle Timings:
> >> >>>> - ExecTime: 1s394ms
> >> >>>> - ExecTreeExecTime: 67.115ms
> >> >>>> - OpenTime: 32.795ms
> >> >>>> - ExecTreeOpenTime: 73.243us
> >> >>>> - PrepareTime: 27.602ms
> >> >>>> - ExecTreePrepareTime: 243.141us
> >> >>>> DataStreamSender (dst_id=11):(Total: 38.747ms, non-child:
> >> >>>> 38.747ms, % non-child: 100.00%)
> >> >>>> - BytesSent: 39.71 MB (41643000)
> >> >>>> - NetworkThroughput(*): 1.97 GB/sec
> >> >>>> - OverallThroughput: 1.00 GB/sec
> >> >>>> - PeakMemoryUsage: 59.38 KB (60800)
> >> >>>> - RowsReturned: 32.83K (32826)
> >> >>>> - SerializeBatchTime: 16.860ms
> >> >>>> - TransmitDataRPCTime: 19.698ms
> >> >>>> - UncompressedRowBatchSize: 77.58 MB (81350840)
> >> >>>> CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child:
> >> >>>> 100.00%)
> >> >>>> - CodegenTime: 1.299ms
> >> >>>> - CompileTime: 10.672ms
> >> >>>> - LoadTime: 0.000ns
> >> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
> >> >>>> - NumFunctions: 16 (16)
> >> >>>> - NumInstructions: 250 (250)
> >> >>>> - OptimizationTime: 21.023ms
> >> >>>> - PeakMemoryUsage: 125.00 KB (128000)
> >> >>>> - PrepareTime: 24.116ms
> >> >>>> SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms,
> %
> >> >>>> non-child: 17.85%)
> >> >>>> - PeakMemoryUsage: 627.94 KB (643015)
> >> >>>> - RowsReturned: 32.77K (32768)
> >> >>>> - RowsReturnedRate: 486.81 K/sec
> >> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child:
> >> >>>> 25.197ms, % non-child: 74.11%)
> >> >>>> - BuildRows: 0 (0)
> >> >>>> - BuildTime: 0.000ns
> >> >>>> - PeakMemoryUsage: 24.00 KB (24576)
> >> >>>> - ProbeRows: 32.83K (32826)
> >> >>>> - ProbeTime: 0.000ns
> >> >>>> - RowsReturned: 16.80M (16795311)
> >> >>>> - RowsReturnedRate: 493.99 M/sec
> >> >>>> Nested Loop Join Builder:
> >> >>>> - PeakMemoryUsage: 8.00 KB (8192)
> >> >>>> SINGULAR_ROW_SRC_NODE (id=7):
> >> >>>> - PeakMemoryUsage: 0
> >> >>>> - RowsReturned: 0 (0)
> >> >>>> - RowsReturnedRate: 0
> >> >>>> UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, %
> >> >>>> non-child: 100.00%)
> >> >>>> - AvgCollectionSize: 1.00
> >> >>>> - MaxCollectionSize: 1 (1)
> >> >>>> - MinCollectionSize: 1 (1)
> >> >>>> - NumCollections: 32.83K (32826)
> >> >>>> - PeakMemoryUsage: 0
> >> >>>> - RowsReturned: 1 (1)
> >> >>>> - RowsReturnedRate: 113.00 /sec
> >> >>>> HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child:
> 21.299ms,
> >> >>>> %
> >> >>>> non-child: 100.00%)
> >> >>>> Hdfs split stats (<volume id>:<# splits>/<split
> lengths>):
> >> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB
> >> >>>> 9:1/191.64
> >> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB
> >> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11
> >> >>>> out
> >> >>>> of 11
> >> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
> 3:0%
> >> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> >> >>>> 16:0%
> >> >>>> File Formats: PARQUET/SNAPPY:66
> >> >>>> BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB
> >> >>>> - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ;
> >> >>>> Max:
> >> >>>> 29.275ms ; Number of samples: 11)
> >> >>>> - AverageHdfsReadThreadConcurrency: 0.00
> >> >>>> - AverageScannerThreadConcurrency: 0.00
> >> >>>> - BytesRead: 2.26 MB (2373738)
> >> >>>> - BytesReadDataNodeCache: 0
> >> >>>> - BytesReadLocal: 2.26 MB (2373738)
> >> >>>> - BytesReadRemoteUnexpected: 0
> >> >>>> - BytesReadShortCircuit: 2.26 MB (2373738)
> >> >>>> - CachedFileHandlesHitCount: 0 (0)
> >> >>>> - CachedFileHandlesMissCount: 77 (77)
> >> >>>> - DecompressionTime: 1.543ms
> >> >>>> - MaxCompressedTextFileLength: 0
> >> >>>> - NumColumns: 6 (6)
> >> >>>> - NumDictFilteredRowGroups: 0 (0)
> >> >>>> - NumDisksAccessed: 8 (8)
> >> >>>> - NumRowGroups: 11 (11)
> >> >>>> - NumScannerThreadsStarted: 11 (11)
> >> >>>> - NumScannersWithNoReads: 0 (0)
> >> >>>> - NumStatsFilteredRowGroups: 0 (0)
> >> >>>> - PeakMemoryUsage: 6.51 MB (6821203)
> >> >>>> - PerReadThreadRawHdfsThroughput: 764.51 MB/sec
> >> >>>> - RemoteScanRanges: 0 (0)
> >> >>>> - RowBatchQueueGetWaitTime: 18.825ms
> >> >>>> - RowBatchQueuePutWaitTime: 0.000ns
> >> >>>> - RowsRead: 73.46K (73464)
> >> >>>> - RowsReturned: 32.83K (32826)
> >> >>>> - RowsReturnedRate: 1.54 M/sec
> >> >>>> - ScanRangesComplete: 11 (11)
> >> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> >> >>>> - ScannerThreadsTotalWallClockTime: 287.702ms
> >> >>>> - MaterializeTupleTime(*): 11.921ms
> >> >>>> - ScannerThreadsSysTime: 999.000us
> >> >>>> - ScannerThreadsUserTime: 26.990ms
> >> >>>> - ScannerThreadsVoluntaryContextSwitches: 232 (232)
> >> >>>> - TotalRawHdfsReadTime(*): 2.961ms
> >> >>>> - TotalReadThroughput: 1.51 MB/sec
> >> >>>> Instance 744de1b6228736fa:b54bfaa70000000d
> >> >>>> (host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child:
> >> >>>> 1s254ms, %
> >> >>>> non-child: 86.28%)
> >> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
> >> >>>> MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB
> >> >>>> ThreadUsage(500.000ms): 1, 1, 1
> >> >>>> - AverageThreadTokens: 1.00
> >> >>>> - BloomFilterBytes: 0
> >> >>>> - PeakMemoryUsage: 5.86 MB (6145964)
> >> >>>> - PeakReservation: 0
> >> >>>> - PeakUsedReservation: 0
> >> >>>> - PerHostPeakMemUsage: 106.66 MB (111842766)
> >> >>>> - RowsProduced: 30.34K (30340)
> >> >>>> - TotalNetworkReceiveTime: 0.000ns
> >> >>>> - TotalNetworkSendTime: 1s256ms
> >> >>>> - TotalStorageWaitTime: 106.530ms
> >> >>>> - TotalThreadsInvoluntaryContextSwitches: 177 (177)
> >> >>>> - TotalThreadsTotalWallClockTime: 1s557ms
> >> >>>> - TotalThreadsSysTime: 6.998ms
> >> >>>> - TotalThreadsUserTime: 143.976ms
> >> >>>> - TotalThreadsVoluntaryContextSwitches: 136 (136)
> >> >>>> Fragment Instance Lifecycle Timings:
> >> >>>> - ExecTime: 1s390ms
> >> >>>> - ExecTreeExecTime: 91.500ms
> >> >>>> - OpenTime: 34.085ms
> >> >>>> - ExecTreeOpenTime: 81.436us
> >> >>>> - PrepareTime: 28.524ms
> >> >>>> - ExecTreePrepareTime: 217.949us
> >> >>>> DataStreamSender (dst_id=11):(Total: 48.674ms, non-child:
> >> >>>> 48.674ms, % non-child: 100.00%)
> >> >>>> - BytesSent: 36.61 MB (38388760)
> >> >>>> - NetworkThroughput(*): 1.51 GB/sec
> >> >>>> - OverallThroughput: 752.15 MB/sec
> >> >>>> - PeakMemoryUsage: 59.38 KB (60800)
> >> >>>> - RowsReturned: 30.34K (30340)
> >> >>>> - SerializeBatchTime: 23.132ms
> >> >>>> - TransmitDataRPCTime: 23.609ms
> >> >>>> - UncompressedRowBatchSize: 71.70 MB (75187520)
> >> >>>> CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child:
> >> >>>> 100.00%)
> >> >>>> - CodegenTime: 1.382ms
> >> >>>> - CompileTime: 10.435ms
> >> >>>> - LoadTime: 0.000ns
> >> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
> >> >>>> - NumFunctions: 16 (16)
> >> >>>> - NumInstructions: 250 (250)
> >> >>>> - OptimizationTime: 22.325ms
> >> >>>> - PeakMemoryUsage: 125.00 KB (128000)
> >> >>>> - PrepareTime: 25.459ms
> >> >>>> SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms,
> %
> >> >>>> non-child: 22.90%)
> >> >>>> - PeakMemoryUsage: 699.55 KB (716335)
> >> >>>> - RowsReturned: 29.70K (29696)
> >> >>>> - RowsReturnedRate: 323.80 K/sec
> >> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child:
> >> >>>> 40.263ms, % non-child: 74.95%)
> >> >>>> - BuildRows: 0 (0)
> >> >>>> - BuildTime: 0.000ns
> >> >>>> - PeakMemoryUsage: 24.00 KB (24576)
> >> >>>> - ProbeRows: 30.34K (30340)
> >> >>>> - ProbeTime: 0.000ns
> >> >>>> - RowsReturned: 15.43M (15426890)
> >> >>>> - RowsReturnedRate: 287.16 M/sec
> >> >>>> Nested Loop Join Builder:
> >> >>>> - PeakMemoryUsage: 8.00 KB (8192)
> >> >>>> SINGULAR_ROW_SRC_NODE (id=7):
> >> >>>> - PeakMemoryUsage: 0
> >> >>>> - RowsReturned: 0 (0)
> >> >>>> - RowsReturnedRate: 0
> >> >>>> UNNEST_NODE (id=8):(Total: 13.460ms, non-child:
> 13.460ms, %
> >> >>>> non-child: 100.00%)
> >> >>>> - AvgCollectionSize: 1.00
> >> >>>> - MaxCollectionSize: 1 (1)
> >> >>>> - MinCollectionSize: 1 (1)
> >> >>>> - NumCollections: 30.34K (30340)
> >> >>>> - PeakMemoryUsage: 0
> >> >>>> - RowsReturned: 1 (1)
> >> >>>> - RowsReturnedRate: 74.00 /sec
> >> >>>> HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child:
> 16.990ms,
> >> >>>> %
> >> >>>> non-child: 100.00%)
> >> >>>> Hdfs split stats (<volume id>:<# splits>/<split
> lengths>):
> >> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
> >> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5
> out
> >> >>>> of
> >> >>>> 5
> >> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
> 3:0%
> >> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> >> >>>> 16:0%
> >> >>>> File Formats: PARQUET/SNAPPY:30
> >> >>>> BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB
> >> >>>> - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ;
> >> >>>> Max:
> >> >>>> 28.687ms ; Number of samples: 5)
> >> >>>> - AverageHdfsReadThreadConcurrency: 0.00
> >> >>>> - AverageScannerThreadConcurrency: 0.00
> >> >>>> - BytesRead: 1.56 MB (1630546)
> >> >>>> - BytesReadDataNodeCache: 0
> >> >>>> - BytesReadLocal: 1.56 MB (1630546)
> >> >>>> - BytesReadRemoteUnexpected: 0
> >> >>>> - BytesReadShortCircuit: 1.56 MB (1630546)
> >> >>>> - CachedFileHandlesHitCount: 0 (0)
> >> >>>> - CachedFileHandlesMissCount: 35 (35)
> >> >>>> - DecompressionTime: 1.267ms
> >> >>>> - MaxCompressedTextFileLength: 0
> >> >>>> - NumColumns: 6 (6)
> >> >>>> - NumDictFilteredRowGroups: 0 (0)
> >> >>>> - NumDisksAccessed: 4 (4)
> >> >>>> - NumRowGroups: 5 (5)
> >> >>>> - NumScannerThreadsStarted: 5 (5)
> >> >>>> - NumScannersWithNoReads: 0 (0)
> >> >>>> - NumStatsFilteredRowGroups: 0 (0)
> >> >>>> - PeakMemoryUsage: 5.59 MB (5859290)
> >> >>>> - PerReadThreadRawHdfsThroughput: 920.32 MB/sec
> >> >>>> - RemoteScanRanges: 0 (0)
> >> >>>> - RowBatchQueueGetWaitTime: 15.335ms
> >> >>>> - RowBatchQueuePutWaitTime: 0.000ns
> >> >>>> - RowsRead: 64.69K (64692)
> >> >>>> - RowsReturned: 30.34K (30340)
> >> >>>> - RowsReturnedRate: 1.79 M/sec
> >> >>>> - ScanRangesComplete: 5 (5)
> >> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> >> >>>> - ScannerThreadsTotalWallClockTime: 132.548ms
> >> >>>> - MaterializeTupleTime(*): 10.585ms
> >> >>>> - ScannerThreadsSysTime: 999.000us
> >> >>>> - ScannerThreadsUserTime: 18.995ms
> >> >>>> - ScannerThreadsVoluntaryContextSwitches: 79 (79)
> >> >>>> - TotalRawHdfsReadTime(*): 1.689ms
> >> >>>> - TotalReadThroughput: 1.04 MB/sec
> >> >>>> Instance 744de1b6228736fa:b54bfaa700000002
> >> >>>> (host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child:
> >> >>>> 1s255ms, %
> >> >>>> non-child: 86.41%)
> >> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
> >> >>>> MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB
> >> >>>> ThreadUsage(500.000ms): 1, 1, 1
> >> >>>> - AverageThreadTokens: 1.00
> >> >>>> - BloomFilterBytes: 0
> >> >>>> - PeakMemoryUsage: 6.17 MB (6473729)
> >> >>>> - PeakReservation: 0
> >> >>>> - PeakUsedReservation: 0
> >> >>>> - PerHostPeakMemUsage: 107.32 MB (112537640)
> >> >>>> - RowsProduced: 29.97K (29969)
> >> >>>> - TotalNetworkReceiveTime: 0.000ns
> >> >>>> - TotalNetworkSendTime: 1s260ms
> >> >>>> - TotalStorageWaitTime: 130.815ms
> >> >>>> - TotalThreadsInvoluntaryContextSwitches: 15 (15)
> >> >>>> - TotalThreadsTotalWallClockTime: 1s586ms
> >> >>>> - TotalThreadsSysTime: 11.995ms
> >> >>>> - TotalThreadsUserTime: 146.975ms
> >> >>>> - TotalThreadsVoluntaryContextSwitches: 197 (197)
> >> >>>> Fragment Instance Lifecycle Timings:
> >> >>>> - ExecTime: 1s354ms
> >> >>>> - ExecTreeExecTime: 67.791ms
> >> >>>> - OpenTime: 52.012ms
> >> >>>> - ExecTreeOpenTime: 101.232us
> >> >>>> - PrepareTime: 45.946ms
> >> >>>> - ExecTreePrepareTime: 309.644us
> >> >>>> DataStreamSender (dst_id=11):(Total: 38.995ms, non-child:
> >> >>>> 38.995ms, % non-child: 100.00%)
> >> >>>> - BytesSent: 35.95 MB (37697260)
> >> >>>> - NetworkThroughput(*): 2.14 GB/sec
> >> >>>> - OverallThroughput: 921.92 MB/sec
> >> >>>> - PeakMemoryUsage: 59.38 KB (60800)
> >> >>>> - RowsReturned: 29.97K (29969)
> >> >>>> - SerializeBatchTime: 18.055ms
> >> >>>> - TransmitDataRPCTime: 16.428ms
> >> >>>> - UncompressedRowBatchSize: 70.83 MB (74269800)
> >> >>>> CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child:
> >> >>>> 100.00%)
> >> >>>> - CodegenTime: 2.224ms
> >> >>>> - CompileTime: 15.517ms
> >> >>>> - LoadTime: 0.000ns
> >> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
> >> >>>> - NumFunctions: 16 (16)
> >> >>>> - NumInstructions: 250 (250)
> >> >>>> - OptimizationTime: 35.099ms
> >> >>>> - PeakMemoryUsage: 125.00 KB (128000)
> >> >>>> - PrepareTime: 38.847ms
> >> >>>> SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms,
> %
> >> >>>> non-child: 18.53%)
> >> >>>> - PeakMemoryUsage: 636.17 KB (651439)
> >> >>>> - RowsReturned: 29.70K (29696)
> >> >>>> - RowsReturnedRate: 435.89 K/sec
> >> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child:
> >> >>>> 27.325ms, % non-child: 77.31%)
> >> >>>> - BuildRows: 0 (0)
> >> >>>> - BuildTime: 0.000ns
> >> >>>> - PeakMemoryUsage: 24.00 KB (24576)
> >> >>>> - ProbeRows: 29.97K (29969)
> >> >>>> - ProbeTime: 0.000ns
> >> >>>> - RowsReturned: 15.26M (15256601)
> >> >>>> - RowsReturnedRate: 431.62 M/sec
> >> >>>> Nested Loop Join Builder:
> >> >>>> - PeakMemoryUsage: 8.00 KB (8192)
> >> >>>> SINGULAR_ROW_SRC_NODE (id=7):
> >> >>>> - PeakMemoryUsage: 0
> >> >>>> - RowsReturned: 0 (0)
> >> >>>> - RowsReturnedRate: 0
> >> >>>> UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, %
> >> >>>> non-child: 100.00%)
> >> >>>> - AvgCollectionSize: 1.00
> >> >>>> - MaxCollectionSize: 1 (1)
> >> >>>> - MinCollectionSize: 1 (1)
> >> >>>> - NumCollections: 29.97K (29969)
> >> >>>> - PeakMemoryUsage: 0
> >> >>>> - RowsReturned: 1 (1)
> >> >>>> - RowsReturnedRate: 124.00 /sec
> >> >>>> HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child:
> 20.153ms,
> >> >>>> %
> >> >>>> non-child: 100.00%)
> >> >>>> Hdfs split stats (<volume id>:<# splits>/<split
> lengths>):
> >> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
> >> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7
> out
> >> >>>> of
> >> >>>> 7
> >> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
> 3:0%
> >> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> >> >>>> 16:0%
> >> >>>> File Formats: PARQUET/SNAPPY:42
> >> >>>> BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB
> >> >>>> - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ;
> >> >>>> Max:
> >> >>>> 10.464ms ; Number of samples: 7)
> >> >>>> - AverageHdfsReadThreadConcurrency: 0.00
> >> >>>> - AverageScannerThreadConcurrency: 0.00
> >> >>>> - BytesRead: 1.75 MB (1833560)
> >> >>>> - BytesReadDataNodeCache: 0
> >> >>>> - BytesReadLocal: 1.75 MB (1833560)
> >> >>>> - BytesReadRemoteUnexpected: 0
> >> >>>> - BytesReadShortCircuit: 1.75 MB (1833560)
> >> >>>> - CachedFileHandlesHitCount: 0 (0)
> >> >>>> - CachedFileHandlesMissCount: 49 (49)
> >> >>>> - DecompressionTime: 1.758ms
> >> >>>> - MaxCompressedTextFileLength: 0
> >> >>>> - NumColumns: 6 (6)
> >> >>>> - NumDictFilteredRowGroups: 0 (0)
> >> >>>> - NumDisksAccessed: 5 (5)
> >> >>>> - NumRowGroups: 7 (7)
> >> >>>> - NumScannerThreadsStarted: 7 (7)
> >> >>>> - NumScannersWithNoReads: 0 (0)
> >> >>>> - NumStatsFilteredRowGroups: 0 (0)
> >> >>>> - PeakMemoryUsage: 5.82 MB (6098991)
> >> >>>> - PerReadThreadRawHdfsThroughput: 621.09 MB/sec
> >> >>>> - RemoteScanRanges: 0 (0)
> >> >>>> - RowBatchQueueGetWaitTime: 17.938ms
> >> >>>> - RowBatchQueuePutWaitTime: 0.000ns
> >> >>>> - RowsRead: 65.17K (65166)
> >> >>>> - RowsReturned: 29.97K (29969)
> >> >>>> - RowsReturnedRate: 1.49 M/sec
> >> >>>> - ScanRangesComplete: 7 (7)
> >> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> >> >>>> - ScannerThreadsTotalWallClockTime: 179.698ms
> >> >>>> - MaterializeTupleTime(*): 13.922ms
> >> >>>> - ScannerThreadsSysTime: 2.997ms
> >> >>>> - ScannerThreadsUserTime: 29.993ms
> >> >>>> - ScannerThreadsVoluntaryContextSwitches: 130 (130)
> >> >>>> - TotalRawHdfsReadTime(*): 2.815ms
> >> >>>> - TotalReadThroughput: 1.17 MB/sec
> >> >>>> Instance 744de1b6228736fa:b54bfaa700000008
> >> >>>> (host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child:
> >> >>>> 1s305ms, %
> >> >>>> non-child: 90.01%)
> >> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
> >> >>>> MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB
> >> >>>> ThreadUsage(500.000ms): 1, 1, 1
> >> >>>> - AverageThreadTokens: 1.00
> >> >>>> - BloomFilterBytes: 0
> >> >>>> - PeakMemoryUsage: 6.69 MB (7017157)
> >> >>>> - PeakReservation: 0
> >> >>>> - PeakUsedReservation: 0
> >> >>>> - PerHostPeakMemUsage: 106.39 MB (111556990)
> >> >>>> - RowsProduced: 42.68K (42681)
> >> >>>> - TotalNetworkReceiveTime: 0.000ns
> >> >>>> - TotalNetworkSendTime: 1s315ms
> >> >>>> - TotalStorageWaitTime: 201.411ms
> >> >>>> - TotalThreadsInvoluntaryContextSwitches: 43 (43)
> >> >>>> - TotalThreadsTotalWallClockTime: 1s658ms
> >> >>>> - TotalThreadsSysTime: 6.997ms
> >> >>>> - TotalThreadsUserTime: 108.980ms
> >> >>>> - TotalThreadsVoluntaryContextSwitches: 179 (179)
> >> >>>> Fragment Instance Lifecycle Timings:
> >> >>>> - ExecTime: 1s411ms
> >> >>>> - ExecTreeExecTime: 68.017ms
> >> >>>> - OpenTime: 19.935ms
> >> >>>> - ExecTreeOpenTime: 48.883us
> >> >>>> - PrepareTime: 19.160ms
> >> >>>> - ExecTreePrepareTime: 134.576us
> >> >>>> DataStreamSender (dst_id=11):(Total: 40.671ms, non-child:
> >> >>>> 40.671ms, % non-child: 100.00%)
> >> >>>> - BytesSent: 51.36 MB (53856680)
> >> >>>> - NetworkThroughput(*): 2.44 GB/sec
> >> >>>> - OverallThroughput: 1.23 GB/sec
> >> >>>> - PeakMemoryUsage: 59.38 KB (60800)
> >> >>>> - RowsReturned: 42.68K (42681)
> >> >>>> - SerializeBatchTime: 18.299ms
> >> >>>> - TransmitDataRPCTime: 20.591ms
> >> >>>> - UncompressedRowBatchSize: 100.88 MB (105782860)
> >> >>>> CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child:
> >> >>>> 100.00%)
> >> >>>> - CodegenTime: 828.865us
> >> >>>> - CompileTime: 6.170ms
> >> >>>> - LoadTime: 0.000ns
> >> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
> >> >>>> - NumFunctions: 16 (16)
> >> >>>> - NumInstructions: 250 (250)
> >> >>>> - OptimizationTime: 12.988ms
> >> >>>> - PeakMemoryUsage: 125.00 KB (128000)
> >> >>>> - PrepareTime: 16.608ms
> >> >>>> SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms,
> %
> >> >>>> non-child: 18.27%)
> >> >>>> - PeakMemoryUsage: 689.95 KB (706507)
> >> >>>> - RowsReturned: 41.98K (41984)
> >> >>>> - RowsReturnedRate: 616.99 K/sec
> >> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child:
> >> >>>> 29.422ms, % non-child: 73.86%)
> >> >>>> - BuildRows: 0 (0)
> >> >>>> - BuildTime: 0.000ns
> >> >>>> - PeakMemoryUsage: 24.00 KB (24576)
> >> >>>> - ProbeRows: 42.68K (42681)
> >> >>>> - ProbeTime: 0.000ns
> >> >>>> - RowsReturned: 21.76M (21760053)
> >> >>>> - RowsReturnedRate: 546.22 M/sec
> >> >>>> Nested Loop Join Builder:
> >> >>>> - PeakMemoryUsage: 8.00 KB (8192)
> >> >>>> SINGULAR_ROW_SRC_NODE (id=7):
> >> >>>> - PeakMemoryUsage: 0
> >> >>>> - RowsReturned: 0 (0)
> >> >>>> - RowsReturnedRate: 0
> >> >>>> UNNEST_NODE (id=8):(Total: 10.415ms, non-child:
> 10.415ms, %
> >> >>>> non-child: 100.00%)
> >> >>>> - AvgCollectionSize: 1.00
> >> >>>> - MaxCollectionSize: 1 (1)
> >> >>>> - MinCollectionSize: 1 (1)
> >> >>>> - NumCollections: 42.68K (42681)
> >> >>>> - PeakMemoryUsage: 0
> >> >>>> - RowsReturned: 1 (1)
> >> >>>> - RowsReturnedRate: 96.00 /sec
> >> >>>> HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child:
> 15.774ms,
> >> >>>> %
> >> >>>> non-child: 100.00%)
> >> >>>> Hdfs split stats (<volume id>:<# splits>/<split
> lengths>):
> >> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
> >> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6
> out
> >> >>>> of
> >> >>>> 6
> >> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
> 3:0%
> >> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> >> >>>> 16:0%
> >> >>>> 17:0%
> >> >>>> File Formats: PARQUET/SNAPPY:36
> >> >>>> BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB
> >> >>>> - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ;
> >> >>>> Max:
> >> >>>> 42.732ms ; Number of samples: 6)
> >> >>>> - AverageHdfsReadThreadConcurrency: 0.00
> >> >>>> - AverageScannerThreadConcurrency: 0.00
> >> >>>> - BytesRead: 2.08 MB (2186159)
> >> >>>> - BytesReadDataNodeCache: 0
> >> >>>> - BytesReadLocal: 2.08 MB (2186159)
> >> >>>> - BytesReadRemoteUnexpected: 0
> >> >>>> - BytesReadShortCircuit: 2.08 MB (2186159)
> >> >>>> - CachedFileHandlesHitCount: 0 (0)
> >> >>>> - CachedFileHandlesMissCount: 42 (42)
> >> >>>> - DecompressionTime: 1.480ms
> >> >>>> - MaxCompressedTextFileLength: 0
> >> >>>> - NumColumns: 6 (6)
> >> >>>> - NumDictFilteredRowGroups: 0 (0)
> >> >>>> - NumDisksAccessed: 4 (4)
> >> >>>> - NumRowGroups: 6 (6)
> >> >>>> - NumScannerThreadsStarted: 6 (6)
> >> >>>> - NumScannersWithNoReads: 0 (0)
> >> >>>> - NumStatsFilteredRowGroups: 0 (0)
> >> >>>> - PeakMemoryUsage: 6.42 MB (6730467)
> >> >>>> - PerReadThreadRawHdfsThroughput: 77.02 MB/sec
> >> >>>> - RemoteScanRanges: 0 (0)
> >> >>>> - RowBatchQueueGetWaitTime: 13.979ms
> >> >>>> - RowBatchQueuePutWaitTime: 0.000ns
> >> >>>> - RowsRead: 89.91K (89910)
> >> >>>> - RowsReturned: 42.68K (42681)
> >> >>>> - RowsReturnedRate: 2.71 M/sec
> >> >>>> - ScanRangesComplete: 6 (6)
> >> >>>> - ScannerThreadsInvoluntaryContextSwitches: 3 (3)
> >> >>>> - ScannerThreadsTotalWallClockTime: 227.825ms
> >> >>>>
>
Re: Estimate peak memory VS used peak memory
Posted by Jeszy <je...@gmail.com>.
Queries will be killed based on actual usage (peak memory usage across
hosts), so the 200mb is the interesting value in your example.
Compare the pool's available memory to the query's mem requirement
(based on estimate or mem_limit, as discussed) to predict admission.
On 23 February 2018 at 10:06, Fawze Abujaber <fa...@gmail.com> wrote:
> Thanks jezy for your detailed response.
>
> Yes I read the documentation.
>
> Let simplify my question:
>
> I have pools set up with memory limit per node and concurrency.
>
> If I’m looking on the historical impala queries that I have and the metrics
> I have per query, on which metrics I can understand that impala will kill
> the query, for example if I have a query with estimate of 2GB and the used
> per node is 200mb, what is the default memory values that i need to setup so
> the query will not fail.
>
> The second one is the distribution between pools, if one query is running
> which metrics o have to look into to know if I submit a query it fail or
> not.
>
> On Fri, 23 Feb 2018 at 10:48 Jeszy <je...@gmail.com> wrote:
>>
>> Hey Fawze,
>>
>> Answers inline.
>>
>> On 23 February 2018 at 01:23, Fawze Abujaber <fa...@gmail.com> wrote:
>> > There is no option in the admission control to setup memory limit per
>> > query,
>> > the memory limit is per pool and there is a default memory per node for
>> > query.
>>
>> per node for query memory limit multiplied by number of nodes gives
>> you a per query memory limit. I agree its confusing that the
>> configurations mix and match between per-node and aggregated values.
>> In this case there's a good reason though, as a single node running
>> out of memory will lead to query failure, meaning that in addition to
>> total memory used, distribution of memory usage between hosts also
>> matters.
>>
>> > I have hundreds of impala queries and more add hoc queries, making a
>> > pool
>> > for each query is not a visible solution.
>> >
>> > still waiting to understand how the estimate per node related to the
>> > default
>> > memory per node I set up per pool, is it used in the decision of queuing
>> > and
>> > killing the query? and if this is true how it was not kill a query that
>> > was
>> > estimated it needs 8.2TB memory per node.
>> >
>> > Understanding on which parameters impala decides to kill a query can
>> > help
>> > understand to define and divide the memory between the pools.
>>
>> If you set mem_limit at any level (service level, pool level, or query
>> level), it will be used for admission control purposes instead of
>> estimates. So a 8.2TB estimate would not be a problem, if impala can
>> reserve mem_limit amount on each host, it will start running the
>> query.
>>
>> > Passing memory limit per query manually is also not visible and such
>> > settings not needs admission control.
>> >
>> > I have support pool that runs ad hoc query and I can not ask them to use
>> > memory limit per query, and I have analytics pool which is fully
>> > business
>> > and I can rely on admission control if it extremely in accurate.
>>
>> It's a bit tricky to use memory-based admission control with
>> non-trivial ad hoc queries. For simple ad-hoc queries, you can try to
>> come up with a 'good enough' mem_limit, or omit mem_limit and trust
>> impala's estimations. You can check the estimated vs. actual values
>> for a representative set of ad hoc queries to see what would work in
>> your case. I've found that people tend to go with a large enough
>> mem_limit for the ad hoc pool.
>>
>> > Can someone explain me exactly which recommended setting to use per pool
>> > and
>> > which of them rely on impala memory estimates?
>>
>> The documentation of admission control
>> (https://impala.apache.org/docs/build/html/topics/impala_admission.html)
>> gives you a good view on how stuff works, but you will have to figure
>> out how to use these features for your specific use case. That said,
>> when using memory based admission control, it is best practice to
>> always use a mem_limit due to potential inaccuracy of estimates as
>> well as potential variance of estimates between Impala releases. Keep
>> in mind that you can opt to set a default mem_limit for one pool and
>> leave it unset for another.
>>
>> > So my conclusion right now to avoid using any settings rely on the
>> > estimates
>> > and to ignore the estimates when I want to evaluate query.
>>
>> Sounds good.
>>
>> > @mostafa, since my issue with all the query, I think the profile will
>> > not
>> > help me to solve such huge issue.
>> >
>> > I’m planning to move a way from Vertica and rely on impala as a sql
>> > engine
>> > and now fully confused how I can do this if I can’t use the admission
>> > control.
>> >
>> > Last think, is it recommend to use the impala admission control?
>>
>> Yes. Admission control can take a while to understand, but if done
>> right, it works.
>>
>> HTH
>>
>> > On Fri, 23 Feb 2018 at 1:56 Alexander Behm <al...@cloudera.com>
>> > wrote:
>> >>
>> >> The planner memory estimates are conservative and sometimes extremely
>> >> inaccurate. In their current form, they are rarely appropriate for
>> >> admission
>> >> decisions.
>> >>
>> >> The recommended practice for memory-based admission control it to set a
>> >> mem_limit for every query. You can make this easier by setting up
>> >> different
>> >> pools with different mem_limits, e.g. a small/medium/big queries pool
>> >> or
>> >> similar.
>> >>
>> >> On Thu, Feb 22, 2018 at 3:00 PM, Mostafa Mokhtar
>> >> <mm...@cloudera.com>
>> >> wrote:
>> >>>
>> >>> It is recommended to set a per query memory limit as part of admission
>> >>> and not rely on estimates as they are sometimes inaccurate.
>> >>> Can you please include the full query profile?
>> >>>
>> >>>
>> >>> On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <fa...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi Mostafa,
>> >>>>
>> >>>> It's not a specific query, almost all the query has such differene
>> >>>> between the 2 values.
>> >>>>
>> >>>> I can see even queries showing the estimate per node is 8.2 Tib
>> >>>>
>> >>>> User: psanalytics
>> >>>>
>> >>>> Database: default
>> >>>>
>> >>>> Query Type: QUERY
>> >>>> Coordinator: slpr-dhc014.lpdomain.com
>> >>>>
>> >>>> Duration: 6.48s
>> >>>>
>> >>>> Rows Produced: 708
>> >>>> Estimated per Node Peak Memory: 8.2 TiB
>> >>>>
>> >>>> Per Node Peak Memory Usage: 1.1 GiB
>> >>>>
>> >>>> Pool: root.impanalytics
>> >>>> Threads: CPU Time: 20.1m
>> >>>>
>> >>>>
>> >>>>
>> >>>> How you can explain this behavior, and for sure i don't have 8.2 Tib
>> >>>> memory per node to give neither you.
>> >>>>
>> >>>> Can you please explain me how i should treat Estimated per Node Peak
>> >>>> Memory and if it used by impala for the resource pool and admission
>> >>>> control
>> >>>> and what is the relation of this value to the default memory per node
>> >>>> that i
>> >>>> setup for each resource pool?
>> >>>>
>> >>>> Below is part of one of the queries profile which the estimate per
>> >>>> node
>> >>>> was ~ @GB and the used was 200MB per node.
>> >>>>
>> >>>>
>> >>>> Instance 744de1b6228736fa:b54bfaa70000000f
>> >>>> (host=slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child:
>> >>>> 1s292ms, %
>> >>>> non-child: 88.82%)
>> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB
>> >>>> 9:1/191.64
>> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB
>> >>>> MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB
>> >>>> ThreadUsage(500.000ms): 1, 1, 1
>> >>>> - AverageThreadTokens: 1.00
>> >>>> - BloomFilterBytes: 0
>> >>>> - PeakMemoryUsage: 7.17 MB (7521751)
>> >>>> - PeakReservation: 0
>> >>>> - PeakUsedReservation: 0
>> >>>> - PerHostPeakMemUsage: 106.53 MB (111709581)
>> >>>> - RowsProduced: 32.83K (32826)
>> >>>> - TotalNetworkReceiveTime: 0.000ns
>> >>>> - TotalNetworkSendTime: 1s297ms
>> >>>> - TotalStorageWaitTime: 234.356ms
>> >>>> - TotalThreadsInvoluntaryContextSwitches: 66 (66)
>> >>>> - TotalThreadsTotalWallClockTime: 1s715ms
>> >>>> - TotalThreadsSysTime: 5.998ms
>> >>>> - TotalThreadsUserTime: 124.975ms
>> >>>> - TotalThreadsVoluntaryContextSwitches: 303 (303)
>> >>>> Fragment Instance Lifecycle Timings:
>> >>>> - ExecTime: 1s394ms
>> >>>> - ExecTreeExecTime: 67.115ms
>> >>>> - OpenTime: 32.795ms
>> >>>> - ExecTreeOpenTime: 73.243us
>> >>>> - PrepareTime: 27.602ms
>> >>>> - ExecTreePrepareTime: 243.141us
>> >>>> DataStreamSender (dst_id=11):(Total: 38.747ms, non-child:
>> >>>> 38.747ms, % non-child: 100.00%)
>> >>>> - BytesSent: 39.71 MB (41643000)
>> >>>> - NetworkThroughput(*): 1.97 GB/sec
>> >>>> - OverallThroughput: 1.00 GB/sec
>> >>>> - PeakMemoryUsage: 59.38 KB (60800)
>> >>>> - RowsReturned: 32.83K (32826)
>> >>>> - SerializeBatchTime: 16.860ms
>> >>>> - TransmitDataRPCTime: 19.698ms
>> >>>> - UncompressedRowBatchSize: 77.58 MB (81350840)
>> >>>> CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child:
>> >>>> 100.00%)
>> >>>> - CodegenTime: 1.299ms
>> >>>> - CompileTime: 10.672ms
>> >>>> - LoadTime: 0.000ns
>> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>> >>>> - NumFunctions: 16 (16)
>> >>>> - NumInstructions: 250 (250)
>> >>>> - OptimizationTime: 21.023ms
>> >>>> - PeakMemoryUsage: 125.00 KB (128000)
>> >>>> - PrepareTime: 24.116ms
>> >>>> SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, %
>> >>>> non-child: 17.85%)
>> >>>> - PeakMemoryUsage: 627.94 KB (643015)
>> >>>> - RowsReturned: 32.77K (32768)
>> >>>> - RowsReturnedRate: 486.81 K/sec
>> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child:
>> >>>> 25.197ms, % non-child: 74.11%)
>> >>>> - BuildRows: 0 (0)
>> >>>> - BuildTime: 0.000ns
>> >>>> - PeakMemoryUsage: 24.00 KB (24576)
>> >>>> - ProbeRows: 32.83K (32826)
>> >>>> - ProbeTime: 0.000ns
>> >>>> - RowsReturned: 16.80M (16795311)
>> >>>> - RowsReturnedRate: 493.99 M/sec
>> >>>> Nested Loop Join Builder:
>> >>>> - PeakMemoryUsage: 8.00 KB (8192)
>> >>>> SINGULAR_ROW_SRC_NODE (id=7):
>> >>>> - PeakMemoryUsage: 0
>> >>>> - RowsReturned: 0 (0)
>> >>>> - RowsReturnedRate: 0
>> >>>> UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, %
>> >>>> non-child: 100.00%)
>> >>>> - AvgCollectionSize: 1.00
>> >>>> - MaxCollectionSize: 1 (1)
>> >>>> - MinCollectionSize: 1 (1)
>> >>>> - NumCollections: 32.83K (32826)
>> >>>> - PeakMemoryUsage: 0
>> >>>> - RowsReturned: 1 (1)
>> >>>> - RowsReturnedRate: 113.00 /sec
>> >>>> HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms,
>> >>>> %
>> >>>> non-child: 100.00%)
>> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB
>> >>>> 9:1/191.64
>> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB
>> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11
>> >>>> out
>> >>>> of 11
>> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
>> >>>> 16:0%
>> >>>> File Formats: PARQUET/SNAPPY:66
>> >>>> BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB
>> >>>> - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ;
>> >>>> Max:
>> >>>> 29.275ms ; Number of samples: 11)
>> >>>> - AverageHdfsReadThreadConcurrency: 0.00
>> >>>> - AverageScannerThreadConcurrency: 0.00
>> >>>> - BytesRead: 2.26 MB (2373738)
>> >>>> - BytesReadDataNodeCache: 0
>> >>>> - BytesReadLocal: 2.26 MB (2373738)
>> >>>> - BytesReadRemoteUnexpected: 0
>> >>>> - BytesReadShortCircuit: 2.26 MB (2373738)
>> >>>> - CachedFileHandlesHitCount: 0 (0)
>> >>>> - CachedFileHandlesMissCount: 77 (77)
>> >>>> - DecompressionTime: 1.543ms
>> >>>> - MaxCompressedTextFileLength: 0
>> >>>> - NumColumns: 6 (6)
>> >>>> - NumDictFilteredRowGroups: 0 (0)
>> >>>> - NumDisksAccessed: 8 (8)
>> >>>> - NumRowGroups: 11 (11)
>> >>>> - NumScannerThreadsStarted: 11 (11)
>> >>>> - NumScannersWithNoReads: 0 (0)
>> >>>> - NumStatsFilteredRowGroups: 0 (0)
>> >>>> - PeakMemoryUsage: 6.51 MB (6821203)
>> >>>> - PerReadThreadRawHdfsThroughput: 764.51 MB/sec
>> >>>> - RemoteScanRanges: 0 (0)
>> >>>> - RowBatchQueueGetWaitTime: 18.825ms
>> >>>> - RowBatchQueuePutWaitTime: 0.000ns
>> >>>> - RowsRead: 73.46K (73464)
>> >>>> - RowsReturned: 32.83K (32826)
>> >>>> - RowsReturnedRate: 1.54 M/sec
>> >>>> - ScanRangesComplete: 11 (11)
>> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>> >>>> - ScannerThreadsTotalWallClockTime: 287.702ms
>> >>>> - MaterializeTupleTime(*): 11.921ms
>> >>>> - ScannerThreadsSysTime: 999.000us
>> >>>> - ScannerThreadsUserTime: 26.990ms
>> >>>> - ScannerThreadsVoluntaryContextSwitches: 232 (232)
>> >>>> - TotalRawHdfsReadTime(*): 2.961ms
>> >>>> - TotalReadThroughput: 1.51 MB/sec
>> >>>> Instance 744de1b6228736fa:b54bfaa70000000d
>> >>>> (host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child:
>> >>>> 1s254ms, %
>> >>>> non-child: 86.28%)
>> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
>> >>>> MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB
>> >>>> ThreadUsage(500.000ms): 1, 1, 1
>> >>>> - AverageThreadTokens: 1.00
>> >>>> - BloomFilterBytes: 0
>> >>>> - PeakMemoryUsage: 5.86 MB (6145964)
>> >>>> - PeakReservation: 0
>> >>>> - PeakUsedReservation: 0
>> >>>> - PerHostPeakMemUsage: 106.66 MB (111842766)
>> >>>> - RowsProduced: 30.34K (30340)
>> >>>> - TotalNetworkReceiveTime: 0.000ns
>> >>>> - TotalNetworkSendTime: 1s256ms
>> >>>> - TotalStorageWaitTime: 106.530ms
>> >>>> - TotalThreadsInvoluntaryContextSwitches: 177 (177)
>> >>>> - TotalThreadsTotalWallClockTime: 1s557ms
>> >>>> - TotalThreadsSysTime: 6.998ms
>> >>>> - TotalThreadsUserTime: 143.976ms
>> >>>> - TotalThreadsVoluntaryContextSwitches: 136 (136)
>> >>>> Fragment Instance Lifecycle Timings:
>> >>>> - ExecTime: 1s390ms
>> >>>> - ExecTreeExecTime: 91.500ms
>> >>>> - OpenTime: 34.085ms
>> >>>> - ExecTreeOpenTime: 81.436us
>> >>>> - PrepareTime: 28.524ms
>> >>>> - ExecTreePrepareTime: 217.949us
>> >>>> DataStreamSender (dst_id=11):(Total: 48.674ms, non-child:
>> >>>> 48.674ms, % non-child: 100.00%)
>> >>>> - BytesSent: 36.61 MB (38388760)
>> >>>> - NetworkThroughput(*): 1.51 GB/sec
>> >>>> - OverallThroughput: 752.15 MB/sec
>> >>>> - PeakMemoryUsage: 59.38 KB (60800)
>> >>>> - RowsReturned: 30.34K (30340)
>> >>>> - SerializeBatchTime: 23.132ms
>> >>>> - TransmitDataRPCTime: 23.609ms
>> >>>> - UncompressedRowBatchSize: 71.70 MB (75187520)
>> >>>> CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child:
>> >>>> 100.00%)
>> >>>> - CodegenTime: 1.382ms
>> >>>> - CompileTime: 10.435ms
>> >>>> - LoadTime: 0.000ns
>> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>> >>>> - NumFunctions: 16 (16)
>> >>>> - NumInstructions: 250 (250)
>> >>>> - OptimizationTime: 22.325ms
>> >>>> - PeakMemoryUsage: 125.00 KB (128000)
>> >>>> - PrepareTime: 25.459ms
>> >>>> SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, %
>> >>>> non-child: 22.90%)
>> >>>> - PeakMemoryUsage: 699.55 KB (716335)
>> >>>> - RowsReturned: 29.70K (29696)
>> >>>> - RowsReturnedRate: 323.80 K/sec
>> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child:
>> >>>> 40.263ms, % non-child: 74.95%)
>> >>>> - BuildRows: 0 (0)
>> >>>> - BuildTime: 0.000ns
>> >>>> - PeakMemoryUsage: 24.00 KB (24576)
>> >>>> - ProbeRows: 30.34K (30340)
>> >>>> - ProbeTime: 0.000ns
>> >>>> - RowsReturned: 15.43M (15426890)
>> >>>> - RowsReturnedRate: 287.16 M/sec
>> >>>> Nested Loop Join Builder:
>> >>>> - PeakMemoryUsage: 8.00 KB (8192)
>> >>>> SINGULAR_ROW_SRC_NODE (id=7):
>> >>>> - PeakMemoryUsage: 0
>> >>>> - RowsReturned: 0 (0)
>> >>>> - RowsReturnedRate: 0
>> >>>> UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms, %
>> >>>> non-child: 100.00%)
>> >>>> - AvgCollectionSize: 1.00
>> >>>> - MaxCollectionSize: 1 (1)
>> >>>> - MinCollectionSize: 1 (1)
>> >>>> - NumCollections: 30.34K (30340)
>> >>>> - PeakMemoryUsage: 0
>> >>>> - RowsReturned: 1 (1)
>> >>>> - RowsReturnedRate: 74.00 /sec
>> >>>> HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms,
>> >>>> %
>> >>>> non-child: 100.00%)
>> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
>> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out
>> >>>> of
>> >>>> 5
>> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
>> >>>> 16:0%
>> >>>> File Formats: PARQUET/SNAPPY:30
>> >>>> BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB
>> >>>> - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ;
>> >>>> Max:
>> >>>> 28.687ms ; Number of samples: 5)
>> >>>> - AverageHdfsReadThreadConcurrency: 0.00
>> >>>> - AverageScannerThreadConcurrency: 0.00
>> >>>> - BytesRead: 1.56 MB (1630546)
>> >>>> - BytesReadDataNodeCache: 0
>> >>>> - BytesReadLocal: 1.56 MB (1630546)
>> >>>> - BytesReadRemoteUnexpected: 0
>> >>>> - BytesReadShortCircuit: 1.56 MB (1630546)
>> >>>> - CachedFileHandlesHitCount: 0 (0)
>> >>>> - CachedFileHandlesMissCount: 35 (35)
>> >>>> - DecompressionTime: 1.267ms
>> >>>> - MaxCompressedTextFileLength: 0
>> >>>> - NumColumns: 6 (6)
>> >>>> - NumDictFilteredRowGroups: 0 (0)
>> >>>> - NumDisksAccessed: 4 (4)
>> >>>> - NumRowGroups: 5 (5)
>> >>>> - NumScannerThreadsStarted: 5 (5)
>> >>>> - NumScannersWithNoReads: 0 (0)
>> >>>> - NumStatsFilteredRowGroups: 0 (0)
>> >>>> - PeakMemoryUsage: 5.59 MB (5859290)
>> >>>> - PerReadThreadRawHdfsThroughput: 920.32 MB/sec
>> >>>> - RemoteScanRanges: 0 (0)
>> >>>> - RowBatchQueueGetWaitTime: 15.335ms
>> >>>> - RowBatchQueuePutWaitTime: 0.000ns
>> >>>> - RowsRead: 64.69K (64692)
>> >>>> - RowsReturned: 30.34K (30340)
>> >>>> - RowsReturnedRate: 1.79 M/sec
>> >>>> - ScanRangesComplete: 5 (5)
>> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>> >>>> - ScannerThreadsTotalWallClockTime: 132.548ms
>> >>>> - MaterializeTupleTime(*): 10.585ms
>> >>>> - ScannerThreadsSysTime: 999.000us
>> >>>> - ScannerThreadsUserTime: 18.995ms
>> >>>> - ScannerThreadsVoluntaryContextSwitches: 79 (79)
>> >>>> - TotalRawHdfsReadTime(*): 1.689ms
>> >>>> - TotalReadThroughput: 1.04 MB/sec
>> >>>> Instance 744de1b6228736fa:b54bfaa700000002
>> >>>> (host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child:
>> >>>> 1s255ms, %
>> >>>> non-child: 86.41%)
>> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
>> >>>> MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB
>> >>>> ThreadUsage(500.000ms): 1, 1, 1
>> >>>> - AverageThreadTokens: 1.00
>> >>>> - BloomFilterBytes: 0
>> >>>> - PeakMemoryUsage: 6.17 MB (6473729)
>> >>>> - PeakReservation: 0
>> >>>> - PeakUsedReservation: 0
>> >>>> - PerHostPeakMemUsage: 107.32 MB (112537640)
>> >>>> - RowsProduced: 29.97K (29969)
>> >>>> - TotalNetworkReceiveTime: 0.000ns
>> >>>> - TotalNetworkSendTime: 1s260ms
>> >>>> - TotalStorageWaitTime: 130.815ms
>> >>>> - TotalThreadsInvoluntaryContextSwitches: 15 (15)
>> >>>> - TotalThreadsTotalWallClockTime: 1s586ms
>> >>>> - TotalThreadsSysTime: 11.995ms
>> >>>> - TotalThreadsUserTime: 146.975ms
>> >>>> - TotalThreadsVoluntaryContextSwitches: 197 (197)
>> >>>> Fragment Instance Lifecycle Timings:
>> >>>> - ExecTime: 1s354ms
>> >>>> - ExecTreeExecTime: 67.791ms
>> >>>> - OpenTime: 52.012ms
>> >>>> - ExecTreeOpenTime: 101.232us
>> >>>> - PrepareTime: 45.946ms
>> >>>> - ExecTreePrepareTime: 309.644us
>> >>>> DataStreamSender (dst_id=11):(Total: 38.995ms, non-child:
>> >>>> 38.995ms, % non-child: 100.00%)
>> >>>> - BytesSent: 35.95 MB (37697260)
>> >>>> - NetworkThroughput(*): 2.14 GB/sec
>> >>>> - OverallThroughput: 921.92 MB/sec
>> >>>> - PeakMemoryUsage: 59.38 KB (60800)
>> >>>> - RowsReturned: 29.97K (29969)
>> >>>> - SerializeBatchTime: 18.055ms
>> >>>> - TransmitDataRPCTime: 16.428ms
>> >>>> - UncompressedRowBatchSize: 70.83 MB (74269800)
>> >>>> CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child:
>> >>>> 100.00%)
>> >>>> - CodegenTime: 2.224ms
>> >>>> - CompileTime: 15.517ms
>> >>>> - LoadTime: 0.000ns
>> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>> >>>> - NumFunctions: 16 (16)
>> >>>> - NumInstructions: 250 (250)
>> >>>> - OptimizationTime: 35.099ms
>> >>>> - PeakMemoryUsage: 125.00 KB (128000)
>> >>>> - PrepareTime: 38.847ms
>> >>>> SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, %
>> >>>> non-child: 18.53%)
>> >>>> - PeakMemoryUsage: 636.17 KB (651439)
>> >>>> - RowsReturned: 29.70K (29696)
>> >>>> - RowsReturnedRate: 435.89 K/sec
>> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child:
>> >>>> 27.325ms, % non-child: 77.31%)
>> >>>> - BuildRows: 0 (0)
>> >>>> - BuildTime: 0.000ns
>> >>>> - PeakMemoryUsage: 24.00 KB (24576)
>> >>>> - ProbeRows: 29.97K (29969)
>> >>>> - ProbeTime: 0.000ns
>> >>>> - RowsReturned: 15.26M (15256601)
>> >>>> - RowsReturnedRate: 431.62 M/sec
>> >>>> Nested Loop Join Builder:
>> >>>> - PeakMemoryUsage: 8.00 KB (8192)
>> >>>> SINGULAR_ROW_SRC_NODE (id=7):
>> >>>> - PeakMemoryUsage: 0
>> >>>> - RowsReturned: 0 (0)
>> >>>> - RowsReturnedRate: 0
>> >>>> UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, %
>> >>>> non-child: 100.00%)
>> >>>> - AvgCollectionSize: 1.00
>> >>>> - MaxCollectionSize: 1 (1)
>> >>>> - MinCollectionSize: 1 (1)
>> >>>> - NumCollections: 29.97K (29969)
>> >>>> - PeakMemoryUsage: 0
>> >>>> - RowsReturned: 1 (1)
>> >>>> - RowsReturnedRate: 124.00 /sec
>> >>>> HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms,
>> >>>> %
>> >>>> non-child: 100.00%)
>> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
>> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out
>> >>>> of
>> >>>> 7
>> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
>> >>>> 16:0%
>> >>>> File Formats: PARQUET/SNAPPY:42
>> >>>> BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB
>> >>>> - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ;
>> >>>> Max:
>> >>>> 10.464ms ; Number of samples: 7)
>> >>>> - AverageHdfsReadThreadConcurrency: 0.00
>> >>>> - AverageScannerThreadConcurrency: 0.00
>> >>>> - BytesRead: 1.75 MB (1833560)
>> >>>> - BytesReadDataNodeCache: 0
>> >>>> - BytesReadLocal: 1.75 MB (1833560)
>> >>>> - BytesReadRemoteUnexpected: 0
>> >>>> - BytesReadShortCircuit: 1.75 MB (1833560)
>> >>>> - CachedFileHandlesHitCount: 0 (0)
>> >>>> - CachedFileHandlesMissCount: 49 (49)
>> >>>> - DecompressionTime: 1.758ms
>> >>>> - MaxCompressedTextFileLength: 0
>> >>>> - NumColumns: 6 (6)
>> >>>> - NumDictFilteredRowGroups: 0 (0)
>> >>>> - NumDisksAccessed: 5 (5)
>> >>>> - NumRowGroups: 7 (7)
>> >>>> - NumScannerThreadsStarted: 7 (7)
>> >>>> - NumScannersWithNoReads: 0 (0)
>> >>>> - NumStatsFilteredRowGroups: 0 (0)
>> >>>> - PeakMemoryUsage: 5.82 MB (6098991)
>> >>>> - PerReadThreadRawHdfsThroughput: 621.09 MB/sec
>> >>>> - RemoteScanRanges: 0 (0)
>> >>>> - RowBatchQueueGetWaitTime: 17.938ms
>> >>>> - RowBatchQueuePutWaitTime: 0.000ns
>> >>>> - RowsRead: 65.17K (65166)
>> >>>> - RowsReturned: 29.97K (29969)
>> >>>> - RowsReturnedRate: 1.49 M/sec
>> >>>> - ScanRangesComplete: 7 (7)
>> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>> >>>> - ScannerThreadsTotalWallClockTime: 179.698ms
>> >>>> - MaterializeTupleTime(*): 13.922ms
>> >>>> - ScannerThreadsSysTime: 2.997ms
>> >>>> - ScannerThreadsUserTime: 29.993ms
>> >>>> - ScannerThreadsVoluntaryContextSwitches: 130 (130)
>> >>>> - TotalRawHdfsReadTime(*): 2.815ms
>> >>>> - TotalReadThroughput: 1.17 MB/sec
>> >>>> Instance 744de1b6228736fa:b54bfaa700000008
>> >>>> (host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child:
>> >>>> 1s305ms, %
>> >>>> non-child: 90.01%)
>> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
>> >>>> MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB
>> >>>> ThreadUsage(500.000ms): 1, 1, 1
>> >>>> - AverageThreadTokens: 1.00
>> >>>> - BloomFilterBytes: 0
>> >>>> - PeakMemoryUsage: 6.69 MB (7017157)
>> >>>> - PeakReservation: 0
>> >>>> - PeakUsedReservation: 0
>> >>>> - PerHostPeakMemUsage: 106.39 MB (111556990)
>> >>>> - RowsProduced: 42.68K (42681)
>> >>>> - TotalNetworkReceiveTime: 0.000ns
>> >>>> - TotalNetworkSendTime: 1s315ms
>> >>>> - TotalStorageWaitTime: 201.411ms
>> >>>> - TotalThreadsInvoluntaryContextSwitches: 43 (43)
>> >>>> - TotalThreadsTotalWallClockTime: 1s658ms
>> >>>> - TotalThreadsSysTime: 6.997ms
>> >>>> - TotalThreadsUserTime: 108.980ms
>> >>>> - TotalThreadsVoluntaryContextSwitches: 179 (179)
>> >>>> Fragment Instance Lifecycle Timings:
>> >>>> - ExecTime: 1s411ms
>> >>>> - ExecTreeExecTime: 68.017ms
>> >>>> - OpenTime: 19.935ms
>> >>>> - ExecTreeOpenTime: 48.883us
>> >>>> - PrepareTime: 19.160ms
>> >>>> - ExecTreePrepareTime: 134.576us
>> >>>> DataStreamSender (dst_id=11):(Total: 40.671ms, non-child:
>> >>>> 40.671ms, % non-child: 100.00%)
>> >>>> - BytesSent: 51.36 MB (53856680)
>> >>>> - NetworkThroughput(*): 2.44 GB/sec
>> >>>> - OverallThroughput: 1.23 GB/sec
>> >>>> - PeakMemoryUsage: 59.38 KB (60800)
>> >>>> - RowsReturned: 42.68K (42681)
>> >>>> - SerializeBatchTime: 18.299ms
>> >>>> - TransmitDataRPCTime: 20.591ms
>> >>>> - UncompressedRowBatchSize: 100.88 MB (105782860)
>> >>>> CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child:
>> >>>> 100.00%)
>> >>>> - CodegenTime: 828.865us
>> >>>> - CompileTime: 6.170ms
>> >>>> - LoadTime: 0.000ns
>> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>> >>>> - NumFunctions: 16 (16)
>> >>>> - NumInstructions: 250 (250)
>> >>>> - OptimizationTime: 12.988ms
>> >>>> - PeakMemoryUsage: 125.00 KB (128000)
>> >>>> - PrepareTime: 16.608ms
>> >>>> SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, %
>> >>>> non-child: 18.27%)
>> >>>> - PeakMemoryUsage: 689.95 KB (706507)
>> >>>> - RowsReturned: 41.98K (41984)
>> >>>> - RowsReturnedRate: 616.99 K/sec
>> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child:
>> >>>> 29.422ms, % non-child: 73.86%)
>> >>>> - BuildRows: 0 (0)
>> >>>> - BuildTime: 0.000ns
>> >>>> - PeakMemoryUsage: 24.00 KB (24576)
>> >>>> - ProbeRows: 42.68K (42681)
>> >>>> - ProbeTime: 0.000ns
>> >>>> - RowsReturned: 21.76M (21760053)
>> >>>> - RowsReturnedRate: 546.22 M/sec
>> >>>> Nested Loop Join Builder:
>> >>>> - PeakMemoryUsage: 8.00 KB (8192)
>> >>>> SINGULAR_ROW_SRC_NODE (id=7):
>> >>>> - PeakMemoryUsage: 0
>> >>>> - RowsReturned: 0 (0)
>> >>>> - RowsReturnedRate: 0
>> >>>> UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms, %
>> >>>> non-child: 100.00%)
>> >>>> - AvgCollectionSize: 1.00
>> >>>> - MaxCollectionSize: 1 (1)
>> >>>> - MinCollectionSize: 1 (1)
>> >>>> - NumCollections: 42.68K (42681)
>> >>>> - PeakMemoryUsage: 0
>> >>>> - RowsReturned: 1 (1)
>> >>>> - RowsReturnedRate: 96.00 /sec
>> >>>> HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms,
>> >>>> %
>> >>>> non-child: 100.00%)
>> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
>> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out
>> >>>> of
>> >>>> 6
>> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
>> >>>> 16:0%
>> >>>> 17:0%
>> >>>> File Formats: PARQUET/SNAPPY:36
>> >>>> BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB
>> >>>> - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ;
>> >>>> Max:
>> >>>> 42.732ms ; Number of samples: 6)
>> >>>> - AverageHdfsReadThreadConcurrency: 0.00
>> >>>> - AverageScannerThreadConcurrency: 0.00
>> >>>> - BytesRead: 2.08 MB (2186159)
>> >>>> - BytesReadDataNodeCache: 0
>> >>>> - BytesReadLocal: 2.08 MB (2186159)
>> >>>> - BytesReadRemoteUnexpected: 0
>> >>>> - BytesReadShortCircuit: 2.08 MB (2186159)
>> >>>> - CachedFileHandlesHitCount: 0 (0)
>> >>>> - CachedFileHandlesMissCount: 42 (42)
>> >>>> - DecompressionTime: 1.480ms
>> >>>> - MaxCompressedTextFileLength: 0
>> >>>> - NumColumns: 6 (6)
>> >>>> - NumDictFilteredRowGroups: 0 (0)
>> >>>> - NumDisksAccessed: 4 (4)
>> >>>> - NumRowGroups: 6 (6)
>> >>>> - NumScannerThreadsStarted: 6 (6)
>> >>>> - NumScannersWithNoReads: 0 (0)
>> >>>> - NumStatsFilteredRowGroups: 0 (0)
>> >>>> - PeakMemoryUsage: 6.42 MB (6730467)
>> >>>> - PerReadThreadRawHdfsThroughput: 77.02 MB/sec
>> >>>> - RemoteScanRanges: 0 (0)
>> >>>> - RowBatchQueueGetWaitTime: 13.979ms
>> >>>> - RowBatchQueuePutWaitTime: 0.000ns
>> >>>> - RowsRead: 89.91K (89910)
>> >>>> - RowsReturned: 42.68K (42681)
>> >>>> - RowsReturnedRate: 2.71 M/sec
>> >>>> - ScanRangesComplete: 6 (6)
>> >>>> - ScannerThreadsInvoluntaryContextSwitches: 3 (3)
>> >>>> - ScannerThreadsTotalWallClockTime: 227.825ms
>> >>>>
Re: Estimate peak memory VS used peak memory
Posted by Fawze Abujaber <fa...@gmail.com>.
Thanks jezy for your detailed response.
Yes I read the documentation.
Let simplify my question:
I have pools set up with memory limit per node and concurrency.
If I’m looking on the historical impala queries that I have and the metrics
I have per query, on which metrics I can understand that impala will kill
the query, for example if I have a query with estimate of 2GB and the used
per node is 200mb, what is the default memory values that i need to setup
so the query will not fail.
The second one is the distribution between pools, if one query is running
which metrics o have to look into to know if I submit a query it fail or
not.
On Fri, 23 Feb 2018 at 10:48 Jeszy <je...@gmail.com> wrote:
> Hey Fawze,
>
> Answers inline.
>
> On 23 February 2018 at 01:23, Fawze Abujaber <fa...@gmail.com> wrote:
> > There is no option in the admission control to setup memory limit per
> query,
> > the memory limit is per pool and there is a default memory per node for
> > query.
>
> per node for query memory limit multiplied by number of nodes gives
> you a per query memory limit. I agree its confusing that the
> configurations mix and match between per-node and aggregated values.
> In this case there's a good reason though, as a single node running
> out of memory will lead to query failure, meaning that in addition to
> total memory used, distribution of memory usage between hosts also
> matters.
>
> > I have hundreds of impala queries and more add hoc queries, making a pool
> > for each query is not a visible solution.
> >
> > still waiting to understand how the estimate per node related to the
> default
> > memory per node I set up per pool, is it used in the decision of queuing
> and
> > killing the query? and if this is true how it was not kill a query that
> was
> > estimated it needs 8.2TB memory per node.
> >
> > Understanding on which parameters impala decides to kill a query can help
> > understand to define and divide the memory between the pools.
>
> If you set mem_limit at any level (service level, pool level, or query
> level), it will be used for admission control purposes instead of
> estimates. So a 8.2TB estimate would not be a problem, if impala can
> reserve mem_limit amount on each host, it will start running the
> query.
>
> > Passing memory limit per query manually is also not visible and such
> > settings not needs admission control.
> >
> > I have support pool that runs ad hoc query and I can not ask them to use
> > memory limit per query, and I have analytics pool which is fully business
> > and I can rely on admission control if it extremely in accurate.
>
> It's a bit tricky to use memory-based admission control with
> non-trivial ad hoc queries. For simple ad-hoc queries, you can try to
> come up with a 'good enough' mem_limit, or omit mem_limit and trust
> impala's estimations. You can check the estimated vs. actual values
> for a representative set of ad hoc queries to see what would work in
> your case. I've found that people tend to go with a large enough
> mem_limit for the ad hoc pool.
>
> > Can someone explain me exactly which recommended setting to use per pool
> and
> > which of them rely on impala memory estimates?
>
> The documentation of admission control
> (https://impala.apache.org/docs/build/html/topics/impala_admission.html)
> gives you a good view on how stuff works, but you will have to figure
> out how to use these features for your specific use case. That said,
> when using memory based admission control, it is best practice to
> always use a mem_limit due to potential inaccuracy of estimates as
> well as potential variance of estimates between Impala releases. Keep
> in mind that you can opt to set a default mem_limit for one pool and
> leave it unset for another.
>
> > So my conclusion right now to avoid using any settings rely on the
> estimates
> > and to ignore the estimates when I want to evaluate query.
>
> Sounds good.
>
> > @mostafa, since my issue with all the query, I think the profile will not
> > help me to solve such huge issue.
> >
> > I’m planning to move a way from Vertica and rely on impala as a sql
> engine
> > and now fully confused how I can do this if I can’t use the admission
> > control.
> >
> > Last think, is it recommend to use the impala admission control?
>
> Yes. Admission control can take a while to understand, but if done
> right, it works.
>
> HTH
>
> > On Fri, 23 Feb 2018 at 1:56 Alexander Behm <al...@cloudera.com>
> wrote:
> >>
> >> The planner memory estimates are conservative and sometimes extremely
> >> inaccurate. In their current form, they are rarely appropriate for
> admission
> >> decisions.
> >>
> >> The recommended practice for memory-based admission control it to set a
> >> mem_limit for every query. You can make this easier by setting up
> different
> >> pools with different mem_limits, e.g. a small/medium/big queries pool or
> >> similar.
> >>
> >> On Thu, Feb 22, 2018 at 3:00 PM, Mostafa Mokhtar <mmokhtar@cloudera.com
> >
> >> wrote:
> >>>
> >>> It is recommended to set a per query memory limit as part of admission
> >>> and not rely on estimates as they are sometimes inaccurate.
> >>> Can you please include the full query profile?
> >>>
> >>>
> >>> On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <fa...@gmail.com>
> >>> wrote:
> >>>>
> >>>> Hi Mostafa,
> >>>>
> >>>> It's not a specific query, almost all the query has such differene
> >>>> between the 2 values.
> >>>>
> >>>> I can see even queries showing the estimate per node is 8.2 Tib
> >>>>
> >>>> User: psanalytics
> >>>>
> >>>> Database: default
> >>>>
> >>>> Query Type: QUERY
> >>>> Coordinator: slpr-dhc014.lpdomain.com
> >>>>
> >>>> Duration: 6.48s
> >>>>
> >>>> Rows Produced: 708
> >>>> Estimated per Node Peak Memory: 8.2 TiB
> >>>>
> >>>> Per Node Peak Memory Usage: 1.1 GiB
> >>>>
> >>>> Pool: root.impanalytics
> >>>> Threads: CPU Time: 20.1m
> >>>>
> >>>>
> >>>>
> >>>> How you can explain this behavior, and for sure i don't have 8.2 Tib
> >>>> memory per node to give neither you.
> >>>>
> >>>> Can you please explain me how i should treat Estimated per Node Peak
> >>>> Memory and if it used by impala for the resource pool and admission
> control
> >>>> and what is the relation of this value to the default memory per node
> that i
> >>>> setup for each resource pool?
> >>>>
> >>>> Below is part of one of the queries profile which the estimate per
> node
> >>>> was ~ @GB and the used was 200MB per node.
> >>>>
> >>>>
> >>>> Instance 744de1b6228736fa:b54bfaa70000000f
> >>>> (host=slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child:
> 1s292ms, %
> >>>> non-child: 88.82%)
> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB
> 9:1/191.64
> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB
> >>>> MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB
> >>>> ThreadUsage(500.000ms): 1, 1, 1
> >>>> - AverageThreadTokens: 1.00
> >>>> - BloomFilterBytes: 0
> >>>> - PeakMemoryUsage: 7.17 MB (7521751)
> >>>> - PeakReservation: 0
> >>>> - PeakUsedReservation: 0
> >>>> - PerHostPeakMemUsage: 106.53 MB (111709581)
> >>>> - RowsProduced: 32.83K (32826)
> >>>> - TotalNetworkReceiveTime: 0.000ns
> >>>> - TotalNetworkSendTime: 1s297ms
> >>>> - TotalStorageWaitTime: 234.356ms
> >>>> - TotalThreadsInvoluntaryContextSwitches: 66 (66)
> >>>> - TotalThreadsTotalWallClockTime: 1s715ms
> >>>> - TotalThreadsSysTime: 5.998ms
> >>>> - TotalThreadsUserTime: 124.975ms
> >>>> - TotalThreadsVoluntaryContextSwitches: 303 (303)
> >>>> Fragment Instance Lifecycle Timings:
> >>>> - ExecTime: 1s394ms
> >>>> - ExecTreeExecTime: 67.115ms
> >>>> - OpenTime: 32.795ms
> >>>> - ExecTreeOpenTime: 73.243us
> >>>> - PrepareTime: 27.602ms
> >>>> - ExecTreePrepareTime: 243.141us
> >>>> DataStreamSender (dst_id=11):(Total: 38.747ms, non-child:
> >>>> 38.747ms, % non-child: 100.00%)
> >>>> - BytesSent: 39.71 MB (41643000)
> >>>> - NetworkThroughput(*): 1.97 GB/sec
> >>>> - OverallThroughput: 1.00 GB/sec
> >>>> - PeakMemoryUsage: 59.38 KB (60800)
> >>>> - RowsReturned: 32.83K (32826)
> >>>> - SerializeBatchTime: 16.860ms
> >>>> - TransmitDataRPCTime: 19.698ms
> >>>> - UncompressedRowBatchSize: 77.58 MB (81350840)
> >>>> CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child:
> >>>> 100.00%)
> >>>> - CodegenTime: 1.299ms
> >>>> - CompileTime: 10.672ms
> >>>> - LoadTime: 0.000ns
> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
> >>>> - NumFunctions: 16 (16)
> >>>> - NumInstructions: 250 (250)
> >>>> - OptimizationTime: 21.023ms
> >>>> - PeakMemoryUsage: 125.00 KB (128000)
> >>>> - PrepareTime: 24.116ms
> >>>> SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, %
> >>>> non-child: 17.85%)
> >>>> - PeakMemoryUsage: 627.94 KB (643015)
> >>>> - RowsReturned: 32.77K (32768)
> >>>> - RowsReturnedRate: 486.81 K/sec
> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child:
> >>>> 25.197ms, % non-child: 74.11%)
> >>>> - BuildRows: 0 (0)
> >>>> - BuildTime: 0.000ns
> >>>> - PeakMemoryUsage: 24.00 KB (24576)
> >>>> - ProbeRows: 32.83K (32826)
> >>>> - ProbeTime: 0.000ns
> >>>> - RowsReturned: 16.80M (16795311)
> >>>> - RowsReturnedRate: 493.99 M/sec
> >>>> Nested Loop Join Builder:
> >>>> - PeakMemoryUsage: 8.00 KB (8192)
> >>>> SINGULAR_ROW_SRC_NODE (id=7):
> >>>> - PeakMemoryUsage: 0
> >>>> - RowsReturned: 0 (0)
> >>>> - RowsReturnedRate: 0
> >>>> UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, %
> >>>> non-child: 100.00%)
> >>>> - AvgCollectionSize: 1.00
> >>>> - MaxCollectionSize: 1 (1)
> >>>> - MinCollectionSize: 1 (1)
> >>>> - NumCollections: 32.83K (32826)
> >>>> - PeakMemoryUsage: 0
> >>>> - RowsReturned: 1 (1)
> >>>> - RowsReturnedRate: 113.00 /sec
> >>>> HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms, %
> >>>> non-child: 100.00%)
> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB
> 9:1/191.64
> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB
> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11 out
> >>>> of 11
> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> 16:0%
> >>>> File Formats: PARQUET/SNAPPY:66
> >>>> BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB
> >>>> - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ;
> Max:
> >>>> 29.275ms ; Number of samples: 11)
> >>>> - AverageHdfsReadThreadConcurrency: 0.00
> >>>> - AverageScannerThreadConcurrency: 0.00
> >>>> - BytesRead: 2.26 MB (2373738)
> >>>> - BytesReadDataNodeCache: 0
> >>>> - BytesReadLocal: 2.26 MB (2373738)
> >>>> - BytesReadRemoteUnexpected: 0
> >>>> - BytesReadShortCircuit: 2.26 MB (2373738)
> >>>> - CachedFileHandlesHitCount: 0 (0)
> >>>> - CachedFileHandlesMissCount: 77 (77)
> >>>> - DecompressionTime: 1.543ms
> >>>> - MaxCompressedTextFileLength: 0
> >>>> - NumColumns: 6 (6)
> >>>> - NumDictFilteredRowGroups: 0 (0)
> >>>> - NumDisksAccessed: 8 (8)
> >>>> - NumRowGroups: 11 (11)
> >>>> - NumScannerThreadsStarted: 11 (11)
> >>>> - NumScannersWithNoReads: 0 (0)
> >>>> - NumStatsFilteredRowGroups: 0 (0)
> >>>> - PeakMemoryUsage: 6.51 MB (6821203)
> >>>> - PerReadThreadRawHdfsThroughput: 764.51 MB/sec
> >>>> - RemoteScanRanges: 0 (0)
> >>>> - RowBatchQueueGetWaitTime: 18.825ms
> >>>> - RowBatchQueuePutWaitTime: 0.000ns
> >>>> - RowsRead: 73.46K (73464)
> >>>> - RowsReturned: 32.83K (32826)
> >>>> - RowsReturnedRate: 1.54 M/sec
> >>>> - ScanRangesComplete: 11 (11)
> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> >>>> - ScannerThreadsTotalWallClockTime: 287.702ms
> >>>> - MaterializeTupleTime(*): 11.921ms
> >>>> - ScannerThreadsSysTime: 999.000us
> >>>> - ScannerThreadsUserTime: 26.990ms
> >>>> - ScannerThreadsVoluntaryContextSwitches: 232 (232)
> >>>> - TotalRawHdfsReadTime(*): 2.961ms
> >>>> - TotalReadThroughput: 1.51 MB/sec
> >>>> Instance 744de1b6228736fa:b54bfaa70000000d
> >>>> (host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child:
> 1s254ms, %
> >>>> non-child: 86.28%)
> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
> >>>> MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB
> >>>> ThreadUsage(500.000ms): 1, 1, 1
> >>>> - AverageThreadTokens: 1.00
> >>>> - BloomFilterBytes: 0
> >>>> - PeakMemoryUsage: 5.86 MB (6145964)
> >>>> - PeakReservation: 0
> >>>> - PeakUsedReservation: 0
> >>>> - PerHostPeakMemUsage: 106.66 MB (111842766)
> >>>> - RowsProduced: 30.34K (30340)
> >>>> - TotalNetworkReceiveTime: 0.000ns
> >>>> - TotalNetworkSendTime: 1s256ms
> >>>> - TotalStorageWaitTime: 106.530ms
> >>>> - TotalThreadsInvoluntaryContextSwitches: 177 (177)
> >>>> - TotalThreadsTotalWallClockTime: 1s557ms
> >>>> - TotalThreadsSysTime: 6.998ms
> >>>> - TotalThreadsUserTime: 143.976ms
> >>>> - TotalThreadsVoluntaryContextSwitches: 136 (136)
> >>>> Fragment Instance Lifecycle Timings:
> >>>> - ExecTime: 1s390ms
> >>>> - ExecTreeExecTime: 91.500ms
> >>>> - OpenTime: 34.085ms
> >>>> - ExecTreeOpenTime: 81.436us
> >>>> - PrepareTime: 28.524ms
> >>>> - ExecTreePrepareTime: 217.949us
> >>>> DataStreamSender (dst_id=11):(Total: 48.674ms, non-child:
> >>>> 48.674ms, % non-child: 100.00%)
> >>>> - BytesSent: 36.61 MB (38388760)
> >>>> - NetworkThroughput(*): 1.51 GB/sec
> >>>> - OverallThroughput: 752.15 MB/sec
> >>>> - PeakMemoryUsage: 59.38 KB (60800)
> >>>> - RowsReturned: 30.34K (30340)
> >>>> - SerializeBatchTime: 23.132ms
> >>>> - TransmitDataRPCTime: 23.609ms
> >>>> - UncompressedRowBatchSize: 71.70 MB (75187520)
> >>>> CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child:
> >>>> 100.00%)
> >>>> - CodegenTime: 1.382ms
> >>>> - CompileTime: 10.435ms
> >>>> - LoadTime: 0.000ns
> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
> >>>> - NumFunctions: 16 (16)
> >>>> - NumInstructions: 250 (250)
> >>>> - OptimizationTime: 22.325ms
> >>>> - PeakMemoryUsage: 125.00 KB (128000)
> >>>> - PrepareTime: 25.459ms
> >>>> SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, %
> >>>> non-child: 22.90%)
> >>>> - PeakMemoryUsage: 699.55 KB (716335)
> >>>> - RowsReturned: 29.70K (29696)
> >>>> - RowsReturnedRate: 323.80 K/sec
> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child:
> >>>> 40.263ms, % non-child: 74.95%)
> >>>> - BuildRows: 0 (0)
> >>>> - BuildTime: 0.000ns
> >>>> - PeakMemoryUsage: 24.00 KB (24576)
> >>>> - ProbeRows: 30.34K (30340)
> >>>> - ProbeTime: 0.000ns
> >>>> - RowsReturned: 15.43M (15426890)
> >>>> - RowsReturnedRate: 287.16 M/sec
> >>>> Nested Loop Join Builder:
> >>>> - PeakMemoryUsage: 8.00 KB (8192)
> >>>> SINGULAR_ROW_SRC_NODE (id=7):
> >>>> - PeakMemoryUsage: 0
> >>>> - RowsReturned: 0 (0)
> >>>> - RowsReturnedRate: 0
> >>>> UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms, %
> >>>> non-child: 100.00%)
> >>>> - AvgCollectionSize: 1.00
> >>>> - MaxCollectionSize: 1 (1)
> >>>> - MinCollectionSize: 1 (1)
> >>>> - NumCollections: 30.34K (30340)
> >>>> - PeakMemoryUsage: 0
> >>>> - RowsReturned: 1 (1)
> >>>> - RowsReturnedRate: 74.00 /sec
> >>>> HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms, %
> >>>> non-child: 100.00%)
> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out
> of
> >>>> 5
> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> 16:0%
> >>>> File Formats: PARQUET/SNAPPY:30
> >>>> BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB
> >>>> - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ;
> Max:
> >>>> 28.687ms ; Number of samples: 5)
> >>>> - AverageHdfsReadThreadConcurrency: 0.00
> >>>> - AverageScannerThreadConcurrency: 0.00
> >>>> - BytesRead: 1.56 MB (1630546)
> >>>> - BytesReadDataNodeCache: 0
> >>>> - BytesReadLocal: 1.56 MB (1630546)
> >>>> - BytesReadRemoteUnexpected: 0
> >>>> - BytesReadShortCircuit: 1.56 MB (1630546)
> >>>> - CachedFileHandlesHitCount: 0 (0)
> >>>> - CachedFileHandlesMissCount: 35 (35)
> >>>> - DecompressionTime: 1.267ms
> >>>> - MaxCompressedTextFileLength: 0
> >>>> - NumColumns: 6 (6)
> >>>> - NumDictFilteredRowGroups: 0 (0)
> >>>> - NumDisksAccessed: 4 (4)
> >>>> - NumRowGroups: 5 (5)
> >>>> - NumScannerThreadsStarted: 5 (5)
> >>>> - NumScannersWithNoReads: 0 (0)
> >>>> - NumStatsFilteredRowGroups: 0 (0)
> >>>> - PeakMemoryUsage: 5.59 MB (5859290)
> >>>> - PerReadThreadRawHdfsThroughput: 920.32 MB/sec
> >>>> - RemoteScanRanges: 0 (0)
> >>>> - RowBatchQueueGetWaitTime: 15.335ms
> >>>> - RowBatchQueuePutWaitTime: 0.000ns
> >>>> - RowsRead: 64.69K (64692)
> >>>> - RowsReturned: 30.34K (30340)
> >>>> - RowsReturnedRate: 1.79 M/sec
> >>>> - ScanRangesComplete: 5 (5)
> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> >>>> - ScannerThreadsTotalWallClockTime: 132.548ms
> >>>> - MaterializeTupleTime(*): 10.585ms
> >>>> - ScannerThreadsSysTime: 999.000us
> >>>> - ScannerThreadsUserTime: 18.995ms
> >>>> - ScannerThreadsVoluntaryContextSwitches: 79 (79)
> >>>> - TotalRawHdfsReadTime(*): 1.689ms
> >>>> - TotalReadThroughput: 1.04 MB/sec
> >>>> Instance 744de1b6228736fa:b54bfaa700000002
> >>>> (host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child:
> 1s255ms, %
> >>>> non-child: 86.41%)
> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
> >>>> MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB
> >>>> ThreadUsage(500.000ms): 1, 1, 1
> >>>> - AverageThreadTokens: 1.00
> >>>> - BloomFilterBytes: 0
> >>>> - PeakMemoryUsage: 6.17 MB (6473729)
> >>>> - PeakReservation: 0
> >>>> - PeakUsedReservation: 0
> >>>> - PerHostPeakMemUsage: 107.32 MB (112537640)
> >>>> - RowsProduced: 29.97K (29969)
> >>>> - TotalNetworkReceiveTime: 0.000ns
> >>>> - TotalNetworkSendTime: 1s260ms
> >>>> - TotalStorageWaitTime: 130.815ms
> >>>> - TotalThreadsInvoluntaryContextSwitches: 15 (15)
> >>>> - TotalThreadsTotalWallClockTime: 1s586ms
> >>>> - TotalThreadsSysTime: 11.995ms
> >>>> - TotalThreadsUserTime: 146.975ms
> >>>> - TotalThreadsVoluntaryContextSwitches: 197 (197)
> >>>> Fragment Instance Lifecycle Timings:
> >>>> - ExecTime: 1s354ms
> >>>> - ExecTreeExecTime: 67.791ms
> >>>> - OpenTime: 52.012ms
> >>>> - ExecTreeOpenTime: 101.232us
> >>>> - PrepareTime: 45.946ms
> >>>> - ExecTreePrepareTime: 309.644us
> >>>> DataStreamSender (dst_id=11):(Total: 38.995ms, non-child:
> >>>> 38.995ms, % non-child: 100.00%)
> >>>> - BytesSent: 35.95 MB (37697260)
> >>>> - NetworkThroughput(*): 2.14 GB/sec
> >>>> - OverallThroughput: 921.92 MB/sec
> >>>> - PeakMemoryUsage: 59.38 KB (60800)
> >>>> - RowsReturned: 29.97K (29969)
> >>>> - SerializeBatchTime: 18.055ms
> >>>> - TransmitDataRPCTime: 16.428ms
> >>>> - UncompressedRowBatchSize: 70.83 MB (74269800)
> >>>> CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child:
> >>>> 100.00%)
> >>>> - CodegenTime: 2.224ms
> >>>> - CompileTime: 15.517ms
> >>>> - LoadTime: 0.000ns
> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
> >>>> - NumFunctions: 16 (16)
> >>>> - NumInstructions: 250 (250)
> >>>> - OptimizationTime: 35.099ms
> >>>> - PeakMemoryUsage: 125.00 KB (128000)
> >>>> - PrepareTime: 38.847ms
> >>>> SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, %
> >>>> non-child: 18.53%)
> >>>> - PeakMemoryUsage: 636.17 KB (651439)
> >>>> - RowsReturned: 29.70K (29696)
> >>>> - RowsReturnedRate: 435.89 K/sec
> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child:
> >>>> 27.325ms, % non-child: 77.31%)
> >>>> - BuildRows: 0 (0)
> >>>> - BuildTime: 0.000ns
> >>>> - PeakMemoryUsage: 24.00 KB (24576)
> >>>> - ProbeRows: 29.97K (29969)
> >>>> - ProbeTime: 0.000ns
> >>>> - RowsReturned: 15.26M (15256601)
> >>>> - RowsReturnedRate: 431.62 M/sec
> >>>> Nested Loop Join Builder:
> >>>> - PeakMemoryUsage: 8.00 KB (8192)
> >>>> SINGULAR_ROW_SRC_NODE (id=7):
> >>>> - PeakMemoryUsage: 0
> >>>> - RowsReturned: 0 (0)
> >>>> - RowsReturnedRate: 0
> >>>> UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, %
> >>>> non-child: 100.00%)
> >>>> - AvgCollectionSize: 1.00
> >>>> - MaxCollectionSize: 1 (1)
> >>>> - MinCollectionSize: 1 (1)
> >>>> - NumCollections: 29.97K (29969)
> >>>> - PeakMemoryUsage: 0
> >>>> - RowsReturned: 1 (1)
> >>>> - RowsReturnedRate: 124.00 /sec
> >>>> HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms, %
> >>>> non-child: 100.00%)
> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out
> of
> >>>> 7
> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> 16:0%
> >>>> File Formats: PARQUET/SNAPPY:42
> >>>> BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB
> >>>> - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ; Max:
> >>>> 10.464ms ; Number of samples: 7)
> >>>> - AverageHdfsReadThreadConcurrency: 0.00
> >>>> - AverageScannerThreadConcurrency: 0.00
> >>>> - BytesRead: 1.75 MB (1833560)
> >>>> - BytesReadDataNodeCache: 0
> >>>> - BytesReadLocal: 1.75 MB (1833560)
> >>>> - BytesReadRemoteUnexpected: 0
> >>>> - BytesReadShortCircuit: 1.75 MB (1833560)
> >>>> - CachedFileHandlesHitCount: 0 (0)
> >>>> - CachedFileHandlesMissCount: 49 (49)
> >>>> - DecompressionTime: 1.758ms
> >>>> - MaxCompressedTextFileLength: 0
> >>>> - NumColumns: 6 (6)
> >>>> - NumDictFilteredRowGroups: 0 (0)
> >>>> - NumDisksAccessed: 5 (5)
> >>>> - NumRowGroups: 7 (7)
> >>>> - NumScannerThreadsStarted: 7 (7)
> >>>> - NumScannersWithNoReads: 0 (0)
> >>>> - NumStatsFilteredRowGroups: 0 (0)
> >>>> - PeakMemoryUsage: 5.82 MB (6098991)
> >>>> - PerReadThreadRawHdfsThroughput: 621.09 MB/sec
> >>>> - RemoteScanRanges: 0 (0)
> >>>> - RowBatchQueueGetWaitTime: 17.938ms
> >>>> - RowBatchQueuePutWaitTime: 0.000ns
> >>>> - RowsRead: 65.17K (65166)
> >>>> - RowsReturned: 29.97K (29969)
> >>>> - RowsReturnedRate: 1.49 M/sec
> >>>> - ScanRangesComplete: 7 (7)
> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> >>>> - ScannerThreadsTotalWallClockTime: 179.698ms
> >>>> - MaterializeTupleTime(*): 13.922ms
> >>>> - ScannerThreadsSysTime: 2.997ms
> >>>> - ScannerThreadsUserTime: 29.993ms
> >>>> - ScannerThreadsVoluntaryContextSwitches: 130 (130)
> >>>> - TotalRawHdfsReadTime(*): 2.815ms
> >>>> - TotalReadThroughput: 1.17 MB/sec
> >>>> Instance 744de1b6228736fa:b54bfaa700000008
> >>>> (host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child:
> 1s305ms, %
> >>>> non-child: 90.01%)
> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
> >>>> MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB
> >>>> ThreadUsage(500.000ms): 1, 1, 1
> >>>> - AverageThreadTokens: 1.00
> >>>> - BloomFilterBytes: 0
> >>>> - PeakMemoryUsage: 6.69 MB (7017157)
> >>>> - PeakReservation: 0
> >>>> - PeakUsedReservation: 0
> >>>> - PerHostPeakMemUsage: 106.39 MB (111556990)
> >>>> - RowsProduced: 42.68K (42681)
> >>>> - TotalNetworkReceiveTime: 0.000ns
> >>>> - TotalNetworkSendTime: 1s315ms
> >>>> - TotalStorageWaitTime: 201.411ms
> >>>> - TotalThreadsInvoluntaryContextSwitches: 43 (43)
> >>>> - TotalThreadsTotalWallClockTime: 1s658ms
> >>>> - TotalThreadsSysTime: 6.997ms
> >>>> - TotalThreadsUserTime: 108.980ms
> >>>> - TotalThreadsVoluntaryContextSwitches: 179 (179)
> >>>> Fragment Instance Lifecycle Timings:
> >>>> - ExecTime: 1s411ms
> >>>> - ExecTreeExecTime: 68.017ms
> >>>> - OpenTime: 19.935ms
> >>>> - ExecTreeOpenTime: 48.883us
> >>>> - PrepareTime: 19.160ms
> >>>> - ExecTreePrepareTime: 134.576us
> >>>> DataStreamSender (dst_id=11):(Total: 40.671ms, non-child:
> >>>> 40.671ms, % non-child: 100.00%)
> >>>> - BytesSent: 51.36 MB (53856680)
> >>>> - NetworkThroughput(*): 2.44 GB/sec
> >>>> - OverallThroughput: 1.23 GB/sec
> >>>> - PeakMemoryUsage: 59.38 KB (60800)
> >>>> - RowsReturned: 42.68K (42681)
> >>>> - SerializeBatchTime: 18.299ms
> >>>> - TransmitDataRPCTime: 20.591ms
> >>>> - UncompressedRowBatchSize: 100.88 MB (105782860)
> >>>> CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child:
> >>>> 100.00%)
> >>>> - CodegenTime: 828.865us
> >>>> - CompileTime: 6.170ms
> >>>> - LoadTime: 0.000ns
> >>>> - ModuleBitcodeSize: 1.96 MB (2050180)
> >>>> - NumFunctions: 16 (16)
> >>>> - NumInstructions: 250 (250)
> >>>> - OptimizationTime: 12.988ms
> >>>> - PeakMemoryUsage: 125.00 KB (128000)
> >>>> - PrepareTime: 16.608ms
> >>>> SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, %
> >>>> non-child: 18.27%)
> >>>> - PeakMemoryUsage: 689.95 KB (706507)
> >>>> - RowsReturned: 41.98K (41984)
> >>>> - RowsReturnedRate: 616.99 K/sec
> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child:
> >>>> 29.422ms, % non-child: 73.86%)
> >>>> - BuildRows: 0 (0)
> >>>> - BuildTime: 0.000ns
> >>>> - PeakMemoryUsage: 24.00 KB (24576)
> >>>> - ProbeRows: 42.68K (42681)
> >>>> - ProbeTime: 0.000ns
> >>>> - RowsReturned: 21.76M (21760053)
> >>>> - RowsReturnedRate: 546.22 M/sec
> >>>> Nested Loop Join Builder:
> >>>> - PeakMemoryUsage: 8.00 KB (8192)
> >>>> SINGULAR_ROW_SRC_NODE (id=7):
> >>>> - PeakMemoryUsage: 0
> >>>> - RowsReturned: 0 (0)
> >>>> - RowsReturnedRate: 0
> >>>> UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms, %
> >>>> non-child: 100.00%)
> >>>> - AvgCollectionSize: 1.00
> >>>> - MaxCollectionSize: 1 (1)
> >>>> - MinCollectionSize: 1 (1)
> >>>> - NumCollections: 42.68K (42681)
> >>>> - PeakMemoryUsage: 0
> >>>> - RowsReturned: 1 (1)
> >>>> - RowsReturnedRate: 96.00 /sec
> >>>> HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms, %
> >>>> non-child: 100.00%)
> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out
> of
> >>>> 6
> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> 16:0%
> >>>> 17:0%
> >>>> File Formats: PARQUET/SNAPPY:36
> >>>> BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB
> >>>> - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ;
> Max:
> >>>> 42.732ms ; Number of samples: 6)
> >>>> - AverageHdfsReadThreadConcurrency: 0.00
> >>>> - AverageScannerThreadConcurrency: 0.00
> >>>> - BytesRead: 2.08 MB (2186159)
> >>>> - BytesReadDataNodeCache: 0
> >>>> - BytesReadLocal: 2.08 MB (2186159)
> >>>> - BytesReadRemoteUnexpected: 0
> >>>> - BytesReadShortCircuit: 2.08 MB (2186159)
> >>>> - CachedFileHandlesHitCount: 0 (0)
> >>>> - CachedFileHandlesMissCount: 42 (42)
> >>>> - DecompressionTime: 1.480ms
> >>>> - MaxCompressedTextFileLength: 0
> >>>> - NumColumns: 6 (6)
> >>>> - NumDictFilteredRowGroups: 0 (0)
> >>>> - NumDisksAccessed: 4 (4)
> >>>> - NumRowGroups: 6 (6)
> >>>> - NumScannerThreadsStarted: 6 (6)
> >>>> - NumScannersWithNoReads: 0 (0)
> >>>> - NumStatsFilteredRowGroups: 0 (0)
> >>>> - PeakMemoryUsage: 6.42 MB (6730467)
> >>>> - PerReadThreadRawHdfsThroughput: 77.02 MB/sec
> >>>> - RemoteScanRanges: 0 (0)
> >>>> - RowBatchQueueGetWaitTime: 13.979ms
> >>>> - RowBatchQueuePutWaitTime: 0.000ns
> >>>> - RowsRead: 89.91K (89910)
> >>>> - RowsReturned: 42.68K (42681)
> >>>> - RowsReturnedRate: 2.71 M/sec
> >>>> - ScanRangesComplete: 6 (6)
> >>>> - ScannerThreadsInvoluntaryContextSwitches: 3 (3)
> >>>> - ScannerThreadsTotalWallClockTime: 227.825ms
> >>>>
>
Re: Estimate peak memory VS used peak memory
Posted by Jeszy <je...@gmail.com>.
Hey Fawze,
Answers inline.
On 23 February 2018 at 01:23, Fawze Abujaber <fa...@gmail.com> wrote:
> There is no option in the admission control to setup memory limit per query,
> the memory limit is per pool and there is a default memory per node for
> query.
per node for query memory limit multiplied by number of nodes gives
you a per query memory limit. I agree its confusing that the
configurations mix and match between per-node and aggregated values.
In this case there's a good reason though, as a single node running
out of memory will lead to query failure, meaning that in addition to
total memory used, distribution of memory usage between hosts also
matters.
> I have hundreds of impala queries and more add hoc queries, making a pool
> for each query is not a visible solution.
>
> still waiting to understand how the estimate per node related to the default
> memory per node I set up per pool, is it used in the decision of queuing and
> killing the query? and if this is true how it was not kill a query that was
> estimated it needs 8.2TB memory per node.
>
> Understanding on which parameters impala decides to kill a query can help
> understand to define and divide the memory between the pools.
If you set mem_limit at any level (service level, pool level, or query
level), it will be used for admission control purposes instead of
estimates. So a 8.2TB estimate would not be a problem, if impala can
reserve mem_limit amount on each host, it will start running the
query.
> Passing memory limit per query manually is also not visible and such
> settings not needs admission control.
>
> I have support pool that runs ad hoc query and I can not ask them to use
> memory limit per query, and I have analytics pool which is fully business
> and I can rely on admission control if it extremely in accurate.
It's a bit tricky to use memory-based admission control with
non-trivial ad hoc queries. For simple ad-hoc queries, you can try to
come up with a 'good enough' mem_limit, or omit mem_limit and trust
impala's estimations. You can check the estimated vs. actual values
for a representative set of ad hoc queries to see what would work in
your case. I've found that people tend to go with a large enough
mem_limit for the ad hoc pool.
> Can someone explain me exactly which recommended setting to use per pool and
> which of them rely on impala memory estimates?
The documentation of admission control
(https://impala.apache.org/docs/build/html/topics/impala_admission.html)
gives you a good view on how stuff works, but you will have to figure
out how to use these features for your specific use case. That said,
when using memory based admission control, it is best practice to
always use a mem_limit due to potential inaccuracy of estimates as
well as potential variance of estimates between Impala releases. Keep
in mind that you can opt to set a default mem_limit for one pool and
leave it unset for another.
> So my conclusion right now to avoid using any settings rely on the estimates
> and to ignore the estimates when I want to evaluate query.
Sounds good.
> @mostafa, since my issue with all the query, I think the profile will not
> help me to solve such huge issue.
>
> I’m planning to move a way from Vertica and rely on impala as a sql engine
> and now fully confused how I can do this if I can’t use the admission
> control.
>
> Last think, is it recommend to use the impala admission control?
Yes. Admission control can take a while to understand, but if done
right, it works.
HTH
> On Fri, 23 Feb 2018 at 1:56 Alexander Behm <al...@cloudera.com> wrote:
>>
>> The planner memory estimates are conservative and sometimes extremely
>> inaccurate. In their current form, they are rarely appropriate for admission
>> decisions.
>>
>> The recommended practice for memory-based admission control it to set a
>> mem_limit for every query. You can make this easier by setting up different
>> pools with different mem_limits, e.g. a small/medium/big queries pool or
>> similar.
>>
>> On Thu, Feb 22, 2018 at 3:00 PM, Mostafa Mokhtar <mm...@cloudera.com>
>> wrote:
>>>
>>> It is recommended to set a per query memory limit as part of admission
>>> and not rely on estimates as they are sometimes inaccurate.
>>> Can you please include the full query profile?
>>>
>>>
>>> On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <fa...@gmail.com>
>>> wrote:
>>>>
>>>> Hi Mostafa,
>>>>
>>>> It's not a specific query, almost all the query has such differene
>>>> between the 2 values.
>>>>
>>>> I can see even queries showing the estimate per node is 8.2 Tib
>>>>
>>>> User: psanalytics
>>>>
>>>> Database: default
>>>>
>>>> Query Type: QUERY
>>>> Coordinator: slpr-dhc014.lpdomain.com
>>>>
>>>> Duration: 6.48s
>>>>
>>>> Rows Produced: 708
>>>> Estimated per Node Peak Memory: 8.2 TiB
>>>>
>>>> Per Node Peak Memory Usage: 1.1 GiB
>>>>
>>>> Pool: root.impanalytics
>>>> Threads: CPU Time: 20.1m
>>>>
>>>>
>>>>
>>>> How you can explain this behavior, and for sure i don't have 8.2 Tib
>>>> memory per node to give neither you.
>>>>
>>>> Can you please explain me how i should treat Estimated per Node Peak
>>>> Memory and if it used by impala for the resource pool and admission control
>>>> and what is the relation of this value to the default memory per node that i
>>>> setup for each resource pool?
>>>>
>>>> Below is part of one of the queries profile which the estimate per node
>>>> was ~ @GB and the used was 200MB per node.
>>>>
>>>>
>>>> Instance 744de1b6228736fa:b54bfaa70000000f
>>>> (host=slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child: 1s292ms, %
>>>> non-child: 88.82%)
>>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64
>>>> KB 5:2/289.57 KB 3:2/1012.83 KB
>>>> MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB
>>>> ThreadUsage(500.000ms): 1, 1, 1
>>>> - AverageThreadTokens: 1.00
>>>> - BloomFilterBytes: 0
>>>> - PeakMemoryUsage: 7.17 MB (7521751)
>>>> - PeakReservation: 0
>>>> - PeakUsedReservation: 0
>>>> - PerHostPeakMemUsage: 106.53 MB (111709581)
>>>> - RowsProduced: 32.83K (32826)
>>>> - TotalNetworkReceiveTime: 0.000ns
>>>> - TotalNetworkSendTime: 1s297ms
>>>> - TotalStorageWaitTime: 234.356ms
>>>> - TotalThreadsInvoluntaryContextSwitches: 66 (66)
>>>> - TotalThreadsTotalWallClockTime: 1s715ms
>>>> - TotalThreadsSysTime: 5.998ms
>>>> - TotalThreadsUserTime: 124.975ms
>>>> - TotalThreadsVoluntaryContextSwitches: 303 (303)
>>>> Fragment Instance Lifecycle Timings:
>>>> - ExecTime: 1s394ms
>>>> - ExecTreeExecTime: 67.115ms
>>>> - OpenTime: 32.795ms
>>>> - ExecTreeOpenTime: 73.243us
>>>> - PrepareTime: 27.602ms
>>>> - ExecTreePrepareTime: 243.141us
>>>> DataStreamSender (dst_id=11):(Total: 38.747ms, non-child:
>>>> 38.747ms, % non-child: 100.00%)
>>>> - BytesSent: 39.71 MB (41643000)
>>>> - NetworkThroughput(*): 1.97 GB/sec
>>>> - OverallThroughput: 1.00 GB/sec
>>>> - PeakMemoryUsage: 59.38 KB (60800)
>>>> - RowsReturned: 32.83K (32826)
>>>> - SerializeBatchTime: 16.860ms
>>>> - TransmitDataRPCTime: 19.698ms
>>>> - UncompressedRowBatchSize: 77.58 MB (81350840)
>>>> CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child:
>>>> 100.00%)
>>>> - CodegenTime: 1.299ms
>>>> - CompileTime: 10.672ms
>>>> - LoadTime: 0.000ns
>>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>>>> - NumFunctions: 16 (16)
>>>> - NumInstructions: 250 (250)
>>>> - OptimizationTime: 21.023ms
>>>> - PeakMemoryUsage: 125.00 KB (128000)
>>>> - PrepareTime: 24.116ms
>>>> SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, %
>>>> non-child: 17.85%)
>>>> - PeakMemoryUsage: 627.94 KB (643015)
>>>> - RowsReturned: 32.77K (32768)
>>>> - RowsReturnedRate: 486.81 K/sec
>>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child:
>>>> 25.197ms, % non-child: 74.11%)
>>>> - BuildRows: 0 (0)
>>>> - BuildTime: 0.000ns
>>>> - PeakMemoryUsage: 24.00 KB (24576)
>>>> - ProbeRows: 32.83K (32826)
>>>> - ProbeTime: 0.000ns
>>>> - RowsReturned: 16.80M (16795311)
>>>> - RowsReturnedRate: 493.99 M/sec
>>>> Nested Loop Join Builder:
>>>> - PeakMemoryUsage: 8.00 KB (8192)
>>>> SINGULAR_ROW_SRC_NODE (id=7):
>>>> - PeakMemoryUsage: 0
>>>> - RowsReturned: 0 (0)
>>>> - RowsReturnedRate: 0
>>>> UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, %
>>>> non-child: 100.00%)
>>>> - AvgCollectionSize: 1.00
>>>> - MaxCollectionSize: 1 (1)
>>>> - MinCollectionSize: 1 (1)
>>>> - NumCollections: 32.83K (32826)
>>>> - PeakMemoryUsage: 0
>>>> - RowsReturned: 1 (1)
>>>> - RowsReturnedRate: 113.00 /sec
>>>> HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms, %
>>>> non-child: 100.00%)
>>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64
>>>> KB 5:2/289.57 KB 3:2/1012.83 KB
>>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11 out
>>>> of 11
>>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>> File Formats: PARQUET/SNAPPY:66
>>>> BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB
>>>> - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ; Max:
>>>> 29.275ms ; Number of samples: 11)
>>>> - AverageHdfsReadThreadConcurrency: 0.00
>>>> - AverageScannerThreadConcurrency: 0.00
>>>> - BytesRead: 2.26 MB (2373738)
>>>> - BytesReadDataNodeCache: 0
>>>> - BytesReadLocal: 2.26 MB (2373738)
>>>> - BytesReadRemoteUnexpected: 0
>>>> - BytesReadShortCircuit: 2.26 MB (2373738)
>>>> - CachedFileHandlesHitCount: 0 (0)
>>>> - CachedFileHandlesMissCount: 77 (77)
>>>> - DecompressionTime: 1.543ms
>>>> - MaxCompressedTextFileLength: 0
>>>> - NumColumns: 6 (6)
>>>> - NumDictFilteredRowGroups: 0 (0)
>>>> - NumDisksAccessed: 8 (8)
>>>> - NumRowGroups: 11 (11)
>>>> - NumScannerThreadsStarted: 11 (11)
>>>> - NumScannersWithNoReads: 0 (0)
>>>> - NumStatsFilteredRowGroups: 0 (0)
>>>> - PeakMemoryUsage: 6.51 MB (6821203)
>>>> - PerReadThreadRawHdfsThroughput: 764.51 MB/sec
>>>> - RemoteScanRanges: 0 (0)
>>>> - RowBatchQueueGetWaitTime: 18.825ms
>>>> - RowBatchQueuePutWaitTime: 0.000ns
>>>> - RowsRead: 73.46K (73464)
>>>> - RowsReturned: 32.83K (32826)
>>>> - RowsReturnedRate: 1.54 M/sec
>>>> - ScanRangesComplete: 11 (11)
>>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>>>> - ScannerThreadsTotalWallClockTime: 287.702ms
>>>> - MaterializeTupleTime(*): 11.921ms
>>>> - ScannerThreadsSysTime: 999.000us
>>>> - ScannerThreadsUserTime: 26.990ms
>>>> - ScannerThreadsVoluntaryContextSwitches: 232 (232)
>>>> - TotalRawHdfsReadTime(*): 2.961ms
>>>> - TotalReadThroughput: 1.51 MB/sec
>>>> Instance 744de1b6228736fa:b54bfaa70000000d
>>>> (host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child: 1s254ms, %
>>>> non-child: 86.28%)
>>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
>>>> MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB
>>>> ThreadUsage(500.000ms): 1, 1, 1
>>>> - AverageThreadTokens: 1.00
>>>> - BloomFilterBytes: 0
>>>> - PeakMemoryUsage: 5.86 MB (6145964)
>>>> - PeakReservation: 0
>>>> - PeakUsedReservation: 0
>>>> - PerHostPeakMemUsage: 106.66 MB (111842766)
>>>> - RowsProduced: 30.34K (30340)
>>>> - TotalNetworkReceiveTime: 0.000ns
>>>> - TotalNetworkSendTime: 1s256ms
>>>> - TotalStorageWaitTime: 106.530ms
>>>> - TotalThreadsInvoluntaryContextSwitches: 177 (177)
>>>> - TotalThreadsTotalWallClockTime: 1s557ms
>>>> - TotalThreadsSysTime: 6.998ms
>>>> - TotalThreadsUserTime: 143.976ms
>>>> - TotalThreadsVoluntaryContextSwitches: 136 (136)
>>>> Fragment Instance Lifecycle Timings:
>>>> - ExecTime: 1s390ms
>>>> - ExecTreeExecTime: 91.500ms
>>>> - OpenTime: 34.085ms
>>>> - ExecTreeOpenTime: 81.436us
>>>> - PrepareTime: 28.524ms
>>>> - ExecTreePrepareTime: 217.949us
>>>> DataStreamSender (dst_id=11):(Total: 48.674ms, non-child:
>>>> 48.674ms, % non-child: 100.00%)
>>>> - BytesSent: 36.61 MB (38388760)
>>>> - NetworkThroughput(*): 1.51 GB/sec
>>>> - OverallThroughput: 752.15 MB/sec
>>>> - PeakMemoryUsage: 59.38 KB (60800)
>>>> - RowsReturned: 30.34K (30340)
>>>> - SerializeBatchTime: 23.132ms
>>>> - TransmitDataRPCTime: 23.609ms
>>>> - UncompressedRowBatchSize: 71.70 MB (75187520)
>>>> CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child:
>>>> 100.00%)
>>>> - CodegenTime: 1.382ms
>>>> - CompileTime: 10.435ms
>>>> - LoadTime: 0.000ns
>>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>>>> - NumFunctions: 16 (16)
>>>> - NumInstructions: 250 (250)
>>>> - OptimizationTime: 22.325ms
>>>> - PeakMemoryUsage: 125.00 KB (128000)
>>>> - PrepareTime: 25.459ms
>>>> SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, %
>>>> non-child: 22.90%)
>>>> - PeakMemoryUsage: 699.55 KB (716335)
>>>> - RowsReturned: 29.70K (29696)
>>>> - RowsReturnedRate: 323.80 K/sec
>>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child:
>>>> 40.263ms, % non-child: 74.95%)
>>>> - BuildRows: 0 (0)
>>>> - BuildTime: 0.000ns
>>>> - PeakMemoryUsage: 24.00 KB (24576)
>>>> - ProbeRows: 30.34K (30340)
>>>> - ProbeTime: 0.000ns
>>>> - RowsReturned: 15.43M (15426890)
>>>> - RowsReturnedRate: 287.16 M/sec
>>>> Nested Loop Join Builder:
>>>> - PeakMemoryUsage: 8.00 KB (8192)
>>>> SINGULAR_ROW_SRC_NODE (id=7):
>>>> - PeakMemoryUsage: 0
>>>> - RowsReturned: 0 (0)
>>>> - RowsReturnedRate: 0
>>>> UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms, %
>>>> non-child: 100.00%)
>>>> - AvgCollectionSize: 1.00
>>>> - MaxCollectionSize: 1 (1)
>>>> - MinCollectionSize: 1 (1)
>>>> - NumCollections: 30.34K (30340)
>>>> - PeakMemoryUsage: 0
>>>> - RowsReturned: 1 (1)
>>>> - RowsReturnedRate: 74.00 /sec
>>>> HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms, %
>>>> non-child: 100.00%)
>>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
>>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out of
>>>> 5
>>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>> File Formats: PARQUET/SNAPPY:30
>>>> BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB
>>>> - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ; Max:
>>>> 28.687ms ; Number of samples: 5)
>>>> - AverageHdfsReadThreadConcurrency: 0.00
>>>> - AverageScannerThreadConcurrency: 0.00
>>>> - BytesRead: 1.56 MB (1630546)
>>>> - BytesReadDataNodeCache: 0
>>>> - BytesReadLocal: 1.56 MB (1630546)
>>>> - BytesReadRemoteUnexpected: 0
>>>> - BytesReadShortCircuit: 1.56 MB (1630546)
>>>> - CachedFileHandlesHitCount: 0 (0)
>>>> - CachedFileHandlesMissCount: 35 (35)
>>>> - DecompressionTime: 1.267ms
>>>> - MaxCompressedTextFileLength: 0
>>>> - NumColumns: 6 (6)
>>>> - NumDictFilteredRowGroups: 0 (0)
>>>> - NumDisksAccessed: 4 (4)
>>>> - NumRowGroups: 5 (5)
>>>> - NumScannerThreadsStarted: 5 (5)
>>>> - NumScannersWithNoReads: 0 (0)
>>>> - NumStatsFilteredRowGroups: 0 (0)
>>>> - PeakMemoryUsage: 5.59 MB (5859290)
>>>> - PerReadThreadRawHdfsThroughput: 920.32 MB/sec
>>>> - RemoteScanRanges: 0 (0)
>>>> - RowBatchQueueGetWaitTime: 15.335ms
>>>> - RowBatchQueuePutWaitTime: 0.000ns
>>>> - RowsRead: 64.69K (64692)
>>>> - RowsReturned: 30.34K (30340)
>>>> - RowsReturnedRate: 1.79 M/sec
>>>> - ScanRangesComplete: 5 (5)
>>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>>>> - ScannerThreadsTotalWallClockTime: 132.548ms
>>>> - MaterializeTupleTime(*): 10.585ms
>>>> - ScannerThreadsSysTime: 999.000us
>>>> - ScannerThreadsUserTime: 18.995ms
>>>> - ScannerThreadsVoluntaryContextSwitches: 79 (79)
>>>> - TotalRawHdfsReadTime(*): 1.689ms
>>>> - TotalReadThroughput: 1.04 MB/sec
>>>> Instance 744de1b6228736fa:b54bfaa700000002
>>>> (host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child: 1s255ms, %
>>>> non-child: 86.41%)
>>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
>>>> MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB
>>>> ThreadUsage(500.000ms): 1, 1, 1
>>>> - AverageThreadTokens: 1.00
>>>> - BloomFilterBytes: 0
>>>> - PeakMemoryUsage: 6.17 MB (6473729)
>>>> - PeakReservation: 0
>>>> - PeakUsedReservation: 0
>>>> - PerHostPeakMemUsage: 107.32 MB (112537640)
>>>> - RowsProduced: 29.97K (29969)
>>>> - TotalNetworkReceiveTime: 0.000ns
>>>> - TotalNetworkSendTime: 1s260ms
>>>> - TotalStorageWaitTime: 130.815ms
>>>> - TotalThreadsInvoluntaryContextSwitches: 15 (15)
>>>> - TotalThreadsTotalWallClockTime: 1s586ms
>>>> - TotalThreadsSysTime: 11.995ms
>>>> - TotalThreadsUserTime: 146.975ms
>>>> - TotalThreadsVoluntaryContextSwitches: 197 (197)
>>>> Fragment Instance Lifecycle Timings:
>>>> - ExecTime: 1s354ms
>>>> - ExecTreeExecTime: 67.791ms
>>>> - OpenTime: 52.012ms
>>>> - ExecTreeOpenTime: 101.232us
>>>> - PrepareTime: 45.946ms
>>>> - ExecTreePrepareTime: 309.644us
>>>> DataStreamSender (dst_id=11):(Total: 38.995ms, non-child:
>>>> 38.995ms, % non-child: 100.00%)
>>>> - BytesSent: 35.95 MB (37697260)
>>>> - NetworkThroughput(*): 2.14 GB/sec
>>>> - OverallThroughput: 921.92 MB/sec
>>>> - PeakMemoryUsage: 59.38 KB (60800)
>>>> - RowsReturned: 29.97K (29969)
>>>> - SerializeBatchTime: 18.055ms
>>>> - TransmitDataRPCTime: 16.428ms
>>>> - UncompressedRowBatchSize: 70.83 MB (74269800)
>>>> CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child:
>>>> 100.00%)
>>>> - CodegenTime: 2.224ms
>>>> - CompileTime: 15.517ms
>>>> - LoadTime: 0.000ns
>>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>>>> - NumFunctions: 16 (16)
>>>> - NumInstructions: 250 (250)
>>>> - OptimizationTime: 35.099ms
>>>> - PeakMemoryUsage: 125.00 KB (128000)
>>>> - PrepareTime: 38.847ms
>>>> SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, %
>>>> non-child: 18.53%)
>>>> - PeakMemoryUsage: 636.17 KB (651439)
>>>> - RowsReturned: 29.70K (29696)
>>>> - RowsReturnedRate: 435.89 K/sec
>>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child:
>>>> 27.325ms, % non-child: 77.31%)
>>>> - BuildRows: 0 (0)
>>>> - BuildTime: 0.000ns
>>>> - PeakMemoryUsage: 24.00 KB (24576)
>>>> - ProbeRows: 29.97K (29969)
>>>> - ProbeTime: 0.000ns
>>>> - RowsReturned: 15.26M (15256601)
>>>> - RowsReturnedRate: 431.62 M/sec
>>>> Nested Loop Join Builder:
>>>> - PeakMemoryUsage: 8.00 KB (8192)
>>>> SINGULAR_ROW_SRC_NODE (id=7):
>>>> - PeakMemoryUsage: 0
>>>> - RowsReturned: 0 (0)
>>>> - RowsReturnedRate: 0
>>>> UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, %
>>>> non-child: 100.00%)
>>>> - AvgCollectionSize: 1.00
>>>> - MaxCollectionSize: 1 (1)
>>>> - MinCollectionSize: 1 (1)
>>>> - NumCollections: 29.97K (29969)
>>>> - PeakMemoryUsage: 0
>>>> - RowsReturned: 1 (1)
>>>> - RowsReturnedRate: 124.00 /sec
>>>> HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms, %
>>>> non-child: 100.00%)
>>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
>>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of
>>>> 7
>>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>> File Formats: PARQUET/SNAPPY:42
>>>> BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB
>>>> - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ; Max:
>>>> 10.464ms ; Number of samples: 7)
>>>> - AverageHdfsReadThreadConcurrency: 0.00
>>>> - AverageScannerThreadConcurrency: 0.00
>>>> - BytesRead: 1.75 MB (1833560)
>>>> - BytesReadDataNodeCache: 0
>>>> - BytesReadLocal: 1.75 MB (1833560)
>>>> - BytesReadRemoteUnexpected: 0
>>>> - BytesReadShortCircuit: 1.75 MB (1833560)
>>>> - CachedFileHandlesHitCount: 0 (0)
>>>> - CachedFileHandlesMissCount: 49 (49)
>>>> - DecompressionTime: 1.758ms
>>>> - MaxCompressedTextFileLength: 0
>>>> - NumColumns: 6 (6)
>>>> - NumDictFilteredRowGroups: 0 (0)
>>>> - NumDisksAccessed: 5 (5)
>>>> - NumRowGroups: 7 (7)
>>>> - NumScannerThreadsStarted: 7 (7)
>>>> - NumScannersWithNoReads: 0 (0)
>>>> - NumStatsFilteredRowGroups: 0 (0)
>>>> - PeakMemoryUsage: 5.82 MB (6098991)
>>>> - PerReadThreadRawHdfsThroughput: 621.09 MB/sec
>>>> - RemoteScanRanges: 0 (0)
>>>> - RowBatchQueueGetWaitTime: 17.938ms
>>>> - RowBatchQueuePutWaitTime: 0.000ns
>>>> - RowsRead: 65.17K (65166)
>>>> - RowsReturned: 29.97K (29969)
>>>> - RowsReturnedRate: 1.49 M/sec
>>>> - ScanRangesComplete: 7 (7)
>>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>>>> - ScannerThreadsTotalWallClockTime: 179.698ms
>>>> - MaterializeTupleTime(*): 13.922ms
>>>> - ScannerThreadsSysTime: 2.997ms
>>>> - ScannerThreadsUserTime: 29.993ms
>>>> - ScannerThreadsVoluntaryContextSwitches: 130 (130)
>>>> - TotalRawHdfsReadTime(*): 2.815ms
>>>> - TotalReadThroughput: 1.17 MB/sec
>>>> Instance 744de1b6228736fa:b54bfaa700000008
>>>> (host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child: 1s305ms, %
>>>> non-child: 90.01%)
>>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
>>>> MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB
>>>> ThreadUsage(500.000ms): 1, 1, 1
>>>> - AverageThreadTokens: 1.00
>>>> - BloomFilterBytes: 0
>>>> - PeakMemoryUsage: 6.69 MB (7017157)
>>>> - PeakReservation: 0
>>>> - PeakUsedReservation: 0
>>>> - PerHostPeakMemUsage: 106.39 MB (111556990)
>>>> - RowsProduced: 42.68K (42681)
>>>> - TotalNetworkReceiveTime: 0.000ns
>>>> - TotalNetworkSendTime: 1s315ms
>>>> - TotalStorageWaitTime: 201.411ms
>>>> - TotalThreadsInvoluntaryContextSwitches: 43 (43)
>>>> - TotalThreadsTotalWallClockTime: 1s658ms
>>>> - TotalThreadsSysTime: 6.997ms
>>>> - TotalThreadsUserTime: 108.980ms
>>>> - TotalThreadsVoluntaryContextSwitches: 179 (179)
>>>> Fragment Instance Lifecycle Timings:
>>>> - ExecTime: 1s411ms
>>>> - ExecTreeExecTime: 68.017ms
>>>> - OpenTime: 19.935ms
>>>> - ExecTreeOpenTime: 48.883us
>>>> - PrepareTime: 19.160ms
>>>> - ExecTreePrepareTime: 134.576us
>>>> DataStreamSender (dst_id=11):(Total: 40.671ms, non-child:
>>>> 40.671ms, % non-child: 100.00%)
>>>> - BytesSent: 51.36 MB (53856680)
>>>> - NetworkThroughput(*): 2.44 GB/sec
>>>> - OverallThroughput: 1.23 GB/sec
>>>> - PeakMemoryUsage: 59.38 KB (60800)
>>>> - RowsReturned: 42.68K (42681)
>>>> - SerializeBatchTime: 18.299ms
>>>> - TransmitDataRPCTime: 20.591ms
>>>> - UncompressedRowBatchSize: 100.88 MB (105782860)
>>>> CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child:
>>>> 100.00%)
>>>> - CodegenTime: 828.865us
>>>> - CompileTime: 6.170ms
>>>> - LoadTime: 0.000ns
>>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>>>> - NumFunctions: 16 (16)
>>>> - NumInstructions: 250 (250)
>>>> - OptimizationTime: 12.988ms
>>>> - PeakMemoryUsage: 125.00 KB (128000)
>>>> - PrepareTime: 16.608ms
>>>> SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, %
>>>> non-child: 18.27%)
>>>> - PeakMemoryUsage: 689.95 KB (706507)
>>>> - RowsReturned: 41.98K (41984)
>>>> - RowsReturnedRate: 616.99 K/sec
>>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child:
>>>> 29.422ms, % non-child: 73.86%)
>>>> - BuildRows: 0 (0)
>>>> - BuildTime: 0.000ns
>>>> - PeakMemoryUsage: 24.00 KB (24576)
>>>> - ProbeRows: 42.68K (42681)
>>>> - ProbeTime: 0.000ns
>>>> - RowsReturned: 21.76M (21760053)
>>>> - RowsReturnedRate: 546.22 M/sec
>>>> Nested Loop Join Builder:
>>>> - PeakMemoryUsage: 8.00 KB (8192)
>>>> SINGULAR_ROW_SRC_NODE (id=7):
>>>> - PeakMemoryUsage: 0
>>>> - RowsReturned: 0 (0)
>>>> - RowsReturnedRate: 0
>>>> UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms, %
>>>> non-child: 100.00%)
>>>> - AvgCollectionSize: 1.00
>>>> - MaxCollectionSize: 1 (1)
>>>> - MinCollectionSize: 1 (1)
>>>> - NumCollections: 42.68K (42681)
>>>> - PeakMemoryUsage: 0
>>>> - RowsReturned: 1 (1)
>>>> - RowsReturnedRate: 96.00 /sec
>>>> HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms, %
>>>> non-child: 100.00%)
>>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
>>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of
>>>> 6
>>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>> 17:0%
>>>> File Formats: PARQUET/SNAPPY:36
>>>> BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB
>>>> - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ; Max:
>>>> 42.732ms ; Number of samples: 6)
>>>> - AverageHdfsReadThreadConcurrency: 0.00
>>>> - AverageScannerThreadConcurrency: 0.00
>>>> - BytesRead: 2.08 MB (2186159)
>>>> - BytesReadDataNodeCache: 0
>>>> - BytesReadLocal: 2.08 MB (2186159)
>>>> - BytesReadRemoteUnexpected: 0
>>>> - BytesReadShortCircuit: 2.08 MB (2186159)
>>>> - CachedFileHandlesHitCount: 0 (0)
>>>> - CachedFileHandlesMissCount: 42 (42)
>>>> - DecompressionTime: 1.480ms
>>>> - MaxCompressedTextFileLength: 0
>>>> - NumColumns: 6 (6)
>>>> - NumDictFilteredRowGroups: 0 (0)
>>>> - NumDisksAccessed: 4 (4)
>>>> - NumRowGroups: 6 (6)
>>>> - NumScannerThreadsStarted: 6 (6)
>>>> - NumScannersWithNoReads: 0 (0)
>>>> - NumStatsFilteredRowGroups: 0 (0)
>>>> - PeakMemoryUsage: 6.42 MB (6730467)
>>>> - PerReadThreadRawHdfsThroughput: 77.02 MB/sec
>>>> - RemoteScanRanges: 0 (0)
>>>> - RowBatchQueueGetWaitTime: 13.979ms
>>>> - RowBatchQueuePutWaitTime: 0.000ns
>>>> - RowsRead: 89.91K (89910)
>>>> - RowsReturned: 42.68K (42681)
>>>> - RowsReturnedRate: 2.71 M/sec
>>>> - ScanRangesComplete: 6 (6)
>>>> - ScannerThreadsInvoluntaryContextSwitches: 3 (3)
>>>> - ScannerThreadsTotalWallClockTime: 227.825ms
>>>>
Re: Estimate peak memory VS used peak memory
Posted by Fawze Abujaber <fa...@gmail.com>.
There is no option in the admission control to setup memory limit per
query, the memory limit is per pool and there is a default memory per node
for query.
I have hundreds of impala queries and more add hoc queries, making a pool
for each query is not a visible solution.
still waiting to understand how the estimate per node related to the
default memory per node I set up per pool, is it used in the decision of
queuing and killing the query? and if this is true how it was not kill a
query that was estimated it needs 8.2TB memory per node.
Understanding on which parameters impala decides to kill a query can help
understand to define and divide the memory between the pools.
Passing memory limit per query manually is also not visible and such
settings not needs admission control.
I have support pool that runs ad hoc query and I can not ask them to use
memory limit per query, and I have analytics pool which is fully business
and I can rely on admission control if it extremely in accurate.
Can someone explain me exactly which recommended setting to use per pool
and which of them rely on impala memory estimates?
So my conclusion right now to avoid using any settings rely on the
estimates and to ignore the estimates when I want to evaluate query.
@mostafa, since my issue with all the query, I think the profile will not
help me to solve such huge issue.
I’m planning to move a way from Vertica and rely on impala as a sql engine
and now fully confused how I can do this if I can’t use the admission
control.
Last think, is it recommend to use the impala admission control?
On Fri, 23 Feb 2018 at 1:56 Alexander Behm <al...@cloudera.com> wrote:
> The planner memory estimates are conservative and sometimes extremely
> inaccurate. In their current form, they are rarely appropriate for
> admission decisions.
>
> The recommended practice for memory-based admission control it to set a
> mem_limit for every query. You can make this easier by setting up different
> pools with different mem_limits, e.g. a small/medium/big queries pool or
> similar.
>
> On Thu, Feb 22, 2018 at 3:00 PM, Mostafa Mokhtar <mm...@cloudera.com>
> wrote:
>
>> It is recommended to set a per query memory limit as part of admission
>> and not rely on estimates as they are sometimes inaccurate.
>> Can you please include the full query profile?
>>
>>
>> On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <fa...@gmail.com>
>> wrote:
>>
>>> Hi Mostafa,
>>>
>>> It's not a specific query, almost all the query has such differene
>>> between the 2 values.
>>>
>>> I can see even queries showing the estimate per node is 8.2 Tib
>>>
>>>
>>> - User: psanalytics
>>> - Database: default
>>> - Query Type: QUERY
>>> - Coordinator: slpr-dhc014.lpdomain.com
>>> <http://slvr-cmc101:7180/cmf/hardware/hosts/22/status>
>>> - Duration: 6.48s
>>> - Rows Produced: 708
>>> - Estimated per Node Peak Memory: 8.2 TiB
>>> - Per Node Peak Memory Usage: 1.1 GiB
>>> - Pool: root.impanalytics
>>> - Threads: CPU Time: 20.1m
>>>
>>>
>>>
>>> How you can explain this behavior, and for sure i don't have 8.2 Tib
>>> memory per node to give neither you.
>>>
>>> Can you please explain me how i should treat Estimated per Node Peak
>>> Memory and if it used by impala for the resource pool and admission control
>>> and what is the relation of this value to the default memory per node that
>>> i setup for each resource pool?
>>>
>>> Below is part of one of the queries profile which the estimate per node
>>> was ~ @GB and the used was 200MB per node.
>>>
>>>
>>> Instance 744de1b6228736fa:b54bfaa70000000f (host=
>>> slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child: 1s292ms, %
>>> non-child: 88.82%)
>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64
>>> KB 5:2/289.57 KB 3:2/1012.83 KB
>>> MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB
>>> ThreadUsage(500.000ms): 1, 1, 1
>>> - AverageThreadTokens: 1.00
>>> - BloomFilterBytes: 0
>>> - PeakMemoryUsage: 7.17 MB (7521751)
>>> - PeakReservation: 0
>>> - PeakUsedReservation: 0
>>> - PerHostPeakMemUsage: 106.53 MB (111709581)
>>> - RowsProduced: 32.83K (32826)
>>> - TotalNetworkReceiveTime: 0.000ns
>>> - TotalNetworkSendTime: 1s297ms
>>> - TotalStorageWaitTime: 234.356ms
>>> - TotalThreadsInvoluntaryContextSwitches: 66 (66)
>>> - TotalThreadsTotalWallClockTime: 1s715ms
>>> - TotalThreadsSysTime: 5.998ms
>>> - TotalThreadsUserTime: 124.975ms
>>> - TotalThreadsVoluntaryContextSwitches: 303 (303)
>>> Fragment Instance Lifecycle Timings:
>>> - ExecTime: 1s394ms
>>> - ExecTreeExecTime: 67.115ms
>>> - OpenTime: 32.795ms
>>> - ExecTreeOpenTime: 73.243us
>>> - PrepareTime: 27.602ms
>>> - ExecTreePrepareTime: 243.141us
>>> DataStreamSender (dst_id=11):(Total: 38.747ms, non-child:
>>> 38.747ms, % non-child: 100.00%)
>>> - BytesSent: 39.71 MB (41643000)
>>> - NetworkThroughput(*): 1.97 GB/sec
>>> - OverallThroughput: 1.00 GB/sec
>>> - PeakMemoryUsage: 59.38 KB (60800)
>>> - RowsReturned: 32.83K (32826)
>>> - SerializeBatchTime: 16.860ms
>>> - TransmitDataRPCTime: 19.698ms
>>> - UncompressedRowBatchSize: 77.58 MB (81350840)
>>> CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child:
>>> 100.00%)
>>> - CodegenTime: 1.299ms
>>> - CompileTime: 10.672ms
>>> - LoadTime: 0.000ns
>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>>> - NumFunctions: 16 (16)
>>> - NumInstructions: 250 (250)
>>> - OptimizationTime: 21.023ms
>>> - PeakMemoryUsage: 125.00 KB (128000)
>>> - PrepareTime: 24.116ms
>>> SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, %
>>> non-child: 17.85%)
>>> - PeakMemoryUsage: 627.94 KB (643015)
>>> - RowsReturned: 32.77K (32768)
>>> - RowsReturnedRate: 486.81 K/sec
>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child:
>>> 25.197ms, % non-child: 74.11%)
>>> - BuildRows: 0 (0)
>>> - BuildTime: 0.000ns
>>> - PeakMemoryUsage: 24.00 KB (24576)
>>> - ProbeRows: 32.83K (32826)
>>> - ProbeTime: 0.000ns
>>> - RowsReturned: 16.80M (16795311)
>>> - RowsReturnedRate: 493.99 M/sec
>>> Nested Loop Join Builder:
>>> - PeakMemoryUsage: 8.00 KB (8192)
>>> SINGULAR_ROW_SRC_NODE (id=7):
>>> - PeakMemoryUsage: 0
>>> - RowsReturned: 0 (0)
>>> - RowsReturnedRate: 0
>>> UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, %
>>> non-child: 100.00%)
>>> - AvgCollectionSize: 1.00
>>> - MaxCollectionSize: 1 (1)
>>> - MinCollectionSize: 1 (1)
>>> - NumCollections: 32.83K (32826)
>>> - PeakMemoryUsage: 0
>>> - RowsReturned: 1 (1)
>>> - RowsReturnedRate: 113.00 /sec
>>> HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms, %
>>> non-child: 100.00%)
>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64
>>> KB 5:2/289.57 KB 3:2/1012.83 KB
>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11 out
>>> of 11
>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>> File Formats: PARQUET/SNAPPY:66
>>> BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB
>>> - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ; Max:
>>> 29.275ms ; Number of samples: 11)
>>> - AverageHdfsReadThreadConcurrency: 0.00
>>> - AverageScannerThreadConcurrency: 0.00
>>> - BytesRead: 2.26 MB (2373738)
>>> - BytesReadDataNodeCache: 0
>>> - BytesReadLocal: 2.26 MB (2373738)
>>> - BytesReadRemoteUnexpected: 0
>>> - BytesReadShortCircuit: 2.26 MB (2373738)
>>> - CachedFileHandlesHitCount: 0 (0)
>>> - CachedFileHandlesMissCount: 77 (77)
>>> - DecompressionTime: 1.543ms
>>> - MaxCompressedTextFileLength: 0
>>> - NumColumns: 6 (6)
>>> - NumDictFilteredRowGroups: 0 (0)
>>> - NumDisksAccessed: 8 (8)
>>> - NumRowGroups: 11 (11)
>>> - NumScannerThreadsStarted: 11 (11)
>>> - NumScannersWithNoReads: 0 (0)
>>> - NumStatsFilteredRowGroups: 0 (0)
>>> - PeakMemoryUsage: 6.51 MB (6821203)
>>> - PerReadThreadRawHdfsThroughput: 764.51 MB/sec
>>> - RemoteScanRanges: 0 (0)
>>> - RowBatchQueueGetWaitTime: 18.825ms
>>> - RowBatchQueuePutWaitTime: 0.000ns
>>> - RowsRead: 73.46K (73464)
>>> - RowsReturned: 32.83K (32826)
>>> - RowsReturnedRate: 1.54 M/sec
>>> - ScanRangesComplete: 11 (11)
>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>>> - ScannerThreadsTotalWallClockTime: 287.702ms
>>> - MaterializeTupleTime(*): 11.921ms
>>> - ScannerThreadsSysTime: 999.000us
>>> - ScannerThreadsUserTime: 26.990ms
>>> - ScannerThreadsVoluntaryContextSwitches: 232 (232)
>>> - TotalRawHdfsReadTime(*): 2.961ms
>>> - TotalReadThroughput: 1.51 MB/sec
>>> Instance 744de1b6228736fa:b54bfaa70000000d (host=
>>> slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child: 1s254ms, %
>>> non-child: 86.28%)
>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
>>> MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB
>>> ThreadUsage(500.000ms): 1, 1, 1
>>> - AverageThreadTokens: 1.00
>>> - BloomFilterBytes: 0
>>> - PeakMemoryUsage: 5.86 MB (6145964)
>>> - PeakReservation: 0
>>> - PeakUsedReservation: 0
>>> - PerHostPeakMemUsage: 106.66 MB (111842766)
>>> - RowsProduced: 30.34K (30340)
>>> - TotalNetworkReceiveTime: 0.000ns
>>> - TotalNetworkSendTime: 1s256ms
>>> - TotalStorageWaitTime: 106.530ms
>>> - TotalThreadsInvoluntaryContextSwitches: 177 (177)
>>> - TotalThreadsTotalWallClockTime: 1s557ms
>>> - TotalThreadsSysTime: 6.998ms
>>> - TotalThreadsUserTime: 143.976ms
>>> - TotalThreadsVoluntaryContextSwitches: 136 (136)
>>> Fragment Instance Lifecycle Timings:
>>> - ExecTime: 1s390ms
>>> - ExecTreeExecTime: 91.500ms
>>> - OpenTime: 34.085ms
>>> - ExecTreeOpenTime: 81.436us
>>> - PrepareTime: 28.524ms
>>> - ExecTreePrepareTime: 217.949us
>>> DataStreamSender (dst_id=11):(Total: 48.674ms, non-child:
>>> 48.674ms, % non-child: 100.00%)
>>> - BytesSent: 36.61 MB (38388760)
>>> - NetworkThroughput(*): 1.51 GB/sec
>>> - OverallThroughput: 752.15 MB/sec
>>> - PeakMemoryUsage: 59.38 KB (60800)
>>> - RowsReturned: 30.34K (30340)
>>> - SerializeBatchTime: 23.132ms
>>> - TransmitDataRPCTime: 23.609ms
>>> - UncompressedRowBatchSize: 71.70 MB (75187520)
>>> CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child:
>>> 100.00%)
>>> - CodegenTime: 1.382ms
>>> - CompileTime: 10.435ms
>>> - LoadTime: 0.000ns
>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>>> - NumFunctions: 16 (16)
>>> - NumInstructions: 250 (250)
>>> - OptimizationTime: 22.325ms
>>> - PeakMemoryUsage: 125.00 KB (128000)
>>> - PrepareTime: 25.459ms
>>> SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, %
>>> non-child: 22.90%)
>>> - PeakMemoryUsage: 699.55 KB (716335)
>>> - RowsReturned: 29.70K (29696)
>>> - RowsReturnedRate: 323.80 K/sec
>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child:
>>> 40.263ms, % non-child: 74.95%)
>>> - BuildRows: 0 (0)
>>> - BuildTime: 0.000ns
>>> - PeakMemoryUsage: 24.00 KB (24576)
>>> - ProbeRows: 30.34K (30340)
>>> - ProbeTime: 0.000ns
>>> - RowsReturned: 15.43M (15426890)
>>> - RowsReturnedRate: 287.16 M/sec
>>> Nested Loop Join Builder:
>>> - PeakMemoryUsage: 8.00 KB (8192)
>>> SINGULAR_ROW_SRC_NODE (id=7):
>>> - PeakMemoryUsage: 0
>>> - RowsReturned: 0 (0)
>>> - RowsReturnedRate: 0
>>> UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms, %
>>> non-child: 100.00%)
>>> - AvgCollectionSize: 1.00
>>> - MaxCollectionSize: 1 (1)
>>> - MinCollectionSize: 1 (1)
>>> - NumCollections: 30.34K (30340)
>>> - PeakMemoryUsage: 0
>>> - RowsReturned: 1 (1)
>>> - RowsReturnedRate: 74.00 /sec
>>> HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms, %
>>> non-child: 100.00%)
>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out of
>>> 5
>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>> File Formats: PARQUET/SNAPPY:30
>>> BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB
>>> - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ; Max:
>>> 28.687ms ; Number of samples: 5)
>>> - AverageHdfsReadThreadConcurrency: 0.00
>>> - AverageScannerThreadConcurrency: 0.00
>>> - BytesRead: 1.56 MB (1630546)
>>> - BytesReadDataNodeCache: 0
>>> - BytesReadLocal: 1.56 MB (1630546)
>>> - BytesReadRemoteUnexpected: 0
>>> - BytesReadShortCircuit: 1.56 MB (1630546)
>>> - CachedFileHandlesHitCount: 0 (0)
>>> - CachedFileHandlesMissCount: 35 (35)
>>> - DecompressionTime: 1.267ms
>>> - MaxCompressedTextFileLength: 0
>>> - NumColumns: 6 (6)
>>> - NumDictFilteredRowGroups: 0 (0)
>>> - NumDisksAccessed: 4 (4)
>>> - NumRowGroups: 5 (5)
>>> - NumScannerThreadsStarted: 5 (5)
>>> - NumScannersWithNoReads: 0 (0)
>>> - NumStatsFilteredRowGroups: 0 (0)
>>> - PeakMemoryUsage: 5.59 MB (5859290)
>>> - PerReadThreadRawHdfsThroughput: 920.32 MB/sec
>>> - RemoteScanRanges: 0 (0)
>>> - RowBatchQueueGetWaitTime: 15.335ms
>>> - RowBatchQueuePutWaitTime: 0.000ns
>>> - RowsRead: 64.69K (64692)
>>> - RowsReturned: 30.34K (30340)
>>> - RowsReturnedRate: 1.79 M/sec
>>> - ScanRangesComplete: 5 (5)
>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>>> - ScannerThreadsTotalWallClockTime: 132.548ms
>>> - MaterializeTupleTime(*): 10.585ms
>>> - ScannerThreadsSysTime: 999.000us
>>> - ScannerThreadsUserTime: 18.995ms
>>> - ScannerThreadsVoluntaryContextSwitches: 79 (79)
>>> - TotalRawHdfsReadTime(*): 1.689ms
>>> - TotalReadThroughput: 1.04 MB/sec
>>> Instance 744de1b6228736fa:b54bfaa700000002 (host=
>>> slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child: 1s255ms, %
>>> non-child: 86.41%)
>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
>>> MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB
>>> ThreadUsage(500.000ms): 1, 1, 1
>>> - AverageThreadTokens: 1.00
>>> - BloomFilterBytes: 0
>>> - PeakMemoryUsage: 6.17 MB (6473729)
>>> - PeakReservation: 0
>>> - PeakUsedReservation: 0
>>> - PerHostPeakMemUsage: 107.32 MB (112537640)
>>> - RowsProduced: 29.97K (29969)
>>> - TotalNetworkReceiveTime: 0.000ns
>>> - TotalNetworkSendTime: 1s260ms
>>> - TotalStorageWaitTime: 130.815ms
>>> - TotalThreadsInvoluntaryContextSwitches: 15 (15)
>>> - TotalThreadsTotalWallClockTime: 1s586ms
>>> - TotalThreadsSysTime: 11.995ms
>>> - TotalThreadsUserTime: 146.975ms
>>> - TotalThreadsVoluntaryContextSwitches: 197 (197)
>>> Fragment Instance Lifecycle Timings:
>>> - ExecTime: 1s354ms
>>> - ExecTreeExecTime: 67.791ms
>>> - OpenTime: 52.012ms
>>> - ExecTreeOpenTime: 101.232us
>>> - PrepareTime: 45.946ms
>>> - ExecTreePrepareTime: 309.644us
>>> DataStreamSender (dst_id=11):(Total: 38.995ms, non-child:
>>> 38.995ms, % non-child: 100.00%)
>>> - BytesSent: 35.95 MB (37697260)
>>> - NetworkThroughput(*): 2.14 GB/sec
>>> - OverallThroughput: 921.92 MB/sec
>>> - PeakMemoryUsage: 59.38 KB (60800)
>>> - RowsReturned: 29.97K (29969)
>>> - SerializeBatchTime: 18.055ms
>>> - TransmitDataRPCTime: 16.428ms
>>> - UncompressedRowBatchSize: 70.83 MB (74269800)
>>> CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child:
>>> 100.00%)
>>> - CodegenTime: 2.224ms
>>> - CompileTime: 15.517ms
>>> - LoadTime: 0.000ns
>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>>> - NumFunctions: 16 (16)
>>> - NumInstructions: 250 (250)
>>> - OptimizationTime: 35.099ms
>>> - PeakMemoryUsage: 125.00 KB (128000)
>>> - PrepareTime: 38.847ms
>>> SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, %
>>> non-child: 18.53%)
>>> - PeakMemoryUsage: 636.17 KB (651439)
>>> - RowsReturned: 29.70K (29696)
>>> - RowsReturnedRate: 435.89 K/sec
>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child:
>>> 27.325ms, % non-child: 77.31%)
>>> - BuildRows: 0 (0)
>>> - BuildTime: 0.000ns
>>> - PeakMemoryUsage: 24.00 KB (24576)
>>> - ProbeRows: 29.97K (29969)
>>> - ProbeTime: 0.000ns
>>> - RowsReturned: 15.26M (15256601)
>>> - RowsReturnedRate: 431.62 M/sec
>>> Nested Loop Join Builder:
>>> - PeakMemoryUsage: 8.00 KB (8192)
>>> SINGULAR_ROW_SRC_NODE (id=7):
>>> - PeakMemoryUsage: 0
>>> - RowsReturned: 0 (0)
>>> - RowsReturnedRate: 0
>>> UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, %
>>> non-child: 100.00%)
>>> - AvgCollectionSize: 1.00
>>> - MaxCollectionSize: 1 (1)
>>> - MinCollectionSize: 1 (1)
>>> - NumCollections: 29.97K (29969)
>>> - PeakMemoryUsage: 0
>>> - RowsReturned: 1 (1)
>>> - RowsReturnedRate: 124.00 /sec
>>> HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms, %
>>> non-child: 100.00%)
>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of
>>> 7
>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>> File Formats: PARQUET/SNAPPY:42
>>> BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB
>>> - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ; Max:
>>> 10.464ms ; Number of samples: 7)
>>> - AverageHdfsReadThreadConcurrency: 0.00
>>> - AverageScannerThreadConcurrency: 0.00
>>> - BytesRead: 1.75 MB (1833560)
>>> - BytesReadDataNodeCache: 0
>>> - BytesReadLocal: 1.75 MB (1833560)
>>> - BytesReadRemoteUnexpected: 0
>>> - BytesReadShortCircuit: 1.75 MB (1833560)
>>> - CachedFileHandlesHitCount: 0 (0)
>>> - CachedFileHandlesMissCount: 49 (49)
>>> - DecompressionTime: 1.758ms
>>> - MaxCompressedTextFileLength: 0
>>> - NumColumns: 6 (6)
>>> - NumDictFilteredRowGroups: 0 (0)
>>> - NumDisksAccessed: 5 (5)
>>> - NumRowGroups: 7 (7)
>>> - NumScannerThreadsStarted: 7 (7)
>>> - NumScannersWithNoReads: 0 (0)
>>> - NumStatsFilteredRowGroups: 0 (0)
>>> - PeakMemoryUsage: 5.82 MB (6098991)
>>> - PerReadThreadRawHdfsThroughput: 621.09 MB/sec
>>> - RemoteScanRanges: 0 (0)
>>> - RowBatchQueueGetWaitTime: 17.938ms
>>> - RowBatchQueuePutWaitTime: 0.000ns
>>> - RowsRead: 65.17K (65166)
>>> - RowsReturned: 29.97K (29969)
>>> - RowsReturnedRate: 1.49 M/sec
>>> - ScanRangesComplete: 7 (7)
>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>>> - ScannerThreadsTotalWallClockTime: 179.698ms
>>> - MaterializeTupleTime(*): 13.922ms
>>> - ScannerThreadsSysTime: 2.997ms
>>> - ScannerThreadsUserTime: 29.993ms
>>> - ScannerThreadsVoluntaryContextSwitches: 130 (130)
>>> - TotalRawHdfsReadTime(*): 2.815ms
>>> - TotalReadThroughput: 1.17 MB/sec
>>> Instance 744de1b6228736fa:b54bfaa700000008 (host=
>>> slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child: 1s305ms, %
>>> non-child: 90.01%)
>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
>>> MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB
>>> ThreadUsage(500.000ms): 1, 1, 1
>>> - AverageThreadTokens: 1.00
>>> - BloomFilterBytes: 0
>>> - PeakMemoryUsage: 6.69 MB (7017157)
>>> - PeakReservation: 0
>>> - PeakUsedReservation: 0
>>> - PerHostPeakMemUsage: 106.39 MB (111556990)
>>> - RowsProduced: 42.68K (42681)
>>> - TotalNetworkReceiveTime: 0.000ns
>>> - TotalNetworkSendTime: 1s315ms
>>> - TotalStorageWaitTime: 201.411ms
>>> - TotalThreadsInvoluntaryContextSwitches: 43 (43)
>>> - TotalThreadsTotalWallClockTime: 1s658ms
>>> - TotalThreadsSysTime: 6.997ms
>>> - TotalThreadsUserTime: 108.980ms
>>> - TotalThreadsVoluntaryContextSwitches: 179 (179)
>>> Fragment Instance Lifecycle Timings:
>>> - ExecTime: 1s411ms
>>> - ExecTreeExecTime: 68.017ms
>>> - OpenTime: 19.935ms
>>> - ExecTreeOpenTime: 48.883us
>>> - PrepareTime: 19.160ms
>>> - ExecTreePrepareTime: 134.576us
>>> DataStreamSender (dst_id=11):(Total: 40.671ms, non-child:
>>> 40.671ms, % non-child: 100.00%)
>>> - BytesSent: 51.36 MB (53856680)
>>> - NetworkThroughput(*): 2.44 GB/sec
>>> - OverallThroughput: 1.23 GB/sec
>>> - PeakMemoryUsage: 59.38 KB (60800)
>>> - RowsReturned: 42.68K (42681)
>>> - SerializeBatchTime: 18.299ms
>>> - TransmitDataRPCTime: 20.591ms
>>> - UncompressedRowBatchSize: 100.88 MB (105782860)
>>> CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child:
>>> 100.00%)
>>> - CodegenTime: 828.865us
>>> - CompileTime: 6.170ms
>>> - LoadTime: 0.000ns
>>> - ModuleBitcodeSize: 1.96 MB (2050180)
>>> - NumFunctions: 16 (16)
>>> - NumInstructions: 250 (250)
>>> - OptimizationTime: 12.988ms
>>> - PeakMemoryUsage: 125.00 KB (128000)
>>> - PrepareTime: 16.608ms
>>> SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, %
>>> non-child: 18.27%)
>>> - PeakMemoryUsage: 689.95 KB (706507)
>>> - RowsReturned: 41.98K (41984)
>>> - RowsReturnedRate: 616.99 K/sec
>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child:
>>> 29.422ms, % non-child: 73.86%)
>>> - BuildRows: 0 (0)
>>> - BuildTime: 0.000ns
>>> - PeakMemoryUsage: 24.00 KB (24576)
>>> - ProbeRows: 42.68K (42681)
>>> - ProbeTime: 0.000ns
>>> - RowsReturned: 21.76M (21760053)
>>> - RowsReturnedRate: 546.22 M/sec
>>> Nested Loop Join Builder:
>>> - PeakMemoryUsage: 8.00 KB (8192)
>>> SINGULAR_ROW_SRC_NODE (id=7):
>>> - PeakMemoryUsage: 0
>>> - RowsReturned: 0 (0)
>>> - RowsReturnedRate: 0
>>> UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms, %
>>> non-child: 100.00%)
>>> - AvgCollectionSize: 1.00
>>> - MaxCollectionSize: 1 (1)
>>> - MinCollectionSize: 1 (1)
>>> - NumCollections: 42.68K (42681)
>>> - PeakMemoryUsage: 0
>>> - RowsReturned: 1 (1)
>>> - RowsReturnedRate: 96.00 /sec
>>> HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms, %
>>> non-child: 100.00%)
>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of
>>> 6
>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>> 17:0%
>>> File Formats: PARQUET/SNAPPY:36
>>> BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB
>>> - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ; Max:
>>> 42.732ms ; Number of samples: 6)
>>> - AverageHdfsReadThreadConcurrency: 0.00
>>> - AverageScannerThreadConcurrency: 0.00
>>> - BytesRead: 2.08 MB (2186159)
>>> - BytesReadDataNodeCache: 0
>>> - BytesReadLocal: 2.08 MB (2186159)
>>> - BytesReadRemoteUnexpected: 0
>>> - BytesReadShortCircuit: 2.08 MB (2186159)
>>> - CachedFileHandlesHitCount: 0 (0)
>>> - CachedFileHandlesMissCount: 42 (42)
>>> - DecompressionTime: 1.480ms
>>> - MaxCompressedTextFileLength: 0
>>> - NumColumns: 6 (6)
>>> - NumDictFilteredRowGroups: 0 (0)
>>> - NumDisksAccessed: 4 (4)
>>> - NumRowGroups: 6 (6)
>>> - NumScannerThreadsStarted: 6 (6)
>>> - NumScannersWithNoReads: 0 (0)
>>> - NumStatsFilteredRowGroups: 0 (0)
>>> - PeakMemoryUsage: 6.42 MB (6730467)
>>> - PerReadThreadRawHdfsThroughput: 77.02 MB/sec
>>> - RemoteScanRanges: 0 (0)
>>> - RowBatchQueueGetWaitTime: 13.979ms
>>> - RowBatchQueuePutWaitTime: 0.000ns
>>> - RowsRead: 89.91K (89910)
>>> - RowsReturned: 42.68K (42681)
>>> - RowsReturnedRate: 2.71 M/sec
>>> - ScanRangesComplete: 6 (6)
>>> - ScannerThreadsInvoluntaryContextSwitches: 3 (3)
>>> - ScannerThreadsTotalWallClockTime: 227.825ms
>>>
>>>
>>
Re: Estimate peak memory VS used peak memory
Posted by Alexander Behm <al...@cloudera.com>.
The planner memory estimates are conservative and sometimes extremely
inaccurate. In their current form, they are rarely appropriate for
admission decisions.
The recommended practice for memory-based admission control it to set a
mem_limit for every query. You can make this easier by setting up different
pools with different mem_limits, e.g. a small/medium/big queries pool or
similar.
On Thu, Feb 22, 2018 at 3:00 PM, Mostafa Mokhtar <mm...@cloudera.com>
wrote:
> It is recommended to set a per query memory limit as part of admission and
> not rely on estimates as they are sometimes inaccurate.
> Can you please include the full query profile?
>
>
> On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <fa...@gmail.com>
> wrote:
>
>> Hi Mostafa,
>>
>> It's not a specific query, almost all the query has such differene
>> between the 2 values.
>>
>> I can see even queries showing the estimate per node is 8.2 Tib
>>
>>
>> - User: psanalytics
>> - Database: default
>> - Query Type: QUERY
>> - Coordinator: slpr-dhc014.lpdomain.com
>> <http://slvr-cmc101:7180/cmf/hardware/hosts/22/status>
>> - Duration: 6.48s
>> - Rows Produced: 708
>> - Estimated per Node Peak Memory: 8.2 TiB
>> - Per Node Peak Memory Usage: 1.1 GiB
>> - Pool: root.impanalytics
>> - Threads: CPU Time: 20.1m
>>
>>
>>
>> How you can explain this behavior, and for sure i don't have 8.2 Tib
>> memory per node to give neither you.
>>
>> Can you please explain me how i should treat Estimated per Node Peak
>> Memory and if it used by impala for the resource pool and admission control
>> and what is the relation of this value to the default memory per node that
>> i setup for each resource pool?
>>
>> Below is part of one of the queries profile which the estimate per node
>> was ~ @GB and the used was 200MB per node.
>>
>>
>> Instance 744de1b6228736fa:b54bfaa70000000f (host=
>> slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child: 1s292ms, %
>> non-child: 88.82%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64
>> KB 5:2/289.57 KB 3:2/1012.83 KB
>> MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB
>> ThreadUsage(500.000ms): 1, 1, 1
>> - AverageThreadTokens: 1.00
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 7.17 MB (7521751)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 106.53 MB (111709581)
>> - RowsProduced: 32.83K (32826)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 1s297ms
>> - TotalStorageWaitTime: 234.356ms
>> - TotalThreadsInvoluntaryContextSwitches: 66 (66)
>> - TotalThreadsTotalWallClockTime: 1s715ms
>> - TotalThreadsSysTime: 5.998ms
>> - TotalThreadsUserTime: 124.975ms
>> - TotalThreadsVoluntaryContextSwitches: 303 (303)
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 1s394ms
>> - ExecTreeExecTime: 67.115ms
>> - OpenTime: 32.795ms
>> - ExecTreeOpenTime: 73.243us
>> - PrepareTime: 27.602ms
>> - ExecTreePrepareTime: 243.141us
>> DataStreamSender (dst_id=11):(Total: 38.747ms, non-child:
>> 38.747ms, % non-child: 100.00%)
>> - BytesSent: 39.71 MB (41643000)
>> - NetworkThroughput(*): 1.97 GB/sec
>> - OverallThroughput: 1.00 GB/sec
>> - PeakMemoryUsage: 59.38 KB (60800)
>> - RowsReturned: 32.83K (32826)
>> - SerializeBatchTime: 16.860ms
>> - TransmitDataRPCTime: 19.698ms
>> - UncompressedRowBatchSize: 77.58 MB (81350840)
>> CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child:
>> 100.00%)
>> - CodegenTime: 1.299ms
>> - CompileTime: 10.672ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 1.96 MB (2050180)
>> - NumFunctions: 16 (16)
>> - NumInstructions: 250 (250)
>> - OptimizationTime: 21.023ms
>> - PeakMemoryUsage: 125.00 KB (128000)
>> - PrepareTime: 24.116ms
>> SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, %
>> non-child: 17.85%)
>> - PeakMemoryUsage: 627.94 KB (643015)
>> - RowsReturned: 32.77K (32768)
>> - RowsReturnedRate: 486.81 K/sec
>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child:
>> 25.197ms, % non-child: 74.11%)
>> - BuildRows: 0 (0)
>> - BuildTime: 0.000ns
>> - PeakMemoryUsage: 24.00 KB (24576)
>> - ProbeRows: 32.83K (32826)
>> - ProbeTime: 0.000ns
>> - RowsReturned: 16.80M (16795311)
>> - RowsReturnedRate: 493.99 M/sec
>> Nested Loop Join Builder:
>> - PeakMemoryUsage: 8.00 KB (8192)
>> SINGULAR_ROW_SRC_NODE (id=7):
>> - PeakMemoryUsage: 0
>> - RowsReturned: 0 (0)
>> - RowsReturnedRate: 0
>> UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, %
>> non-child: 100.00%)
>> - AvgCollectionSize: 1.00
>> - MaxCollectionSize: 1 (1)
>> - MinCollectionSize: 1 (1)
>> - NumCollections: 32.83K (32826)
>> - PeakMemoryUsage: 0
>> - RowsReturned: 1 (1)
>> - RowsReturnedRate: 113.00 /sec
>> HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms, %
>> non-child: 100.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64
>> KB 5:2/289.57 KB 3:2/1012.83 KB
>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11 out of
>> 11
>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
>> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> File Formats: PARQUET/SNAPPY:66
>> BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB
>> - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ; Max:
>> 29.275ms ; Number of samples: 11)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 0.00
>> - BytesRead: 2.26 MB (2373738)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 2.26 MB (2373738)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 2.26 MB (2373738)
>> - CachedFileHandlesHitCount: 0 (0)
>> - CachedFileHandlesMissCount: 77 (77)
>> - DecompressionTime: 1.543ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 6 (6)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 8 (8)
>> - NumRowGroups: 11 (11)
>> - NumScannerThreadsStarted: 11 (11)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 6.51 MB (6821203)
>> - PerReadThreadRawHdfsThroughput: 764.51 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 18.825ms
>> - RowBatchQueuePutWaitTime: 0.000ns
>> - RowsRead: 73.46K (73464)
>> - RowsReturned: 32.83K (32826)
>> - RowsReturnedRate: 1.54 M/sec
>> - ScanRangesComplete: 11 (11)
>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>> - ScannerThreadsTotalWallClockTime: 287.702ms
>> - MaterializeTupleTime(*): 11.921ms
>> - ScannerThreadsSysTime: 999.000us
>> - ScannerThreadsUserTime: 26.990ms
>> - ScannerThreadsVoluntaryContextSwitches: 232 (232)
>> - TotalRawHdfsReadTime(*): 2.961ms
>> - TotalReadThroughput: 1.51 MB/sec
>> Instance 744de1b6228736fa:b54bfaa70000000d (host=
>> slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child: 1s254ms, %
>> non-child: 86.28%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
>> MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB
>> ThreadUsage(500.000ms): 1, 1, 1
>> - AverageThreadTokens: 1.00
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 5.86 MB (6145964)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 106.66 MB (111842766)
>> - RowsProduced: 30.34K (30340)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 1s256ms
>> - TotalStorageWaitTime: 106.530ms
>> - TotalThreadsInvoluntaryContextSwitches: 177 (177)
>> - TotalThreadsTotalWallClockTime: 1s557ms
>> - TotalThreadsSysTime: 6.998ms
>> - TotalThreadsUserTime: 143.976ms
>> - TotalThreadsVoluntaryContextSwitches: 136 (136)
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 1s390ms
>> - ExecTreeExecTime: 91.500ms
>> - OpenTime: 34.085ms
>> - ExecTreeOpenTime: 81.436us
>> - PrepareTime: 28.524ms
>> - ExecTreePrepareTime: 217.949us
>> DataStreamSender (dst_id=11):(Total: 48.674ms, non-child:
>> 48.674ms, % non-child: 100.00%)
>> - BytesSent: 36.61 MB (38388760)
>> - NetworkThroughput(*): 1.51 GB/sec
>> - OverallThroughput: 752.15 MB/sec
>> - PeakMemoryUsage: 59.38 KB (60800)
>> - RowsReturned: 30.34K (30340)
>> - SerializeBatchTime: 23.132ms
>> - TransmitDataRPCTime: 23.609ms
>> - UncompressedRowBatchSize: 71.70 MB (75187520)
>> CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child:
>> 100.00%)
>> - CodegenTime: 1.382ms
>> - CompileTime: 10.435ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 1.96 MB (2050180)
>> - NumFunctions: 16 (16)
>> - NumInstructions: 250 (250)
>> - OptimizationTime: 22.325ms
>> - PeakMemoryUsage: 125.00 KB (128000)
>> - PrepareTime: 25.459ms
>> SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, %
>> non-child: 22.90%)
>> - PeakMemoryUsage: 699.55 KB (716335)
>> - RowsReturned: 29.70K (29696)
>> - RowsReturnedRate: 323.80 K/sec
>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child:
>> 40.263ms, % non-child: 74.95%)
>> - BuildRows: 0 (0)
>> - BuildTime: 0.000ns
>> - PeakMemoryUsage: 24.00 KB (24576)
>> - ProbeRows: 30.34K (30340)
>> - ProbeTime: 0.000ns
>> - RowsReturned: 15.43M (15426890)
>> - RowsReturnedRate: 287.16 M/sec
>> Nested Loop Join Builder:
>> - PeakMemoryUsage: 8.00 KB (8192)
>> SINGULAR_ROW_SRC_NODE (id=7):
>> - PeakMemoryUsage: 0
>> - RowsReturned: 0 (0)
>> - RowsReturnedRate: 0
>> UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms, %
>> non-child: 100.00%)
>> - AvgCollectionSize: 1.00
>> - MaxCollectionSize: 1 (1)
>> - MinCollectionSize: 1 (1)
>> - NumCollections: 30.34K (30340)
>> - PeakMemoryUsage: 0
>> - RowsReturned: 1 (1)
>> - RowsReturnedRate: 74.00 /sec
>> HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms, %
>> non-child: 100.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out of 5
>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
>> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> File Formats: PARQUET/SNAPPY:30
>> BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB
>> - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ; Max:
>> 28.687ms ; Number of samples: 5)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 0.00
>> - BytesRead: 1.56 MB (1630546)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 1.56 MB (1630546)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 1.56 MB (1630546)
>> - CachedFileHandlesHitCount: 0 (0)
>> - CachedFileHandlesMissCount: 35 (35)
>> - DecompressionTime: 1.267ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 6 (6)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 4 (4)
>> - NumRowGroups: 5 (5)
>> - NumScannerThreadsStarted: 5 (5)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 5.59 MB (5859290)
>> - PerReadThreadRawHdfsThroughput: 920.32 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 15.335ms
>> - RowBatchQueuePutWaitTime: 0.000ns
>> - RowsRead: 64.69K (64692)
>> - RowsReturned: 30.34K (30340)
>> - RowsReturnedRate: 1.79 M/sec
>> - ScanRangesComplete: 5 (5)
>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>> - ScannerThreadsTotalWallClockTime: 132.548ms
>> - MaterializeTupleTime(*): 10.585ms
>> - ScannerThreadsSysTime: 999.000us
>> - ScannerThreadsUserTime: 18.995ms
>> - ScannerThreadsVoluntaryContextSwitches: 79 (79)
>> - TotalRawHdfsReadTime(*): 1.689ms
>> - TotalReadThroughput: 1.04 MB/sec
>> Instance 744de1b6228736fa:b54bfaa700000002 (host=
>> slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child: 1s255ms, %
>> non-child: 86.41%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
>> MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB
>> ThreadUsage(500.000ms): 1, 1, 1
>> - AverageThreadTokens: 1.00
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 6.17 MB (6473729)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 107.32 MB (112537640)
>> - RowsProduced: 29.97K (29969)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 1s260ms
>> - TotalStorageWaitTime: 130.815ms
>> - TotalThreadsInvoluntaryContextSwitches: 15 (15)
>> - TotalThreadsTotalWallClockTime: 1s586ms
>> - TotalThreadsSysTime: 11.995ms
>> - TotalThreadsUserTime: 146.975ms
>> - TotalThreadsVoluntaryContextSwitches: 197 (197)
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 1s354ms
>> - ExecTreeExecTime: 67.791ms
>> - OpenTime: 52.012ms
>> - ExecTreeOpenTime: 101.232us
>> - PrepareTime: 45.946ms
>> - ExecTreePrepareTime: 309.644us
>> DataStreamSender (dst_id=11):(Total: 38.995ms, non-child:
>> 38.995ms, % non-child: 100.00%)
>> - BytesSent: 35.95 MB (37697260)
>> - NetworkThroughput(*): 2.14 GB/sec
>> - OverallThroughput: 921.92 MB/sec
>> - PeakMemoryUsage: 59.38 KB (60800)
>> - RowsReturned: 29.97K (29969)
>> - SerializeBatchTime: 18.055ms
>> - TransmitDataRPCTime: 16.428ms
>> - UncompressedRowBatchSize: 70.83 MB (74269800)
>> CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child:
>> 100.00%)
>> - CodegenTime: 2.224ms
>> - CompileTime: 15.517ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 1.96 MB (2050180)
>> - NumFunctions: 16 (16)
>> - NumInstructions: 250 (250)
>> - OptimizationTime: 35.099ms
>> - PeakMemoryUsage: 125.00 KB (128000)
>> - PrepareTime: 38.847ms
>> SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, %
>> non-child: 18.53%)
>> - PeakMemoryUsage: 636.17 KB (651439)
>> - RowsReturned: 29.70K (29696)
>> - RowsReturnedRate: 435.89 K/sec
>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child:
>> 27.325ms, % non-child: 77.31%)
>> - BuildRows: 0 (0)
>> - BuildTime: 0.000ns
>> - PeakMemoryUsage: 24.00 KB (24576)
>> - ProbeRows: 29.97K (29969)
>> - ProbeTime: 0.000ns
>> - RowsReturned: 15.26M (15256601)
>> - RowsReturnedRate: 431.62 M/sec
>> Nested Loop Join Builder:
>> - PeakMemoryUsage: 8.00 KB (8192)
>> SINGULAR_ROW_SRC_NODE (id=7):
>> - PeakMemoryUsage: 0
>> - RowsReturned: 0 (0)
>> - RowsReturnedRate: 0
>> UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, %
>> non-child: 100.00%)
>> - AvgCollectionSize: 1.00
>> - MaxCollectionSize: 1 (1)
>> - MinCollectionSize: 1 (1)
>> - NumCollections: 29.97K (29969)
>> - PeakMemoryUsage: 0
>> - RowsReturned: 1 (1)
>> - RowsReturnedRate: 124.00 /sec
>> HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms, %
>> non-child: 100.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
>> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> File Formats: PARQUET/SNAPPY:42
>> BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB
>> - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ; Max:
>> 10.464ms ; Number of samples: 7)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 0.00
>> - BytesRead: 1.75 MB (1833560)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 1.75 MB (1833560)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 1.75 MB (1833560)
>> - CachedFileHandlesHitCount: 0 (0)
>> - CachedFileHandlesMissCount: 49 (49)
>> - DecompressionTime: 1.758ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 6 (6)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 5 (5)
>> - NumRowGroups: 7 (7)
>> - NumScannerThreadsStarted: 7 (7)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 5.82 MB (6098991)
>> - PerReadThreadRawHdfsThroughput: 621.09 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 17.938ms
>> - RowBatchQueuePutWaitTime: 0.000ns
>> - RowsRead: 65.17K (65166)
>> - RowsReturned: 29.97K (29969)
>> - RowsReturnedRate: 1.49 M/sec
>> - ScanRangesComplete: 7 (7)
>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
>> - ScannerThreadsTotalWallClockTime: 179.698ms
>> - MaterializeTupleTime(*): 13.922ms
>> - ScannerThreadsSysTime: 2.997ms
>> - ScannerThreadsUserTime: 29.993ms
>> - ScannerThreadsVoluntaryContextSwitches: 130 (130)
>> - TotalRawHdfsReadTime(*): 2.815ms
>> - TotalReadThroughput: 1.17 MB/sec
>> Instance 744de1b6228736fa:b54bfaa700000008 (host=
>> slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child: 1s305ms, %
>> non-child: 90.01%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
>> MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB
>> ThreadUsage(500.000ms): 1, 1, 1
>> - AverageThreadTokens: 1.00
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 6.69 MB (7017157)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 106.39 MB (111556990)
>> - RowsProduced: 42.68K (42681)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 1s315ms
>> - TotalStorageWaitTime: 201.411ms
>> - TotalThreadsInvoluntaryContextSwitches: 43 (43)
>> - TotalThreadsTotalWallClockTime: 1s658ms
>> - TotalThreadsSysTime: 6.997ms
>> - TotalThreadsUserTime: 108.980ms
>> - TotalThreadsVoluntaryContextSwitches: 179 (179)
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 1s411ms
>> - ExecTreeExecTime: 68.017ms
>> - OpenTime: 19.935ms
>> - ExecTreeOpenTime: 48.883us
>> - PrepareTime: 19.160ms
>> - ExecTreePrepareTime: 134.576us
>> DataStreamSender (dst_id=11):(Total: 40.671ms, non-child:
>> 40.671ms, % non-child: 100.00%)
>> - BytesSent: 51.36 MB (53856680)
>> - NetworkThroughput(*): 2.44 GB/sec
>> - OverallThroughput: 1.23 GB/sec
>> - PeakMemoryUsage: 59.38 KB (60800)
>> - RowsReturned: 42.68K (42681)
>> - SerializeBatchTime: 18.299ms
>> - TransmitDataRPCTime: 20.591ms
>> - UncompressedRowBatchSize: 100.88 MB (105782860)
>> CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child:
>> 100.00%)
>> - CodegenTime: 828.865us
>> - CompileTime: 6.170ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 1.96 MB (2050180)
>> - NumFunctions: 16 (16)
>> - NumInstructions: 250 (250)
>> - OptimizationTime: 12.988ms
>> - PeakMemoryUsage: 125.00 KB (128000)
>> - PrepareTime: 16.608ms
>> SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, %
>> non-child: 18.27%)
>> - PeakMemoryUsage: 689.95 KB (706507)
>> - RowsReturned: 41.98K (41984)
>> - RowsReturnedRate: 616.99 K/sec
>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child:
>> 29.422ms, % non-child: 73.86%)
>> - BuildRows: 0 (0)
>> - BuildTime: 0.000ns
>> - PeakMemoryUsage: 24.00 KB (24576)
>> - ProbeRows: 42.68K (42681)
>> - ProbeTime: 0.000ns
>> - RowsReturned: 21.76M (21760053)
>> - RowsReturnedRate: 546.22 M/sec
>> Nested Loop Join Builder:
>> - PeakMemoryUsage: 8.00 KB (8192)
>> SINGULAR_ROW_SRC_NODE (id=7):
>> - PeakMemoryUsage: 0
>> - RowsReturned: 0 (0)
>> - RowsReturnedRate: 0
>> UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms, %
>> non-child: 100.00%)
>> - AvgCollectionSize: 1.00
>> - MaxCollectionSize: 1 (1)
>> - MinCollectionSize: 1 (1)
>> - NumCollections: 42.68K (42681)
>> - PeakMemoryUsage: 0
>> - RowsReturned: 1 (1)
>> - RowsReturnedRate: 96.00 /sec
>> HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms, %
>> non-child: 100.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
>> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
>> File Formats: PARQUET/SNAPPY:36
>> BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB
>> - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ; Max:
>> 42.732ms ; Number of samples: 6)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 0.00
>> - BytesRead: 2.08 MB (2186159)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 2.08 MB (2186159)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 2.08 MB (2186159)
>> - CachedFileHandlesHitCount: 0 (0)
>> - CachedFileHandlesMissCount: 42 (42)
>> - DecompressionTime: 1.480ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 6 (6)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 4 (4)
>> - NumRowGroups: 6 (6)
>> - NumScannerThreadsStarted: 6 (6)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 6.42 MB (6730467)
>> - PerReadThreadRawHdfsThroughput: 77.02 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 13.979ms
>> - RowBatchQueuePutWaitTime: 0.000ns
>> - RowsRead: 89.91K (89910)
>> - RowsReturned: 42.68K (42681)
>> - RowsReturnedRate: 2.71 M/sec
>> - ScanRangesComplete: 6 (6)
>> - ScannerThreadsInvoluntaryContextSwitches: 3 (3)
>> - ScannerThreadsTotalWallClockTime: 227.825ms
>> - MaterializeTupleTime(*): 9.931ms
>> - ScannerThreadsSysTime: 1.998ms
>> - ScannerThreadsUserTime: 18.994ms
>> - ScannerThreadsVoluntaryContextSwitches: 85 (85)
>> - TotalRawHdfsReadTime(*): 27.069ms
>> - TotalReadThroughput: 1.39 MB/sec
>> Instance 744de1b6228736fa:b54bfaa700000005 (host=
>> slpr-dhc001.lpdomain.com:22000):(Total: 1s447ms, non-child: 1s301ms, %
>> non-child: 89.88%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 6:1/1.20 MB 1:2/2.50 MB 3:1/1.32 MB
>> MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 1.25 MB
>> ThreadUsage(500.000ms): 1, 1, 1
>> ...
>>
>> [Message clipped]
>
>
>
Re: Estimate peak memory VS used peak memory
Posted by Mostafa Mokhtar <mm...@cloudera.com>.
It is recommended to set a per query memory limit as part of admission and
not rely on estimates as they are sometimes inaccurate.
Can you please include the full query profile?
On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Mostafa,
>
> It's not a specific query, almost all the query has such differene between
> the 2 values.
>
> I can see even queries showing the estimate per node is 8.2 Tib
>
>
> - User: psanalytics
> - Database: default
> - Query Type: QUERY
> - Coordinator: slpr-dhc014.lpdomain.com
> <http://slvr-cmc101:7180/cmf/hardware/hosts/22/status>
> - Duration: 6.48s
> - Rows Produced: 708
> - Estimated per Node Peak Memory: 8.2 TiB
> - Per Node Peak Memory Usage: 1.1 GiB
> - Pool: root.impanalytics
> - Threads: CPU Time: 20.1m
>
>
>
> How you can explain this behavior, and for sure i don't have 8.2 Tib
> memory per node to give neither you.
>
> Can you please explain me how i should treat Estimated per Node Peak
> Memory and if it used by impala for the resource pool and admission control
> and what is the relation of this value to the default memory per node that
> i setup for each resource pool?
>
> Below is part of one of the queries profile which the estimate per node
> was ~ @GB and the used was 200MB per node.
>
>
> Instance 744de1b6228736fa:b54bfaa70000000f (host=slpr-dhc004.lpdomain.com:22000):(Total:
> 1s455ms, non-child: 1s292ms, % non-child: 88.82%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64
> KB 5:2/289.57 KB 3:2/1012.83 KB
> MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB
> ThreadUsage(500.000ms): 1, 1, 1
> - AverageThreadTokens: 1.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 7.17 MB (7521751)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 106.53 MB (111709581)
> - RowsProduced: 32.83K (32826)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 1s297ms
> - TotalStorageWaitTime: 234.356ms
> - TotalThreadsInvoluntaryContextSwitches: 66 (66)
> - TotalThreadsTotalWallClockTime: 1s715ms
> - TotalThreadsSysTime: 5.998ms
> - TotalThreadsUserTime: 124.975ms
> - TotalThreadsVoluntaryContextSwitches: 303 (303)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 1s394ms
> - ExecTreeExecTime: 67.115ms
> - OpenTime: 32.795ms
> - ExecTreeOpenTime: 73.243us
> - PrepareTime: 27.602ms
> - ExecTreePrepareTime: 243.141us
> DataStreamSender (dst_id=11):(Total: 38.747ms, non-child:
> 38.747ms, % non-child: 100.00%)
> - BytesSent: 39.71 MB (41643000)
> - NetworkThroughput(*): 1.97 GB/sec
> - OverallThroughput: 1.00 GB/sec
> - PeakMemoryUsage: 59.38 KB (60800)
> - RowsReturned: 32.83K (32826)
> - SerializeBatchTime: 16.860ms
> - TransmitDataRPCTime: 19.698ms
> - UncompressedRowBatchSize: 77.58 MB (81350840)
> CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child:
> 100.00%)
> - CodegenTime: 1.299ms
> - CompileTime: 10.672ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.96 MB (2050180)
> - NumFunctions: 16 (16)
> - NumInstructions: 250 (250)
> - OptimizationTime: 21.023ms
> - PeakMemoryUsage: 125.00 KB (128000)
> - PrepareTime: 24.116ms
> SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, %
> non-child: 17.85%)
> - PeakMemoryUsage: 627.94 KB (643015)
> - RowsReturned: 32.77K (32768)
> - RowsReturnedRate: 486.81 K/sec
> NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child:
> 25.197ms, % non-child: 74.11%)
> - BuildRows: 0 (0)
> - BuildTime: 0.000ns
> - PeakMemoryUsage: 24.00 KB (24576)
> - ProbeRows: 32.83K (32826)
> - ProbeTime: 0.000ns
> - RowsReturned: 16.80M (16795311)
> - RowsReturnedRate: 493.99 M/sec
> Nested Loop Join Builder:
> - PeakMemoryUsage: 8.00 KB (8192)
> SINGULAR_ROW_SRC_NODE (id=7):
> - PeakMemoryUsage: 0
> - RowsReturned: 0 (0)
> - RowsReturnedRate: 0
> UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, %
> non-child: 100.00%)
> - AvgCollectionSize: 1.00
> - MaxCollectionSize: 1 (1)
> - MinCollectionSize: 1 (1)
> - NumCollections: 32.83K (32826)
> - PeakMemoryUsage: 0
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 113.00 /sec
> HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms, %
> non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64
> KB 5:2/289.57 KB 3:2/1012.83 KB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11 out of
> 11
> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> File Formats: PARQUET/SNAPPY:66
> BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB
> - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ; Max:
> 29.275ms ; Number of samples: 11)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 0.00
> - BytesRead: 2.26 MB (2373738)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 2.26 MB (2373738)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 2.26 MB (2373738)
> - CachedFileHandlesHitCount: 0 (0)
> - CachedFileHandlesMissCount: 77 (77)
> - DecompressionTime: 1.543ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 6 (6)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 8 (8)
> - NumRowGroups: 11 (11)
> - NumScannerThreadsStarted: 11 (11)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 6.51 MB (6821203)
> - PerReadThreadRawHdfsThroughput: 764.51 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 18.825ms
> - RowBatchQueuePutWaitTime: 0.000ns
> - RowsRead: 73.46K (73464)
> - RowsReturned: 32.83K (32826)
> - RowsReturnedRate: 1.54 M/sec
> - ScanRangesComplete: 11 (11)
> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> - ScannerThreadsTotalWallClockTime: 287.702ms
> - MaterializeTupleTime(*): 11.921ms
> - ScannerThreadsSysTime: 999.000us
> - ScannerThreadsUserTime: 26.990ms
> - ScannerThreadsVoluntaryContextSwitches: 232 (232)
> - TotalRawHdfsReadTime(*): 2.961ms
> - TotalReadThroughput: 1.51 MB/sec
> Instance 744de1b6228736fa:b54bfaa70000000d
> (host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child:
> 1s254ms, % non-child: 86.28%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
> MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB
> ThreadUsage(500.000ms): 1, 1, 1
> - AverageThreadTokens: 1.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 5.86 MB (6145964)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 106.66 MB (111842766)
> - RowsProduced: 30.34K (30340)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 1s256ms
> - TotalStorageWaitTime: 106.530ms
> - TotalThreadsInvoluntaryContextSwitches: 177 (177)
> - TotalThreadsTotalWallClockTime: 1s557ms
> - TotalThreadsSysTime: 6.998ms
> - TotalThreadsUserTime: 143.976ms
> - TotalThreadsVoluntaryContextSwitches: 136 (136)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 1s390ms
> - ExecTreeExecTime: 91.500ms
> - OpenTime: 34.085ms
> - ExecTreeOpenTime: 81.436us
> - PrepareTime: 28.524ms
> - ExecTreePrepareTime: 217.949us
> DataStreamSender (dst_id=11):(Total: 48.674ms, non-child:
> 48.674ms, % non-child: 100.00%)
> - BytesSent: 36.61 MB (38388760)
> - NetworkThroughput(*): 1.51 GB/sec
> - OverallThroughput: 752.15 MB/sec
> - PeakMemoryUsage: 59.38 KB (60800)
> - RowsReturned: 30.34K (30340)
> - SerializeBatchTime: 23.132ms
> - TransmitDataRPCTime: 23.609ms
> - UncompressedRowBatchSize: 71.70 MB (75187520)
> CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child:
> 100.00%)
> - CodegenTime: 1.382ms
> - CompileTime: 10.435ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.96 MB (2050180)
> - NumFunctions: 16 (16)
> - NumInstructions: 250 (250)
> - OptimizationTime: 22.325ms
> - PeakMemoryUsage: 125.00 KB (128000)
> - PrepareTime: 25.459ms
> SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, %
> non-child: 22.90%)
> - PeakMemoryUsage: 699.55 KB (716335)
> - RowsReturned: 29.70K (29696)
> - RowsReturnedRate: 323.80 K/sec
> NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child:
> 40.263ms, % non-child: 74.95%)
> - BuildRows: 0 (0)
> - BuildTime: 0.000ns
> - PeakMemoryUsage: 24.00 KB (24576)
> - ProbeRows: 30.34K (30340)
> - ProbeTime: 0.000ns
> - RowsReturned: 15.43M (15426890)
> - RowsReturnedRate: 287.16 M/sec
> Nested Loop Join Builder:
> - PeakMemoryUsage: 8.00 KB (8192)
> SINGULAR_ROW_SRC_NODE (id=7):
> - PeakMemoryUsage: 0
> - RowsReturned: 0 (0)
> - RowsReturnedRate: 0
> UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms, %
> non-child: 100.00%)
> - AvgCollectionSize: 1.00
> - MaxCollectionSize: 1 (1)
> - MinCollectionSize: 1 (1)
> - NumCollections: 30.34K (30340)
> - PeakMemoryUsage: 0
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 74.00 /sec
> HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms, %
> non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out of 5
> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> File Formats: PARQUET/SNAPPY:30
> BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB
> - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ; Max:
> 28.687ms ; Number of samples: 5)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 0.00
> - BytesRead: 1.56 MB (1630546)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 1.56 MB (1630546)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 1.56 MB (1630546)
> - CachedFileHandlesHitCount: 0 (0)
> - CachedFileHandlesMissCount: 35 (35)
> - DecompressionTime: 1.267ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 6 (6)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 4 (4)
> - NumRowGroups: 5 (5)
> - NumScannerThreadsStarted: 5 (5)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 5.59 MB (5859290)
> - PerReadThreadRawHdfsThroughput: 920.32 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 15.335ms
> - RowBatchQueuePutWaitTime: 0.000ns
> - RowsRead: 64.69K (64692)
> - RowsReturned: 30.34K (30340)
> - RowsReturnedRate: 1.79 M/sec
> - ScanRangesComplete: 5 (5)
> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> - ScannerThreadsTotalWallClockTime: 132.548ms
> - MaterializeTupleTime(*): 10.585ms
> - ScannerThreadsSysTime: 999.000us
> - ScannerThreadsUserTime: 18.995ms
> - ScannerThreadsVoluntaryContextSwitches: 79 (79)
> - TotalRawHdfsReadTime(*): 1.689ms
> - TotalReadThroughput: 1.04 MB/sec
> Instance 744de1b6228736fa:b54bfaa700000002
> (host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child:
> 1s255ms, % non-child: 86.41%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
> MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB
> ThreadUsage(500.000ms): 1, 1, 1
> - AverageThreadTokens: 1.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 6.17 MB (6473729)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 107.32 MB (112537640)
> - RowsProduced: 29.97K (29969)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 1s260ms
> - TotalStorageWaitTime: 130.815ms
> - TotalThreadsInvoluntaryContextSwitches: 15 (15)
> - TotalThreadsTotalWallClockTime: 1s586ms
> - TotalThreadsSysTime: 11.995ms
> - TotalThreadsUserTime: 146.975ms
> - TotalThreadsVoluntaryContextSwitches: 197 (197)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 1s354ms
> - ExecTreeExecTime: 67.791ms
> - OpenTime: 52.012ms
> - ExecTreeOpenTime: 101.232us
> - PrepareTime: 45.946ms
> - ExecTreePrepareTime: 309.644us
> DataStreamSender (dst_id=11):(Total: 38.995ms, non-child:
> 38.995ms, % non-child: 100.00%)
> - BytesSent: 35.95 MB (37697260)
> - NetworkThroughput(*): 2.14 GB/sec
> - OverallThroughput: 921.92 MB/sec
> - PeakMemoryUsage: 59.38 KB (60800)
> - RowsReturned: 29.97K (29969)
> - SerializeBatchTime: 18.055ms
> - TransmitDataRPCTime: 16.428ms
> - UncompressedRowBatchSize: 70.83 MB (74269800)
> CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child:
> 100.00%)
> - CodegenTime: 2.224ms
> - CompileTime: 15.517ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.96 MB (2050180)
> - NumFunctions: 16 (16)
> - NumInstructions: 250 (250)
> - OptimizationTime: 35.099ms
> - PeakMemoryUsage: 125.00 KB (128000)
> - PrepareTime: 38.847ms
> SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, %
> non-child: 18.53%)
> - PeakMemoryUsage: 636.17 KB (651439)
> - RowsReturned: 29.70K (29696)
> - RowsReturnedRate: 435.89 K/sec
> NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child:
> 27.325ms, % non-child: 77.31%)
> - BuildRows: 0 (0)
> - BuildTime: 0.000ns
> - PeakMemoryUsage: 24.00 KB (24576)
> - ProbeRows: 29.97K (29969)
> - ProbeTime: 0.000ns
> - RowsReturned: 15.26M (15256601)
> - RowsReturnedRate: 431.62 M/sec
> Nested Loop Join Builder:
> - PeakMemoryUsage: 8.00 KB (8192)
> SINGULAR_ROW_SRC_NODE (id=7):
> - PeakMemoryUsage: 0
> - RowsReturned: 0 (0)
> - RowsReturnedRate: 0
> UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, %
> non-child: 100.00%)
> - AvgCollectionSize: 1.00
> - MaxCollectionSize: 1 (1)
> - MinCollectionSize: 1 (1)
> - NumCollections: 29.97K (29969)
> - PeakMemoryUsage: 0
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 124.00 /sec
> HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms, %
> non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> File Formats: PARQUET/SNAPPY:42
> BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB
> - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ; Max:
> 10.464ms ; Number of samples: 7)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 0.00
> - BytesRead: 1.75 MB (1833560)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 1.75 MB (1833560)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 1.75 MB (1833560)
> - CachedFileHandlesHitCount: 0 (0)
> - CachedFileHandlesMissCount: 49 (49)
> - DecompressionTime: 1.758ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 6 (6)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 5 (5)
> - NumRowGroups: 7 (7)
> - NumScannerThreadsStarted: 7 (7)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 5.82 MB (6098991)
> - PerReadThreadRawHdfsThroughput: 621.09 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 17.938ms
> - RowBatchQueuePutWaitTime: 0.000ns
> - RowsRead: 65.17K (65166)
> - RowsReturned: 29.97K (29969)
> - RowsReturnedRate: 1.49 M/sec
> - ScanRangesComplete: 7 (7)
> - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> - ScannerThreadsTotalWallClockTime: 179.698ms
> - MaterializeTupleTime(*): 13.922ms
> - ScannerThreadsSysTime: 2.997ms
> - ScannerThreadsUserTime: 29.993ms
> - ScannerThreadsVoluntaryContextSwitches: 130 (130)
> - TotalRawHdfsReadTime(*): 2.815ms
> - TotalReadThroughput: 1.17 MB/sec
> Instance 744de1b6228736fa:b54bfaa700000008
> (host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child:
> 1s305ms, % non-child: 90.01%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
> MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB
> ThreadUsage(500.000ms): 1, 1, 1
> - AverageThreadTokens: 1.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 6.69 MB (7017157)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 106.39 MB (111556990)
> - RowsProduced: 42.68K (42681)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 1s315ms
> - TotalStorageWaitTime: 201.411ms
> - TotalThreadsInvoluntaryContextSwitches: 43 (43)
> - TotalThreadsTotalWallClockTime: 1s658ms
> - TotalThreadsSysTime: 6.997ms
> - TotalThreadsUserTime: 108.980ms
> - TotalThreadsVoluntaryContextSwitches: 179 (179)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 1s411ms
> - ExecTreeExecTime: 68.017ms
> - OpenTime: 19.935ms
> - ExecTreeOpenTime: 48.883us
> - PrepareTime: 19.160ms
> - ExecTreePrepareTime: 134.576us
> DataStreamSender (dst_id=11):(Total: 40.671ms, non-child:
> 40.671ms, % non-child: 100.00%)
> - BytesSent: 51.36 MB (53856680)
> - NetworkThroughput(*): 2.44 GB/sec
> - OverallThroughput: 1.23 GB/sec
> - PeakMemoryUsage: 59.38 KB (60800)
> - RowsReturned: 42.68K (42681)
> - SerializeBatchTime: 18.299ms
> - TransmitDataRPCTime: 20.591ms
> - UncompressedRowBatchSize: 100.88 MB (105782860)
> CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child:
> 100.00%)
> - CodegenTime: 828.865us
> - CompileTime: 6.170ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.96 MB (2050180)
> - NumFunctions: 16 (16)
> - NumInstructions: 250 (250)
> - OptimizationTime: 12.988ms
> - PeakMemoryUsage: 125.00 KB (128000)
> - PrepareTime: 16.608ms
> SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, %
> non-child: 18.27%)
> - PeakMemoryUsage: 689.95 KB (706507)
> - RowsReturned: 41.98K (41984)
> - RowsReturnedRate: 616.99 K/sec
> NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child:
> 29.422ms, % non-child: 73.86%)
> - BuildRows: 0 (0)
> - BuildTime: 0.000ns
> - PeakMemoryUsage: 24.00 KB (24576)
> - ProbeRows: 42.68K (42681)
> - ProbeTime: 0.000ns
> - RowsReturned: 21.76M (21760053)
> - RowsReturnedRate: 546.22 M/sec
> Nested Loop Join Builder:
> - PeakMemoryUsage: 8.00 KB (8192)
> SINGULAR_ROW_SRC_NODE (id=7):
> - PeakMemoryUsage: 0
> - RowsReturned: 0 (0)
> - RowsReturnedRate: 0
> UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms, %
> non-child: 100.00%)
> - AvgCollectionSize: 1.00
> - MaxCollectionSize: 1 (1)
> - MinCollectionSize: 1 (1)
> - NumCollections: 42.68K (42681)
> - PeakMemoryUsage: 0
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 96.00 /sec
> HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms, %
> non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
> File Formats: PARQUET/SNAPPY:36
> BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB
> - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ; Max:
> 42.732ms ; Number of samples: 6)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 0.00
> - BytesRead: 2.08 MB (2186159)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 2.08 MB (2186159)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 2.08 MB (2186159)
> - CachedFileHandlesHitCount: 0 (0)
> - CachedFileHandlesMissCount: 42 (42)
> - DecompressionTime: 1.480ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 6 (6)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 4 (4)
> - NumRowGroups: 6 (6)
> - NumScannerThreadsStarted: 6 (6)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 6.42 MB (6730467)
> - PerReadThreadRawHdfsThroughput: 77.02 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 13.979ms
> - RowBatchQueuePutWaitTime: 0.000ns
> - RowsRead: 89.91K (89910)
> - RowsReturned: 42.68K (42681)
> - RowsReturnedRate: 2.71 M/sec
> - ScanRangesComplete: 6 (6)
> - ScannerThreadsInvoluntaryContextSwitches: 3 (3)
> - ScannerThreadsTotalWallClockTime: 227.825ms
> - MaterializeTupleTime(*): 9.931ms
> - ScannerThreadsSysTime: 1.998ms
> - ScannerThreadsUserTime: 18.994ms
> - ScannerThreadsVoluntaryContextSwitches: 85 (85)
> - TotalRawHdfsReadTime(*): 27.069ms
> - TotalReadThroughput: 1.39 MB/sec
> Instance 744de1b6228736fa:b54bfaa700000005
> (host=slpr-dhc001.lpdomain.com:22000):(Total: 1s447ms, non-child:
> 1s301ms, % non-child: 89.88%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 6:1/1.20 MB 1:2/2.50 MB 3:1/1.32 MB
> MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 1.25 MB
> ThreadUsage(500.000ms): 1, 1, 1
> ...
>
> [Message clipped]
Re: Estimate peak memory VS used peak memory
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Mostafa,
It's not a specific query, almost all the query has such differene between
the 2 values.
I can see even queries showing the estimate per node is 8.2 Tib
- User: psanalytics
- Database: default
- Query Type: QUERY
- Coordinator: slpr-dhc014.lpdomain.com
<http://slvr-cmc101:7180/cmf/hardware/hosts/22/status>
- Duration: 6.48s
- Rows Produced: 708
- Estimated per Node Peak Memory: 8.2 TiB
- Per Node Peak Memory Usage: 1.1 GiB
- Pool: root.impanalytics
- Threads: CPU Time: 20.1m
How you can explain this behavior, and for sure i don't have 8.2 Tib memory
per node to give neither you.
Can you please explain me how i should treat Estimated per Node Peak Memory
and if it used by impala for the resource pool and admission control and
what is the relation of this value to the default memory per node that i
setup for each resource pool?
Below is part of one of the queries profile which the estimate per node was
~ @GB and the used was 200MB per node.
Instance 744de1b6228736fa:b54bfaa70000000f
(host=slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child: 1s292ms,
% non-child: 88.82%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 6:1/1.20
MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64 KB
5:2/289.57 KB 3:2/1012.83 KB
MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 7.17 MB (7521751)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 106.53 MB (111709581)
- RowsProduced: 32.83K (32826)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s297ms
- TotalStorageWaitTime: 234.356ms
- TotalThreadsInvoluntaryContextSwitches: 66 (66)
- TotalThreadsTotalWallClockTime: 1s715ms
- TotalThreadsSysTime: 5.998ms
- TotalThreadsUserTime: 124.975ms
- TotalThreadsVoluntaryContextSwitches: 303 (303)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s394ms
- ExecTreeExecTime: 67.115ms
- OpenTime: 32.795ms
- ExecTreeOpenTime: 73.243us
- PrepareTime: 27.602ms
- ExecTreePrepareTime: 243.141us
DataStreamSender (dst_id=11):(Total: 38.747ms, non-child: 38.747ms,
% non-child: 100.00%)
- BytesSent: 39.71 MB (41643000)
- NetworkThroughput(*): 1.97 GB/sec
- OverallThroughput: 1.00 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 32.83K (32826)
- SerializeBatchTime: 16.860ms
- TransmitDataRPCTime: 19.698ms
- UncompressedRowBatchSize: 77.58 MB (81350840)
CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child: 100.00%)
- CodegenTime: 1.299ms
- CompileTime: 10.672ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 21.023ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 24.116ms
SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, %
non-child: 17.85%)
- PeakMemoryUsage: 627.94 KB (643015)
- RowsReturned: 32.77K (32768)
- RowsReturnedRate: 486.81 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child:
25.197ms, % non-child: 74.11%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 32.83K (32826)
- ProbeTime: 0.000ns
- RowsReturned: 16.80M (16795311)
- RowsReturnedRate: 493.99 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 32.83K (32826)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 113.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64
KB 5:2/289.57 KB 3:2/1012.83 KB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11 out of 11
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/SNAPPY:66
BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB
- FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ; Max:
29.275ms ; Number of samples: 11)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 2.26 MB (2373738)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 2.26 MB (2373738)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 2.26 MB (2373738)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 77 (77)
- DecompressionTime: 1.543ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 8 (8)
- NumRowGroups: 11 (11)
- NumScannerThreadsStarted: 11 (11)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 6.51 MB (6821203)
- PerReadThreadRawHdfsThroughput: 764.51 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 18.825ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 73.46K (73464)
- RowsReturned: 32.83K (32826)
- RowsReturnedRate: 1.54 M/sec
- ScanRangesComplete: 11 (11)
- ScannerThreadsInvoluntaryContextSwitches: 1 (1)
- ScannerThreadsTotalWallClockTime: 287.702ms
- MaterializeTupleTime(*): 11.921ms
- ScannerThreadsSysTime: 999.000us
- ScannerThreadsUserTime: 26.990ms
- ScannerThreadsVoluntaryContextSwitches: 232 (232)
- TotalRawHdfsReadTime(*): 2.961ms
- TotalReadThroughput: 1.51 MB/sec
Instance 744de1b6228736fa:b54bfaa70000000d
(host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child: 1s254ms,
% non-child: 86.28%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/1.32
MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 5.86 MB (6145964)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 106.66 MB (111842766)
- RowsProduced: 30.34K (30340)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s256ms
- TotalStorageWaitTime: 106.530ms
- TotalThreadsInvoluntaryContextSwitches: 177 (177)
- TotalThreadsTotalWallClockTime: 1s557ms
- TotalThreadsSysTime: 6.998ms
- TotalThreadsUserTime: 143.976ms
- TotalThreadsVoluntaryContextSwitches: 136 (136)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s390ms
- ExecTreeExecTime: 91.500ms
- OpenTime: 34.085ms
- ExecTreeOpenTime: 81.436us
- PrepareTime: 28.524ms
- ExecTreePrepareTime: 217.949us
DataStreamSender (dst_id=11):(Total: 48.674ms, non-child: 48.674ms,
% non-child: 100.00%)
- BytesSent: 36.61 MB (38388760)
- NetworkThroughput(*): 1.51 GB/sec
- OverallThroughput: 752.15 MB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 30.34K (30340)
- SerializeBatchTime: 23.132ms
- TransmitDataRPCTime: 23.609ms
- UncompressedRowBatchSize: 71.70 MB (75187520)
CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child: 100.00%)
- CodegenTime: 1.382ms
- CompileTime: 10.435ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 22.325ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 25.459ms
SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, %
non-child: 22.90%)
- PeakMemoryUsage: 699.55 KB (716335)
- RowsReturned: 29.70K (29696)
- RowsReturnedRate: 323.80 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child:
40.263ms, % non-child: 74.95%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 30.34K (30340)
- ProbeTime: 0.000ns
- RowsReturned: 15.43M (15426890)
- RowsReturnedRate: 287.16 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 30.34K (30340)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 74.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out of 5
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/SNAPPY:30
BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB
- FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ; Max:
28.687ms ; Number of samples: 5)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.56 MB (1630546)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.56 MB (1630546)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.56 MB (1630546)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 35 (35)
- DecompressionTime: 1.267ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 4 (4)
- NumRowGroups: 5 (5)
- NumScannerThreadsStarted: 5 (5)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 5.59 MB (5859290)
- PerReadThreadRawHdfsThroughput: 920.32 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 15.335ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 64.69K (64692)
- RowsReturned: 30.34K (30340)
- RowsReturnedRate: 1.79 M/sec
- ScanRangesComplete: 5 (5)
- ScannerThreadsInvoluntaryContextSwitches: 1 (1)
- ScannerThreadsTotalWallClockTime: 132.548ms
- MaterializeTupleTime(*): 10.585ms
- ScannerThreadsSysTime: 999.000us
- ScannerThreadsUserTime: 18.995ms
- ScannerThreadsVoluntaryContextSwitches: 79 (79)
- TotalRawHdfsReadTime(*): 1.689ms
- TotalReadThroughput: 1.04 MB/sec
Instance 744de1b6228736fa:b54bfaa700000002
(host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child: 1s255ms,
% non-child: 86.41%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 6.17 MB (6473729)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 107.32 MB (112537640)
- RowsProduced: 29.97K (29969)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s260ms
- TotalStorageWaitTime: 130.815ms
- TotalThreadsInvoluntaryContextSwitches: 15 (15)
- TotalThreadsTotalWallClockTime: 1s586ms
- TotalThreadsSysTime: 11.995ms
- TotalThreadsUserTime: 146.975ms
- TotalThreadsVoluntaryContextSwitches: 197 (197)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s354ms
- ExecTreeExecTime: 67.791ms
- OpenTime: 52.012ms
- ExecTreeOpenTime: 101.232us
- PrepareTime: 45.946ms
- ExecTreePrepareTime: 309.644us
DataStreamSender (dst_id=11):(Total: 38.995ms, non-child: 38.995ms,
% non-child: 100.00%)
- BytesSent: 35.95 MB (37697260)
- NetworkThroughput(*): 2.14 GB/sec
- OverallThroughput: 921.92 MB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 29.97K (29969)
- SerializeBatchTime: 18.055ms
- TransmitDataRPCTime: 16.428ms
- UncompressedRowBatchSize: 70.83 MB (74269800)
CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child: 100.00%)
- CodegenTime: 2.224ms
- CompileTime: 15.517ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 35.099ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 38.847ms
SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, %
non-child: 18.53%)
- PeakMemoryUsage: 636.17 KB (651439)
- RowsReturned: 29.70K (29696)
- RowsReturnedRate: 435.89 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child:
27.325ms, % non-child: 77.31%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 29.97K (29969)
- ProbeTime: 0.000ns
- RowsReturned: 15.26M (15256601)
- RowsReturnedRate: 431.62 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 29.97K (29969)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 124.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/SNAPPY:42
BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB
- FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ; Max:
10.464ms ; Number of samples: 7)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.75 MB (1833560)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.75 MB (1833560)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.75 MB (1833560)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 49 (49)
- DecompressionTime: 1.758ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 5 (5)
- NumRowGroups: 7 (7)
- NumScannerThreadsStarted: 7 (7)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 5.82 MB (6098991)
- PerReadThreadRawHdfsThroughput: 621.09 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 17.938ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 65.17K (65166)
- RowsReturned: 29.97K (29969)
- RowsReturnedRate: 1.49 M/sec
- ScanRangesComplete: 7 (7)
- ScannerThreadsInvoluntaryContextSwitches: 1 (1)
- ScannerThreadsTotalWallClockTime: 179.698ms
- MaterializeTupleTime(*): 13.922ms
- ScannerThreadsSysTime: 2.997ms
- ScannerThreadsUserTime: 29.993ms
- ScannerThreadsVoluntaryContextSwitches: 130 (130)
- TotalRawHdfsReadTime(*): 2.815ms
- TotalReadThroughput: 1.17 MB/sec
Instance 744de1b6228736fa:b54bfaa700000008
(host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child: 1s305ms,
% non-child: 90.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 7:1/1.08
MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 6.69 MB (7017157)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 106.39 MB (111556990)
- RowsProduced: 42.68K (42681)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s315ms
- TotalStorageWaitTime: 201.411ms
- TotalThreadsInvoluntaryContextSwitches: 43 (43)
- TotalThreadsTotalWallClockTime: 1s658ms
- TotalThreadsSysTime: 6.997ms
- TotalThreadsUserTime: 108.980ms
- TotalThreadsVoluntaryContextSwitches: 179 (179)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s411ms
- ExecTreeExecTime: 68.017ms
- OpenTime: 19.935ms
- ExecTreeOpenTime: 48.883us
- PrepareTime: 19.160ms
- ExecTreePrepareTime: 134.576us
DataStreamSender (dst_id=11):(Total: 40.671ms, non-child: 40.671ms,
% non-child: 100.00%)
- BytesSent: 51.36 MB (53856680)
- NetworkThroughput(*): 2.44 GB/sec
- OverallThroughput: 1.23 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 42.68K (42681)
- SerializeBatchTime: 18.299ms
- TransmitDataRPCTime: 20.591ms
- UncompressedRowBatchSize: 100.88 MB (105782860)
CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child: 100.00%)
- CodegenTime: 828.865us
- CompileTime: 6.170ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 12.988ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 16.608ms
SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, %
non-child: 18.27%)
- PeakMemoryUsage: 689.95 KB (706507)
- RowsReturned: 41.98K (41984)
- RowsReturnedRate: 616.99 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child:
29.422ms, % non-child: 73.86%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 42.68K (42681)
- ProbeTime: 0.000ns
- RowsReturned: 21.76M (21760053)
- RowsReturnedRate: 546.22 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 42.68K (42681)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 96.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/SNAPPY:36
BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB
- FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ; Max:
42.732ms ; Number of samples: 6)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 2.08 MB (2186159)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 2.08 MB (2186159)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 2.08 MB (2186159)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 42 (42)
- DecompressionTime: 1.480ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 4 (4)
- NumRowGroups: 6 (6)
- NumScannerThreadsStarted: 6 (6)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 6.42 MB (6730467)
- PerReadThreadRawHdfsThroughput: 77.02 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 13.979ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 89.91K (89910)
- RowsReturned: 42.68K (42681)
- RowsReturnedRate: 2.71 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 3 (3)
- ScannerThreadsTotalWallClockTime: 227.825ms
- MaterializeTupleTime(*): 9.931ms
- ScannerThreadsSysTime: 1.998ms
- ScannerThreadsUserTime: 18.994ms
- ScannerThreadsVoluntaryContextSwitches: 85 (85)
- TotalRawHdfsReadTime(*): 27.069ms
- TotalReadThroughput: 1.39 MB/sec
Instance 744de1b6228736fa:b54bfaa700000005
(host=slpr-dhc001.lpdomain.com:22000):(Total: 1s447ms, non-child: 1s301ms,
% non-child: 89.88%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 6:1/1.20
MB 1:2/2.50 MB 3:1/1.32 MB
MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 1.25 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 6.32 MB (6623566)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 106.35 MB (111518199)
- RowsProduced: 33.03K (33027)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s311ms
- TotalStorageWaitTime: 76.832ms
- TotalThreadsInvoluntaryContextSwitches: 7 (7)
- TotalThreadsTotalWallClockTime: 1s518ms
- TotalThreadsSysTime: 8.997ms
- TotalThreadsUserTime: 102.983ms
- TotalThreadsVoluntaryContextSwitches: 128 (128)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s389ms
- ExecTreeExecTime: 55.767ms
- OpenTime: 30.577ms
- ExecTreeOpenTime: 71.274us
- PrepareTime: 27.610ms
- ExecTreePrepareTime: 194.662us
DataStreamSender (dst_id=11):(Total: 36.270ms, non-child: 36.270ms,
% non-child: 100.00%)
- BytesSent: 39.69 MB (41621640)
- NetworkThroughput(*): 2.07 GB/sec
- OverallThroughput: 1.07 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 33.03K (33027)
- SerializeBatchTime: 15.319ms
- TransmitDataRPCTime: 18.709ms
- UncompressedRowBatchSize: 78.06 MB (81851440)
CodeGen:(Total: 54.301ms, non-child: 54.301ms, % non-child: 100.00%)
- CodegenTime: 1.269ms
- CompileTime: 9.193ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 20.433ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 24.055ms
SUBPLAN_NODE (id=6):(Total: 55.933ms, non-child: 10.883ms, %
non-child: 19.46%)
- PeakMemoryUsage: 702.06 KB (718909)
- RowsReturned: 32.77K (32768)
- RowsReturnedRate: 585.83 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 28.594ms, non-child:
22.150ms, % non-child: 77.46%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 33.03K (33027)
- ProbeTime: 0.000ns
- RowsReturned: 16.83M (16827270)
- RowsReturnedRate: 588.47 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 6.444ms, non-child: 6.444ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 33.03K (33027)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 155.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 16.455ms, non-child: 16.455ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
6:1/1.20 MB 1:2/2.50 MB 3:1/1.32 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4 out of 4
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/SNAPPY:24
BytesRead(500.000ms): 1.54 MB, 1.54 MB, 1.54 MB
- FooterProcessingTime: (Avg: 6.502ms ; Min: 5.361ms ; Max:
8.670ms ; Number of samples: 4)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.54 MB (1616732)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.54 MB (1616732)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.54 MB (1616732)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 28 (28)
- DecompressionTime: 1.336ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 3 (3)
- NumRowGroups: 4 (4)
- NumScannerThreadsStarted: 4 (4)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 6.04 MB (6336892)
- PerReadThreadRawHdfsThroughput: 102.07 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 15.237ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 69.25K (69254)
- RowsReturned: 33.03K (33027)
- RowsReturnedRate: 2.01 M/sec
- ScanRangesComplete: 4 (4)
- ScannerThreadsInvoluntaryContextSwitches: 2 (2)
- ScannerThreadsTotalWallClockTime: 98.879ms
- MaterializeTupleTime(*): 9.955ms
- ScannerThreadsSysTime: 1.998ms
- ScannerThreadsUserTime: 16.996ms
- ScannerThreadsVoluntaryContextSwitches: 59 (59)
- TotalRawHdfsReadTime(*): 15.106ms
- TotalReadThroughput: 1.03 MB/sec
Instance 744de1b6228736fa:b54bfaa700000006
(host=slpr-dhc007.lpdomain.com:22000):(Total: 1s444ms, non-child: 1s315ms,
% non-child: 91.12%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 3:1/1.29
MB 2:2/4.14 MB 6:1/444.78 KB
MemoryUsage(500.000ms): 5.10 MB, 5.01 MB, 455.66 KB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 7.24 MB (7591679)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 105.91 MB (111055430)
- RowsProduced: 38.66K (38662)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s320ms
- TotalStorageWaitTime: 69.771ms
- TotalThreadsInvoluntaryContextSwitches: 94 (94)
- TotalThreadsTotalWallClockTime: 1s514ms
- TotalThreadsSysTime: 7.999ms
- TotalThreadsUserTime: 93.983ms
- TotalThreadsVoluntaryContextSwitches: 124 (124)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s404ms
- ExecTreeExecTime: 54.925ms
- OpenTime: 19.389ms
- ExecTreeOpenTime: 53.610us
- PrepareTime: 20.398ms
- ExecTreePrepareTime: 146.339us
DataStreamSender (dst_id=11):(Total: 37.498ms, non-child: 37.498ms,
% non-child: 100.00%)
- BytesSent: 46.98 MB (49262840)
- NetworkThroughput(*): 2.50 GB/sec
- OverallThroughput: 1.22 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 38.66K (38662)
- SerializeBatchTime: 16.134ms
- TransmitDataRPCTime: 18.317ms
- UncompressedRowBatchSize: 91.38 MB (95817060)
CodeGen:(Total: 35.695ms, non-child: 35.695ms, % non-child: 100.00%)
- CodegenTime: 814.514us
- CompileTime: 5.998ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 12.677ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 16.588ms
SUBPLAN_NODE (id=6):(Total: 55.034ms, non-child: 10.012ms, %
non-child: 18.19%)
- PeakMemoryUsage: 1.27 MB (1332424)
- RowsReturned: 37.89K (37888)
- RowsReturnedRate: 688.44 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.183ms, non-child:
24.943ms, % non-child: 75.17%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 38.66K (38662)
- ProbeTime: 0.000ns
- RowsReturned: 19.72M (19717525)
- RowsReturnedRate: 594.20 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 8.240ms, non-child: 8.240ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 38.66K (38662)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 121.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 11.838ms, non-child: 11.838ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
3:1/1.29 MB 2:2/4.14 MB 6:1/444.78 KB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4 out of 4
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/SNAPPY:24
BytesRead(500.000ms): 1.73 MB, 1.73 MB
- FooterProcessingTime: (Avg: 12.691ms ; Min: 5.058ms ; Max:
34.003ms ; Number of samples: 4)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.73 MB (1810031)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.73 MB (1810031)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.73 MB (1810031)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 28 (28)
- DecompressionTime: 1.645ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 3 (3)
- NumRowGroups: 4 (4)
- NumScannerThreadsStarted: 4 (4)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 6.97 MB (7305005)
- PerReadThreadRawHdfsThroughput: 743.47 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 10.650ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 82.44K (82442)
- RowsReturned: 38.66K (38662)
- RowsReturnedRate: 3.27 M/sec
- ScanRangesComplete: 4 (4)
- ScannerThreadsInvoluntaryContextSwitches: 2 (2)
- ScannerThreadsTotalWallClockTime: 90.991ms
- MaterializeTupleTime(*): 10.162ms
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 17.995ms
- ScannerThreadsVoluntaryContextSwitches: 52 (52)
- TotalRawHdfsReadTime(*): 2.321ms
- TotalReadThroughput: 1.73 MB/sec
Instance 744de1b6228736fa:b54bfaa70000000e
(host=slpr-dhc008.lpdomain.com:22000):(Total: 1s441ms, non-child: 1s228ms,
% non-child: 85.19%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:2/857.88 KB 9:1/683.75 KB 7:1/1.12 MB 8:2/1.83 MB 3:1/392.82 KB
MemoryUsage(500.000ms): 5.23 MB, 5.23 MB, 1.64 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 5.93 MB (6212879)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 100.53 MB (105413224)
- RowsProduced: 29.60K (29604)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s203ms
- TotalStorageWaitTime: 189.481ms
- TotalThreadsInvoluntaryContextSwitches: 88 (88)
- TotalThreadsTotalWallClockTime: 1s615ms
- TotalThreadsSysTime: 7.997ms
- TotalThreadsUserTime: 138.976ms
- TotalThreadsVoluntaryContextSwitches: 177 (177)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s330ms
- ExecTreeExecTime: 71.074ms
- OpenTime: 46.838ms
- ExecTreeOpenTime: 104.011us
- PrepareTime: 63.952ms
- ExecTreePrepareTime: 330.896us
DataStreamSender (dst_id=11):(Total: 37.227ms, non-child: 37.227ms,
% non-child: 100.00%)
- BytesSent: 35.43 MB (37153560)
- NetworkThroughput(*): 1.90 GB/sec
- OverallThroughput: 951.78 MB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 29.60K (29604)
- SerializeBatchTime: 15.769ms
- TransmitDataRPCTime: 18.254ms
- UncompressedRowBatchSize: 69.96 MB (73359800)
CodeGen:(Total: 104.826ms, non-child: 104.826ms, % non-child:
100.00%)
- CodegenTime: 2.701ms
- CompileTime: 13.271ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 30.441ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 59.657ms
SUBPLAN_NODE (id=6):(Total: 71.413ms, non-child: 10.965ms, %
non-child: 15.35%)
- PeakMemoryUsage: 612.97 KB (627682)
- RowsReturned: 28.67K (28672)
- RowsReturnedRate: 401.49 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.170ms, non-child:
27.284ms, % non-child: 77.58%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 29.60K (29604)
- ProbeTime: 0.000ns
- RowsReturned: 15.13M (15129178)
- RowsReturnedRate: 430.16 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 7.885ms, non-child: 7.885ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 29.60K (29604)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 126.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 25.277ms, non-child: 25.277ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:2/857.88 KB 9:1/683.75 KB 7:1/1.12 MB 8:2/1.83 MB 3:1/392.82 KB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/SNAPPY:42
BytesRead(500.000ms): 1.73 MB, 1.73 MB, 1.73 MB
- FooterProcessingTime: (Avg: 13.574ms ; Min: 10.108ms ; Max:
28.351ms ; Number of samples: 7)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.73 MB (1812901)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.73 MB (1812901)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.73 MB (1812901)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 49 (49)
- DecompressionTime: 1.905ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 5 (5)
- NumRowGroups: 7 (7)
- NumScannerThreadsStarted: 7 (7)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 5.65 MB (5926189)
- PerReadThreadRawHdfsThroughput: 621.87 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 23.273ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 64.74K (64736)
- RowsReturned: 29.60K (29604)
- RowsReturnedRate: 1.17 M/sec
- ScanRangesComplete: 7 (7)
- ScannerThreadsInvoluntaryContextSwitches: 3 (3)
- ScannerThreadsTotalWallClockTime: 238.025ms
- MaterializeTupleTime(*): 17.662ms
- ScannerThreadsSysTime: 1.998ms
- ScannerThreadsUserTime: 30.992ms
- ScannerThreadsVoluntaryContextSwitches: 113 (113)
- TotalRawHdfsReadTime(*): 2.780ms
- TotalReadThroughput: 1.15 MB/sec
Instance 744de1b6228736fa:b54bfaa70000000b
(host=slpr-dhc018.lpdomain.com:22000):(Total: 1s438ms, non-child: 1s311ms,
% non-child: 91.15%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 3:1/1.38
MB 10:3/1.67 MB 5:2/1.48 MB
MemoryUsage(500.000ms): 4.59 MB, 4.59 MB, 1.53 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 6.33 MB (6640810)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 105.72 MB (110852967)
- RowsProduced: 28.04K (28043)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s288ms
- TotalStorageWaitTime: 93.368ms
- TotalThreadsInvoluntaryContextSwitches: 41 (41)
- TotalThreadsTotalWallClockTime: 1s536ms
- TotalThreadsSysTime: 4.999ms
- TotalThreadsUserTime: 89.983ms
- TotalThreadsVoluntaryContextSwitches: 173 (173)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s394ms
- ExecTreeExecTime: 56.183ms
- OpenTime: 21.596ms
- ExecTreeOpenTime: 55.153us
- PrepareTime: 22.675ms
- ExecTreePrepareTime: 134.513us
DataStreamSender (dst_id=11):(Total: 29.032ms, non-child: 29.032ms,
% non-child: 100.00%)
- BytesSent: 33.86 MB (35507960)
- NetworkThroughput(*): 2.17 GB/sec
- OverallThroughput: 1.14 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 28.04K (28043)
- SerializeBatchTime: 12.745ms
- TransmitDataRPCTime: 15.241ms
- UncompressedRowBatchSize: 66.28 MB (69502140)
CodeGen:(Total: 41.989ms, non-child: 41.989ms, % non-child: 100.00%)
- CodegenTime: 860.034us
- CompileTime: 6.564ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 13.851ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 21.006ms
SUBPLAN_NODE (id=6):(Total: 56.265ms, non-child: 9.315ms, %
non-child: 16.56%)
- PeakMemoryUsage: 688.91 KB (705448)
- RowsReturned: 27.65K (27648)
- RowsReturnedRate: 491.38 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 29.399ms, non-child:
22.501ms, % non-child: 76.54%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 28.04K (28043)
- ProbeTime: 0.000ns
- RowsReturned: 14.25M (14247810)
- RowsReturnedRate: 484.62 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 6.898ms, non-child: 6.898ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 28.04K (28043)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 144.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 17.549ms, non-child: 17.549ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
3:1/1.38 MB 10:3/1.67 MB 5:2/1.48 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/SNAPPY:36
BytesRead(500.000ms): 1.58 MB, 1.58 MB, 1.58 MB
- FooterProcessingTime: (Avg: 5.002ms ; Min: 2.741ms ; Max:
6.341ms ; Number of samples: 6)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.58 MB (1661533)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.58 MB (1661533)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.58 MB (1661533)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 42 (42)
- DecompressionTime: 1.151ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 3 (3)
- NumRowGroups: 6 (6)
- NumScannerThreadsStarted: 6 (6)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 6.06 MB (6354136)
- PerReadThreadRawHdfsThroughput: 778.82 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 16.292ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 60.93K (60934)
- RowsReturned: 28.04K (28043)
- RowsReturnedRate: 1.60 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 8 (8)
- ScannerThreadsTotalWallClockTime: 120.346ms
- MaterializeTupleTime(*): 9.580ms
- ScannerThreadsSysTime: 999.000us
- ScannerThreadsUserTime: 15.994ms
- ScannerThreadsVoluntaryContextSwitches: 109 (109)
- TotalRawHdfsReadTime(*): 2.034ms
- TotalReadThroughput: 1.06 MB/sec
Instance 744de1b6228736fa:b54bfaa700000009
(host=slpr-dhc017.lpdomain.com:22000):(Total: 1s435ms, non-child: 1s320ms,
% non-child: 91.95%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
9:1/340.56 KB 1:1/1.21 MB 5:2/1013.34 KB 10:2/1.48 MB 8:3/1.48 MB
MemoryUsage(500.000ms): 224.38 KB, 4.98 MB, 4.61 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 6.35 MB (6661176)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 106.11 MB (111260649)
- RowsProduced: 32.69K (32687)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s296ms
- TotalStorageWaitTime: 130.081ms
- TotalThreadsInvoluntaryContextSwitches: 6 (6)
- TotalThreadsTotalWallClockTime: 1s582ms
- TotalThreadsSysTime: 7.997ms
- TotalThreadsUserTime: 89.982ms
- TotalThreadsVoluntaryContextSwitches: 259 (259)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s393ms
- ExecTreeExecTime: 48.186ms
- OpenTime: 21.458ms
- ExecTreeOpenTime: 51.008us
- PrepareTime: 20.257ms
- ExecTreePrepareTime: 181.262us
DataStreamSender (dst_id=11):(Total: 28.455ms, non-child: 28.455ms,
% non-child: 100.00%)
- BytesSent: 39.51 MB (41432100)
- NetworkThroughput(*): 2.76 GB/sec
- OverallThroughput: 1.36 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 32.69K (32687)
- SerializeBatchTime: 12.845ms
- TransmitDataRPCTime: 13.988ms
- UncompressedRowBatchSize: 77.25 MB (81006300)
CodeGen:(Total: 38.787ms, non-child: 38.787ms, % non-child: 100.00%)
- CodegenTime: 850.797us
- CompileTime: 6.391ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 14.162ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 17.738ms
SUBPLAN_NODE (id=6):(Total: 48.306ms, non-child: 8.708ms, %
non-child: 18.03%)
- PeakMemoryUsage: 684.77 KB (701203)
- RowsReturned: 31.74K (31744)
- RowsReturnedRate: 657.14 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 26.113ms, non-child:
20.562ms, % non-child: 78.74%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 32.69K (32687)
- ProbeTime: 0.000ns
- RowsReturned: 16.71M (16713896)
- RowsReturnedRate: 640.04 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 5.551ms, non-child: 5.551ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 32.69K (32687)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 180.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 13.483ms, non-child: 13.483ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
9:1/340.56 KB 1:1/1.21 MB 5:2/1013.34 KB 10:2/1.48 MB 8:3/1.48 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 9 out of 9
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/SNAPPY:54
BytesRead(500.000ms): 0, 2.06 MB, 2.06 MB
- FooterProcessingTime: (Avg: 6.123ms ; Min: 3.754ms ; Max:
8.084ms ; Number of samples: 9)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 2.06 MB (2157072)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 2.06 MB (2157072)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 2.06 MB (2157072)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 63 (63)
- DecompressionTime: 1.269ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 5 (5)
- NumRowGroups: 9 (9)
- NumScannerThreadsStarted: 9 (9)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 5.88 MB (6167436)
- PerReadThreadRawHdfsThroughput: 921.10 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 12.190ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 72.05K (72054)
- RowsReturned: 32.69K (32687)
- RowsReturnedRate: 2.42 M/sec
- ScanRangesComplete: 9 (9)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 167.518ms
- MaterializeTupleTime(*): 9.184ms
- ScannerThreadsSysTime: 3.997ms
- ScannerThreadsUserTime: 17.993ms
- ScannerThreadsVoluntaryContextSwitches: 177 (177)
- TotalRawHdfsReadTime(*): 2.233ms
- TotalReadThroughput: 1.37 MB/sec
Instance 744de1b6228736fa:b54bfaa70000000a
(host=slpr-dhc010.lpdomain.com:22000):(Total: 1s412ms, non-child: 1s287ms,
% non-child: 91.14%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 8:1/1.38
MB 0:2/1.45 MB 4:1/1.16 MB 7:1/226.78 KB 9:1/1.28 MB
MemoryUsage(500.000ms): 4.21 MB, 5.06 MB, 3.83 MB
ThreadUsage(500.000ms): 5, 1, 1
- AverageThreadTokens: 2.33
- BloomFilterBytes: 0
- PeakMemoryUsage: 6.21 MB (6507726)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 78.98 MB (82816582)
- RowsProduced: 34.84K (34842)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s211ms
- TotalStorageWaitTime: 112.256ms
- TotalThreadsInvoluntaryContextSwitches: 9 (9)
- TotalThreadsTotalWallClockTime: 1s532ms
- TotalThreadsSysTime: 1.999ms
- TotalThreadsUserTime: 92.982ms
- TotalThreadsVoluntaryContextSwitches: 185 (185)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s375ms
- ExecTreeExecTime: 59.267ms
- OpenTime: 17.728ms
- ExecTreeOpenTime: 46.848us
- PrepareTime: 18.874ms
- ExecTreePrepareTime: 137.282us
DataStreamSender (dst_id=11):(Total: 32.771ms, non-child: 32.771ms,
% non-child: 100.00%)
- BytesSent: 42.05 MB (44091120)
- NetworkThroughput(*): 2.54 GB/sec
- OverallThroughput: 1.25 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 34.84K (34842)
- SerializeBatchTime: 13.977ms
- TransmitDataRPCTime: 16.163ms
- UncompressedRowBatchSize: 82.35 MB (86350200)
CodeGen:(Total: 32.930ms, non-child: 32.930ms, % non-child: 100.00%)
- CodegenTime: 686.488us
- CompileTime: 5.514ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 11.619ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 15.395ms
SUBPLAN_NODE (id=6):(Total: 59.389ms, non-child: 10.261ms, %
non-child: 17.28%)
- PeakMemoryUsage: 683.40 KB (699799)
- RowsReturned: 34.82K (34816)
- RowsReturnedRate: 586.23 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 30.704ms, non-child:
24.298ms, % non-child: 79.14%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 34.84K (34842)
- ProbeTime: 0.000ns
- RowsReturned: 17.84M (17843551)
- RowsReturnedRate: 581.14 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 6.406ms, non-child: 6.406ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 34.84K (34842)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 156.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 18.423ms, non-child: 18.423ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
8:1/1.38 MB 0:2/1.45 MB 4:1/1.16 MB 7:1/226.78 KB 9:1/1.28 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
Hdfs Read Thread Concurrency Bucket: 0:66.67% 1:0% 2:33.33% 3:0%
4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
17:0%
File Formats: PARQUET/SNAPPY:36
BytesRead(500.000ms): 1.61 MB, 1.82 MB, 1.82 MB
- FooterProcessingTime: (Avg: 6.872ms ; Min: 5.472ms ; Max:
7.952ms ; Number of samples: 6)
- AverageHdfsReadThreadConcurrency: 0.67
- AverageScannerThreadConcurrency: 1.33
- BytesRead: 1.82 MB (1908230)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.82 MB (1908230)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.82 MB (1908230)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 42 (42)
- DecompressionTime: 1.046ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 5 (5)
- NumRowGroups: 6 (6)
- NumScannerThreadsStarted: 6 (6)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 5.93 MB (6221052)
- PerReadThreadRawHdfsThroughput: 68.29 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 17.139ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 74.63K (74626)
- RowsReturned: 34.84K (34842)
- RowsReturnedRate: 1.89 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 139.205ms
- MaterializeTupleTime(*): 7.716ms
- ScannerThreadsSysTime: 999.000us
- ScannerThreadsUserTime: 14.994ms
- ScannerThreadsVoluntaryContextSwitches: 98 (98)
- TotalRawHdfsReadTime(*): 26.646ms
- TotalReadThroughput: 1.21 MB/sec
Instance 744de1b6228736fa:b54bfaa700000003
(host=slpr-dhc005.lpdomain.com:22000):(Total: 1s408ms, non-child: 1s277ms,
% non-child: 90.69%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
10:1/519.80 KB 5:2/574.88 KB 2:1/1.39 MB 1:2/1.03 MB 9:1/560.57 KB
6:1/170.23 KB
MemoryUsage(500.000ms): 4.53 MB, 4.44 MB, 1.01 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 5.28 MB (5534493)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 77.37 MB (81124776)
- RowsProduced: 24.11K (24114)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s271ms
- TotalStorageWaitTime: 142.392ms
- TotalThreadsInvoluntaryContextSwitches: 42 (42)
- TotalThreadsTotalWallClockTime: 1s555ms
- TotalThreadsSysTime: 6.998ms
- TotalThreadsUserTime: 99.979ms
- TotalThreadsVoluntaryContextSwitches: 194 (194)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s343ms
- ExecTreeExecTime: 43.864ms
- OpenTime: 38.771ms
- ExecTreeOpenTime: 90.450us
- PrepareTime: 25.964ms
- ExecTreePrepareTime: 225.481us
DataStreamSender (dst_id=11):(Total: 25.743ms, non-child: 25.743ms,
% non-child: 100.00%)
- BytesSent: 28.99 MB (30395020)
- NetworkThroughput(*): 2.37 GB/sec
- OverallThroughput: 1.10 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 24.11K (24114)
- SerializeBatchTime: 12.033ms
- TransmitDataRPCTime: 11.924ms
- UncompressedRowBatchSize: 56.99 MB (59755180)
CodeGen:(Total: 61.321ms, non-child: 61.321ms, % non-child: 100.00%)
- CodegenTime: 1.102ms
- CompileTime: 12.945ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 24.479ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 23.334ms
SUBPLAN_NODE (id=6):(Total: 44.097ms, non-child: 7.911ms, %
non-child: 17.94%)
- PeakMemoryUsage: 730.97 KB (748516)
- RowsReturned: 23.55K (23552)
- RowsReturnedRate: 534.09 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 22.563ms, non-child:
16.978ms, % non-child: 75.25%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 24.11K (24114)
- ProbeTime: 0.000ns
- RowsReturned: 12.23M (12228603)
- RowsReturnedRate: 541.96 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 5.585ms, non-child: 5.585ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 24.11K (24114)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 179.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 13.622ms, non-child: 13.622ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
10:1/519.80 KB 5:2/574.88 KB 2:1/1.39 MB 1:2/1.03 MB 9:1/560.57 KB
6:1/170.23 KB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 8 out of 8
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/SNAPPY:48
BytesRead(500.000ms): 1.65 MB, 1.65 MB, 1.65 MB
- FooterProcessingTime: (Avg: 12.545ms ; Min: 6.441ms ; Max:
22.143ms ; Number of samples: 8)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.65 MB (1734402)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.65 MB (1734402)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.65 MB (1734402)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 56 (56)
- DecompressionTime: 1.145ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 6 (6)
- NumRowGroups: 8 (8)
- NumScannerThreadsStarted: 8 (8)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 5.00 MB (5247819)
- PerReadThreadRawHdfsThroughput: 736.05 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 11.739ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 54.66K (54658)
- RowsReturned: 24.11K (24114)
- RowsReturnedRate: 1.77 M/sec
- ScanRangesComplete: 8 (8)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 173.037ms
- MaterializeTupleTime(*): 9.293ms
- ScannerThreadsSysTime: 999.000us
- ScannerThreadsUserTime: 19.991ms
- ScannerThreadsVoluntaryContextSwitches: 136 (136)
- TotalRawHdfsReadTime(*): 2.247ms
- TotalReadThroughput: 1.10 MB/sec
Instance 744de1b6228736fa:b54bfaa700000010
(host=slpr-dhc011.lpdomain.com:22000):(Total: 1s408ms, non-child: 1s272ms,
% non-child: 90.31%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 8:1/1.34
MB 5:1/177.75 KB 9:2/1.51 MB 11:2/1.48 MB 3:1/173.83 KB
MemoryUsage(500.000ms): 4.41 MB, 4.41 MB, 988.24 KB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 5.24 MB (5494080)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 77.00 MB (80742460)
- RowsProduced: 28.24K (28236)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s271ms
- TotalStorageWaitTime: 104.916ms
- TotalThreadsInvoluntaryContextSwitches: 62 (62)
- TotalThreadsTotalWallClockTime: 1s514ms
- TotalThreadsSysTime: 8.996ms
- TotalThreadsUserTime: 88.982ms
- TotalThreadsVoluntaryContextSwitches: 172 (172)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s356ms
- ExecTreeExecTime: 50.194ms
- OpenTime: 26.941ms
- ExecTreeOpenTime: 59.915us
- PrepareTime: 24.743ms
- ExecTreePrepareTime: 169.120us
DataStreamSender (dst_id=11):(Total: 39.302ms, non-child: 39.302ms,
% non-child: 100.00%)
- BytesSent: 34.08 MB (35740540)
- NetworkThroughput(*): 1.48 GB/sec
- OverallThroughput: 867.24 MB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 28.24K (28236)
- SerializeBatchTime: 13.265ms
- TransmitDataRPCTime: 22.511ms
- UncompressedRowBatchSize: 66.74 MB (69979140)
CodeGen:(Total: 46.908ms, non-child: 46.908ms, % non-child: 100.00%)
- CodegenTime: 999.391us
- CompileTime: 8.197ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 17.861ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 20.338ms
SUBPLAN_NODE (id=6):(Total: 50.346ms, non-child: 8.249ms, %
non-child: 16.39%)
- PeakMemoryUsage: 706.48 KB (723433)
- RowsReturned: 27.65K (27648)
- RowsReturnedRate: 549.15 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 27.022ms, non-child:
20.861ms, % non-child: 77.20%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 28.24K (28236)
- ProbeTime: 0.000ns
- RowsReturned: 14.34M (14342766)
- RowsReturnedRate: 530.77 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 6.160ms, non-child: 6.160ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 28.24K (28236)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 162.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 15.074ms, non-child: 15.074ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
8:1/1.34 MB 5:1/177.75 KB 9:2/1.51 MB 11:2/1.48 MB 3:1/173.83 KB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/SNAPPY:42
BytesRead(500.000ms): 1.69 MB, 1.69 MB, 1.69 MB
- FooterProcessingTime: (Avg: 6.723ms ; Min: 5.085ms ; Max:
8.167ms ; Number of samples: 7)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.69 MB (1777017)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.69 MB (1777017)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.69 MB (1777017)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 49 (49)
- DecompressionTime: 1.245ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 5 (5)
- NumRowGroups: 7 (7)
- NumScannerThreadsStarted: 7 (7)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 5.05 MB (5294444)
- PerReadThreadRawHdfsThroughput: 617.71 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 13.422ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 61.76K (61756)
- RowsReturned: 28.24K (28236)
- RowsReturnedRate: 1.87 M/sec
- ScanRangesComplete: 7 (7)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 131.024ms
- MaterializeTupleTime(*): 8.897ms
- ScannerThreadsSysTime: 3.997ms
- ScannerThreadsUserTime: 12.994ms
- ScannerThreadsVoluntaryContextSwitches: 108 (108)
- TotalRawHdfsReadTime(*): 2.743ms
- TotalReadThroughput: 1.13 MB/sec
Instance 744de1b6228736fa:b54bfaa700000012
(host=slpr-dhc015.lpdomain.com:22000):(Total: 1s407ms, non-child: 1s287ms,
% non-child: 91.44%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:1/1.01 MB 9:1/187.94 KB 10:1/244.30 KB 1:1/1.26 MB 2:1/222.25 KB
6:1/1.29 MB 8:2/1.26 MB
MemoryUsage(500.000ms): 4.95 MB, 4.95 MB, 960.14 KB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 5.85 MB (6131090)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 76.05 MB (79742718)
- RowsProduced: 33.12K (33121)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s287ms
- TotalStorageWaitTime: 163.998ms
- TotalThreadsInvoluntaryContextSwitches: 10 (10)
- TotalThreadsTotalWallClockTime: 1s581ms
- TotalThreadsSysTime: 5.999ms
- TotalThreadsUserTime: 91.981ms
- TotalThreadsVoluntaryContextSwitches: 202 (202)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s369ms
- ExecTreeExecTime: 55.689ms
- OpenTime: 18.168ms
- ExecTreeOpenTime: 46.705us
- PrepareTime: 19.634ms
- ExecTreePrepareTime: 170.693us
DataStreamSender (dst_id=11):(Total: 30.910ms, non-child: 30.910ms,
% non-child: 100.00%)
- BytesSent: 39.88 MB (41812920)
- NetworkThroughput(*): 2.75 GB/sec
- OverallThroughput: 1.26 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 33.12K (33121)
- SerializeBatchTime: 13.970ms
- TransmitDataRPCTime: 14.144ms
- UncompressedRowBatchSize: 78.28 MB (82081040)
CodeGen:(Total: 33.819ms, non-child: 33.819ms, % non-child: 100.00%)
- CodegenTime: 794.441us
- CompileTime: 5.469ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 11.979ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 15.849ms
SUBPLAN_NODE (id=6):(Total: 55.809ms, non-child: 10.461ms, %
non-child: 18.75%)
- PeakMemoryUsage: 675.86 KB (692077)
- RowsReturned: 32.77K (32768)
- RowsReturnedRate: 587.14 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 30.755ms, non-child:
24.689ms, % non-child: 80.28%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 33.12K (33121)
- ProbeTime: 0.000ns
- RowsReturned: 16.86M (16856081)
- RowsReturnedRate: 548.06 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 6.065ms, non-child: 6.065ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 33.12K (33121)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 164.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 14.592ms, non-child: 14.592ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:1/1.01 MB 9:1/187.94 KB 10:1/244.30 KB 1:1/1.26 MB 2:1/222.25 KB
6:1/1.29 MB 8:2/1.26 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 8 out of 8
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/SNAPPY:48
BytesRead(500.000ms): 1.96 MB, 1.96 MB, 1.96 MB
- FooterProcessingTime: (Avg: 15.586ms ; Min: 7.605ms ; Max:
29.109ms ; Number of samples: 8)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.96 MB (2056608)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.96 MB (2056608)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.96 MB (2056608)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 56 (56)
- DecompressionTime: 1.216ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 7 (7)
- NumRowGroups: 8 (8)
- NumScannerThreadsStarted: 8 (8)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 5.57 MB (5844400)
- PerReadThreadRawHdfsThroughput: 1017.18 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 12.906ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 72.55K (72552)
- RowsReturned: 33.12K (33121)
- RowsReturnedRate: 2.27 M/sec
- ScanRangesComplete: 8 (8)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 193.175ms
- MaterializeTupleTime(*): 9.160ms
- ScannerThreadsSysTime: 999.000us
- ScannerThreadsUserTime: 16.992ms
- ScannerThreadsVoluntaryContextSwitches: 128 (128)
- TotalRawHdfsReadTime(*): 1.928ms
- TotalReadThroughput: 1.31 MB/sec
Instance 744de1b6228736fa:b54bfaa700000007
(host=slpr-dhc014.lpdomain.com:22000):(Total: 1s405ms, non-child: 1s285ms,
% non-child: 91.46%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 5:1/1.21
MB 1:1/214.31 KB 2:1/1.35 MB 10:2/1.97 MB 11:1/172.29 KB 9:1/179.34 KB
MemoryUsage(500.000ms): 4.40 MB, 4.40 MB, 1.41 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 6.11 MB (6411936)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 76.12 MB (79813012)
- RowsProduced: 31.24K (31237)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s285ms
- TotalStorageWaitTime: 105.765ms
- TotalThreadsInvoluntaryContextSwitches: 4 (4)
- TotalThreadsTotalWallClockTime: 1s515ms
- TotalThreadsSysTime: 4.998ms
- TotalThreadsUserTime: 87.983ms
- TotalThreadsVoluntaryContextSwitches: 167 (167)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s364ms
- ExecTreeExecTime: 53.780ms
- OpenTime: 20.139ms
- ExecTreeOpenTime: 50.643us
- PrepareTime: 20.954ms
- ExecTreePrepareTime: 151.743us
DataStreamSender (dst_id=11):(Total: 29.295ms, non-child: 29.295ms,
% non-child: 100.00%)
- BytesSent: 37.51 MB (39335440)
- NetworkThroughput(*): 2.73 GB/sec
- OverallThroughput: 1.25 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 31.24K (31237)
- SerializeBatchTime: 12.675ms
- TransmitDataRPCTime: 13.419ms
- UncompressedRowBatchSize: 73.83 MB (77411840)
CodeGen:(Total: 36.845ms, non-child: 36.845ms, % non-child: 100.00%)
- CodegenTime: 695.456us
- CompileTime: 6.334ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 13.147ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 16.920ms
SUBPLAN_NODE (id=6):(Total: 53.928ms, non-child: 9.101ms, %
non-child: 16.88%)
- PeakMemoryUsage: 673.53 KB (689692)
- RowsReturned: 30.72K (30720)
- RowsReturnedRate: 569.64 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 27.770ms, non-child:
21.243ms, % non-child: 76.50%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 31.24K (31237)
- ProbeTime: 0.000ns
- RowsReturned: 15.88M (15877903)
- RowsReturnedRate: 571.75 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 6.527ms, non-child: 6.527ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 31.24K (31237)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 153.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 17.055ms, non-child: 17.055ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
5:1/1.21 MB 1:1/214.31 KB 2:1/1.35 MB 10:2/1.97 MB 11:1/172.29 KB
9:1/179.34 KB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/SNAPPY:42
BytesRead(500.000ms): 1.79 MB, 1.79 MB, 1.79 MB
- FooterProcessingTime: (Avg: 8.079ms ; Min: 7.484ms ; Max:
9.282ms ; Number of samples: 7)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.79 MB (1881688)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.79 MB (1881688)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.79 MB (1881688)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 49 (49)
- DecompressionTime: 1.425ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 6 (6)
- NumRowGroups: 7 (7)
- NumScannerThreadsStarted: 7 (7)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 5.84 MB (6125262)
- PerReadThreadRawHdfsThroughput: 787.42 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 15.401ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 68.03K (68026)
- RowsReturned: 31.24K (31237)
- RowsReturnedRate: 1.83 M/sec
- ScanRangesComplete: 7 (7)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 130.754ms
- MaterializeTupleTime(*): 7.935ms
- ScannerThreadsSysTime: 1.998ms
- ScannerThreadsUserTime: 14.994ms
- ScannerThreadsVoluntaryContextSwitches: 101 (101)
- TotalRawHdfsReadTime(*): 2.278ms
- TotalReadThroughput: 1.20 MB/sec
Instance 744de1b6228736fa:b54bfaa700000011
(host=slpr-dhc016.lpdomain.com:22000):(Total: 1s402ms, non-child: 1s303ms,
% non-child: 92.94%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
6:1/486.13 KB 2:2/1.23 MB 4:1/838.47 KB 3:1/260.54 KB 9:1/192.11 KB
1:2/1.73 MB
MemoryUsage(500.000ms): 4.08 MB, 4.08 MB, 2.62 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 5.50 MB (5763668)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 75.78 MB (79461951)
- RowsProduced: 27.73K (27728)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s301ms
- TotalStorageWaitTime: 121.015ms
- TotalThreadsInvoluntaryContextSwitches: 7 (7)
- TotalThreadsTotalWallClockTime: 1s531ms
- TotalThreadsSysTime: 6.998ms
- TotalThreadsUserTime: 65.984ms
- TotalThreadsVoluntaryContextSwitches: 204 (204)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s363ms
- ExecTreeExecTime: 39.105ms
- OpenTime: 19.516ms
- ExecTreeOpenTime: 47.279us
- PrepareTime: 19.365ms
- ExecTreePrepareTime: 133.101us
DataStreamSender (dst_id=11):(Total: 23.762ms, non-child: 23.762ms,
% non-child: 100.00%)
- BytesSent: 33.39 MB (35012100)
- NetworkThroughput(*): 2.65 GB/sec
- OverallThroughput: 1.37 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 27.73K (27728)
- SerializeBatchTime: 9.546ms
- TransmitDataRPCTime: 12.286ms
- UncompressedRowBatchSize: 65.53 MB (68713240)
CodeGen:(Total: 35.956ms, non-child: 35.956ms, % non-child: 100.00%)
- CodegenTime: 721.314us
- CompileTime: 6.112ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 12.794ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 16.599ms
SUBPLAN_NODE (id=6):(Total: 39.213ms, non-child: 5.759ms, %
non-child: 14.69%)
- PeakMemoryUsage: 671.21 KB (687319)
- RowsReturned: 27.65K (27648)
- RowsReturnedRate: 705.06 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 18.599ms, non-child:
14.247ms, % non-child: 76.60%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 27.73K (27728)
- ProbeTime: 0.000ns
- RowsReturned: 14.17M (14172840)
- RowsReturnedRate: 761.98 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 4.352ms, non-child: 4.352ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 27.73K (27728)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 229.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 14.853ms, non-child: 14.853ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
6:1/486.13 KB 2:2/1.23 MB 4:1/838.47 KB 3:1/260.54 KB 9:1/192.11 KB
1:2/1.73 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 8 out of 8
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/SNAPPY:48
BytesRead(500.000ms): 1.78 MB, 1.78 MB, 1.78 MB
- FooterProcessingTime: (Avg: 7.908ms ; Min: 7.090ms ; Max:
8.892ms ; Number of samples: 8)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.78 MB (1867919)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.78 MB (1867919)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.78 MB (1867919)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 56 (56)
- DecompressionTime: 1.045ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 6 (6)
- NumRowGroups: 8 (8)
- NumScannerThreadsStarted: 8 (8)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 5.14 MB (5388373)
- PerReadThreadRawHdfsThroughput: 923.56 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 13.628ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 61.73K (61732)
- RowsReturned: 27.73K (27728)
- RowsReturnedRate: 1.87 M/sec
- ScanRangesComplete: 8 (8)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 149.221ms
- MaterializeTupleTime(*): 7.339ms
- ScannerThreadsSysTime: 999.000us
- ScannerThreadsUserTime: 12.992ms
- ScannerThreadsVoluntaryContextSwitches: 138 (138)
- TotalRawHdfsReadTime(*): 1.928ms
- TotalReadThroughput: 1.19 MB/sec
Instance 744de1b6228736fa:b54bfaa700000014
(host=slpr-dhc020.lpdomain.com:22000):(Total: 1s399ms, non-child: 1s282ms,
% non-child: 91.64%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:1/182.35 KB 3:1/1.25 MB 0:1/181.94 KB 7:1/374.64 KB 2:1/1.32 MB 8:2/1.62
MB
MemoryUsage(500.000ms): 3.78 MB, 4.10 MB, 2.76 MB
ThreadUsage(500.000ms): 2, 1, 1
- AverageThreadTokens: 1.33
- BloomFilterBytes: 0
- PeakMemoryUsage: 4.83 MB (5062033)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 81.51 MB (85472224)
- RowsProduced: 29.83K (29834)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s274ms
- TotalStorageWaitTime: 106.847ms
- TotalThreadsInvoluntaryContextSwitches: 32 (32)
- TotalThreadsTotalWallClockTime: 1s509ms
- TotalThreadsSysTime: 4.999ms
- TotalThreadsUserTime: 86.982ms
- TotalThreadsVoluntaryContextSwitches: 178 (178)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s359ms
- ExecTreeExecTime: 49.176ms
- OpenTime: 20.060ms
- ExecTreeOpenTime: 46.820us
- PrepareTime: 19.886ms
- ExecTreePrepareTime: 150.664us
DataStreamSender (dst_id=11):(Total: 31.378ms, non-child: 31.378ms,
% non-child: 100.00%)
- BytesSent: 35.99 MB (37739200)
- NetworkThroughput(*): 2.25 GB/sec
- OverallThroughput: 1.12 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 29.83K (29834)
- SerializeBatchTime: 13.343ms
- TransmitDataRPCTime: 15.610ms
- UncompressedRowBatchSize: 70.51 MB (73936460)
CodeGen:(Total: 36.307ms, non-child: 36.307ms, % non-child: 100.00%)
- CodegenTime: 818.733us
- CompileTime: 5.940ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 13.387ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 16.467ms
SUBPLAN_NODE (id=6):(Total: 49.274ms, non-child: 9.236ms, %
non-child: 18.74%)
- PeakMemoryUsage: 735.13 KB (752773)
- RowsReturned: 29.70K (29696)
- RowsReturnedRate: 602.66 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 29.023ms, non-child:
22.027ms, % non-child: 75.89%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 29.83K (29834)
- ProbeTime: 0.000ns
- RowsReturned: 15.23M (15228791)
- RowsReturnedRate: 524.71 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 6.996ms, non-child: 6.996ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 29.83K (29834)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 142.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 11.015ms, non-child: 11.015ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:1/182.35 KB 3:1/1.25 MB 0:1/181.94 KB 7:1/374.64 KB 2:1/1.32 MB 8:2/1.62
MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7
Hdfs Read Thread Concurrency Bucket: 0:66.67% 1:33.33% 2:0% 3:0%
4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
17:0%
File Formats: PARQUET/SNAPPY:42
BytesRead(500.000ms): 1.70 MB, 1.75 MB, 1.75 MB
- FooterProcessingTime: (Avg: 10.602ms ; Min: 5.250ms ; Max:
38.505ms ; Number of samples: 7)
- AverageHdfsReadThreadConcurrency: 0.33
- AverageScannerThreadConcurrency: 0.33
- BytesRead: 1.75 MB (1837043)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.75 MB (1837043)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.75 MB (1837043)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 49 (49)
- DecompressionTime: 979.025us
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 6 (6)
- NumRowGroups: 7 (7)
- NumScannerThreadsStarted: 7 (7)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 4.55 MB (4775359)
- PerReadThreadRawHdfsThroughput: 456.04 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 9.540ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 65.25K (65252)
- RowsReturned: 29.83K (29834)
- RowsReturnedRate: 2.71 M/sec
- ScanRangesComplete: 7 (7)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 129.926ms
- MaterializeTupleTime(*): 6.955ms
- ScannerThreadsSysTime: 999.000us
- ScannerThreadsUserTime: 12.993ms
- ScannerThreadsVoluntaryContextSwitches: 106 (106)
- TotalRawHdfsReadTime(*): 3.841ms
- TotalReadThroughput: 1.17 MB/sec
Instance 744de1b6228736fa:b54bfaa70000000c
(host=slpr-dhc019.lpdomain.com:22000):(Total: 1s392ms, non-child: 1s281ms,
% non-child: 92.04%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 9:1/1.02
MB 7:1/1.10 MB 0:1/1.22 MB 8:1/1.26 MB 1:2/395.98 KB
MemoryUsage(500.000ms): 4.94 MB, 4.94 MB, 1.19 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 7.00 MB (7340476)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 75.63 MB (79303233)
- RowsProduced: 31.25K (31245)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s191ms
- TotalStorageWaitTime: 106.956ms
- TotalThreadsInvoluntaryContextSwitches: 73 (73)
- TotalThreadsTotalWallClockTime: 1s503ms
- TotalThreadsSysTime: 2.999ms
- TotalThreadsUserTime: 81.984ms
- TotalThreadsVoluntaryContextSwitches: 183 (183)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s356ms
- ExecTreeExecTime: 48.776ms
- OpenTime: 17.984ms
- ExecTreeOpenTime: 44.576us
- PrepareTime: 17.956ms
- ExecTreePrepareTime: 161.729us
DataStreamSender (dst_id=11):(Total: 29.063ms, non-child: 29.063ms,
% non-child: 100.00%)
- BytesSent: 37.68 MB (39513400)
- NetworkThroughput(*): 2.30 GB/sec
- OverallThroughput: 1.27 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 31.25K (31245)
- SerializeBatchTime: 11.112ms
- TransmitDataRPCTime: 16.017ms
- UncompressedRowBatchSize: 73.85 MB (77434140)
CodeGen:(Total: 32.932ms, non-child: 32.932ms, % non-child: 100.00%)
- CodegenTime: 720.261us
- CompileTime: 5.503ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 11.844ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 15.153ms
SUBPLAN_NODE (id=6):(Total: 48.848ms, non-child: 7.468ms, %
non-child: 15.29%)
- PeakMemoryUsage: 666.42 KB (682417)
- RowsReturned: 30.72K (30720)
- RowsReturnedRate: 628.88 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 25.718ms, non-child:
19.277ms, % non-child: 74.96%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 31.25K (31245)
- ProbeTime: 0.000ns
- RowsReturned: 15.88M (15882075)
- RowsReturnedRate: 617.53 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 6.440ms, non-child: 6.440ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 31.25K (31245)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 155.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 15.661ms, non-child: 15.661ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
9:1/1.02 MB 7:1/1.10 MB 0:1/1.22 MB 8:1/1.26 MB 1:2/395.98 KB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/SNAPPY:36
BytesRead(500.000ms): 1.69 MB, 1.69 MB, 1.69 MB
- FooterProcessingTime: (Avg: 11.127ms ; Min: 4.612ms ; Max:
23.576ms ; Number of samples: 6)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.69 MB (1772933)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.69 MB (1772933)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.69 MB (1772933)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 42 (42)
- DecompressionTime: 1.000ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 5 (5)
- NumRowGroups: 6 (6)
- NumScannerThreadsStarted: 6 (6)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 6.73 MB (7053786)
- PerReadThreadRawHdfsThroughput: 1.12 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 14.492ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 67.31K (67314)
- RowsReturned: 31.25K (31245)
- RowsReturnedRate: 1.99 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 129.305ms
- MaterializeTupleTime(*): 9.107ms
- ScannerThreadsSysTime: 999.000us
- ScannerThreadsUserTime: 16.994ms
- ScannerThreadsVoluntaryContextSwitches: 111 (111)
- TotalRawHdfsReadTime(*): 1.475ms
- TotalReadThroughput: 1.13 MB/sec
Instance 744de1b6228736fa:b54bfaa700000001
(host=slpr-dhc013.lpdomain.com:22000):(Total: 1s292ms, non-child: 1s188ms,
% non-child: 91.97%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:1/172.94 KB 1:1/680.67 KB 7:1/1.19 MB 6:1/220.94 KB 4:2/2.54 MB
MemoryUsage(500.000ms): 99.38 KB, 4.08 MB, 2.86 MB
ThreadUsage(500.000ms): 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 5.22 MB (5477186)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 76.04 MB (79737600)
- RowsProduced: 29.82K (29816)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1s194ms
- TotalStorageWaitTime: 84.889ms
- TotalThreadsInvoluntaryContextSwitches: 9 (9)
- TotalThreadsTotalWallClockTime: 1s379ms
- TotalThreadsSysTime: 5.998ms
- TotalThreadsUserTime: 75.986ms
- TotalThreadsVoluntaryContextSwitches: 171 (171)
Fragment Instance Lifecycle Timings:
- ExecTime: 1s250ms
- ExecTreeExecTime: 39.392ms
- OpenTime: 20.762ms
- ExecTreeOpenTime: 58.843us
- PrepareTime: 20.558ms
- ExecTreePrepareTime: 182.009us
DataStreamSender (dst_id=11):(Total: 26.737ms, non-child: 26.737ms,
% non-child: 100.00%)
- BytesSent: 35.89 MB (37634460)
- NetworkThroughput(*): 2.64 GB/sec
- OverallThroughput: 1.31 GB/sec
- PeakMemoryUsage: 59.38 KB (60800)
- RowsReturned: 29.82K (29816)
- SerializeBatchTime: 10.888ms
- TransmitDataRPCTime: 13.267ms
- UncompressedRowBatchSize: 70.47 MB (73892760)
CodeGen:(Total: 37.505ms, non-child: 37.505ms, % non-child: 100.00%)
- CodegenTime: 855.587us
- CompileTime: 6.296ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.96 MB (2050180)
- NumFunctions: 16 (16)
- NumInstructions: 250 (250)
- OptimizationTime: 13.802ms
- PeakMemoryUsage: 125.00 KB (128000)
- PrepareTime: 16.861ms
SUBPLAN_NODE (id=6):(Total: 39.529ms, non-child: 6.476ms, %
non-child: 16.38%)
- PeakMemoryUsage: 676.42 KB (692656)
- RowsReturned: 29.70K (29696)
- RowsReturnedRate: 751.23 K/sec
NESTED_LOOP_JOIN_NODE (id=9):(Total: 21.183ms, non-child:
16.175ms, % non-child: 76.36%)
- BuildRows: 0 (0)
- BuildTime: 0.000ns
- PeakMemoryUsage: 24.00 KB (24576)
- ProbeRows: 29.82K (29816)
- ProbeTime: 0.000ns
- RowsReturned: 15.23M (15226460)
- RowsReturnedRate: 718.80 M/sec
Nested Loop Join Builder:
- PeakMemoryUsage: 8.00 KB (8192)
SINGULAR_ROW_SRC_NODE (id=7):
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
UNNEST_NODE (id=8):(Total: 5.007ms, non-child: 5.007ms, %
non-child: 100.00%)
- AvgCollectionSize: 1.00
- MaxCollectionSize: 1 (1)
- MinCollectionSize: 1 (1)
- NumCollections: 29.82K (29816)
- PeakMemoryUsage: 0
- RowsReturned: 1 (1)
- RowsReturnedRate: 199.00 /sec
HDFS_SCAN_NODE (id=5):(Total: 11.870ms, non-child: 11.870ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:1/172.94 KB 1:1/680.67 KB 7:1/1.19 MB 6:1/220.94 KB 4:2/2.54 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/SNAPPY:36
BytesRead(500.000ms): 0, 1.64 MB, 1.64 MB
- FooterProcessingTime: (Avg: 6.385ms ; Min: 5.928ms ; Max:
7.410ms ; Number of samples: 6)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 1.64 MB (1721793)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.64 MB (1721793)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.64 MB (1721793)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 42 (42)
- DecompressionTime: 1.224ms
- MaxCompressedTextFileLength: 0
- NumColumns: 6 (6)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 5 (5)
- NumRowGroups: 6 (6)
- NumScannerThreadsStarted: 6 (6)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 4.87 MB (5102432)
- PerReadThreadRawHdfsThroughput: 86.13 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 10.358ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 64.41K (64410)
- RowsReturned: 29.82K (29816)
- RowsReturnedRate: 2.51 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 107.697ms
- MaterializeTupleTime(*): 8.462ms
- ScannerThreadsSysTime: 1.998ms
- ScannerThreadsUserTime: 14.995ms
- ScannerThreadsVoluntaryContextSwitches: 92 (92)
- TotalRawHdfsReadTime(*): 19.064ms
- TotalReadThroughput: 1.09 MB/sec
[slpr-dhc018.lpdomain.com:21000] >
On Thu, Feb 22, 2018 at 9:41 PM, Mostafa Mokhtar <mm...@cloudera.com>
wrote:
> @Fawze,
>
> To best serve your question please share a profile for the problematic
> query.
>
> Thanks
> Mostafa
>
> On Thu, Feb 22, 2018 at 9:40 AM, Jim Apple <jb...@cloudera.com> wrote:
>
>> I think compute stats will help estimates be more accurate.
>>
>> On Thu, Feb 22, 2018 at 4:48 AM, Fawze Abujaber <fa...@gmail.com>
>> wrote:
>>
>>> Hi Guys,
>>>
>>> In Impala version 2.10 i see a lot that the Estimate per Mode peak
>>> memory can reach X30 times of the Per Node Peak Memory usage.
>>>
>>> For example one of the query the estimate was 9GB while the usage was
>>> 30GB.
>>>
>>> I have 3 questions:
>>>
>>> 1- Do you think the compute stats will help here?
>>> 2- How i can reduce this gap?
>>> 3- Does the resource pools based on the Estimate per Mode peak memory?
>>> so if for such query i put the default max memory limit to 5GB, will this
>>> query fail?
>>>
>>>
>>>
>>
>
Re: Estimate peak memory VS used peak memory
Posted by Mostafa Mokhtar <mm...@cloudera.com>.
@Fawze,
To best serve your question please share a profile for the problematic
query.
Thanks
Mostafa
On Thu, Feb 22, 2018 at 9:40 AM, Jim Apple <jb...@cloudera.com> wrote:
> I think compute stats will help estimates be more accurate.
>
> On Thu, Feb 22, 2018 at 4:48 AM, Fawze Abujaber <fa...@gmail.com> wrote:
>
>> Hi Guys,
>>
>> In Impala version 2.10 i see a lot that the Estimate per Mode peak memory
>> can reach X30 times of the Per Node Peak Memory usage.
>>
>> For example one of the query the estimate was 9GB while the usage was
>> 30GB.
>>
>> I have 3 questions:
>>
>> 1- Do you think the compute stats will help here?
>> 2- How i can reduce this gap?
>> 3- Does the resource pools based on the Estimate per Mode peak memory?
>> so if for such query i put the default max memory limit to 5GB, will this
>> query fail?
>>
>>
>>
>
Re: Estimate peak memory VS used peak memory
Posted by Jim Apple <jb...@cloudera.com>.
I think compute stats will help estimates be more accurate.
On Thu, Feb 22, 2018 at 4:48 AM, Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Guys,
>
> In Impala version 2.10 i see a lot that the Estimate per Mode peak memory
> can reach X30 times of the Per Node Peak Memory usage.
>
> For example one of the query the estimate was 9GB while the usage was 30GB.
>
> I have 3 questions:
>
> 1- Do you think the compute stats will help here?
> 2- How i can reduce this gap?
> 3- Does the resource pools based on the Estimate per Mode peak memory? so
> if for such query i put the default max memory limit to 5GB, will this
> query fail?
>
>
>