You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Laurence Brown via user <us...@hive.apache.org> on 2022/09/05 12:17:48 UTC
HiveServer2 slowly increaseing background CPU usage until restated
Hi
We're using Hive 2.3.1, we recently migrated our production amazon EC2 instance types from r5.24xlarge to r6i.32xlarge
on the r6 instance we have seen steady cpu usage growth that can all be attributed to our org.apache.hive.service.server.HiveServer2
Even when this change is unreleased and this process is going effectively unused the CPU usage grows slowly until we restart that process
In the attached graph you can see that CPU usage grows until we restart HiveServer2 after that it remains stable for a while and then usage starts growing on HiveServer2 .
After we restarted that process we failed back to our previous server (leaving this server unused) but the CPU usage on HiveServer2 on this server continue to grow
[cid:image001.png@01D8C129.E33179C0]
We've since built instances in dev with both r5 and r6i and all the r6i instances have the above problem and all the r5 do not.....
Does anyone have any idea why this might be?
----------------------------------------------------------------------
This email and any attachment is confidential. If you are not the intended recipient, please delete this message. Macquarie does not guarantee the integrity of any emails or attachments. For important disclosures and information about the incorporation and regulated status of Macquarie Group entities please see: www.macquarie.com/disclosures
Re: HiveServer2 slowly increaseing background CPU usage until restated
Posted by Deepak Goel <de...@gmail.com>.
The utilisations on r5 instance is flat and low. This only means you are
missing a library update (Hadoop?) on the r6i instance.
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Wed, Sep 7, 2022 at 5:02 PM Laurence Brown via user <us...@hive.apache.org>
wrote:
> Hi
>
> Thanks for your replies
>
>
>
> We don’t have specific metrics for that processes heap but the memory
> utilization on the box was stable while the CPU usage was growing.
>
> Correspondingly this is the cpu usage of the same software on an r5
> instance
>
>
>
>
>
>
> *From:* gwenael le barzic <gl...@gmail.com>
> *Sent:* 07 September 2022 10:57
> *To:* user@hive.apache.org
> *Subject:* Re: HiveServer2 slowly increaseing background CPU usage until
> restated
>
>
>
> *External Communication*
>
> Hello!
>
>
>
> In addition to the CPU, did you see if the heap usage of the HS2 was
> increasing, please ?
>
>
>
> Le mer. 7 sept. 2022 à 11:48, Deepak Goel <de...@gmail.com> a écrit :
>
> Can you please post the utilisation graph of r5.24xlarge too?
>
>
>
>
> Deepak
>
> "The greatness of a nation can be judged by the way its animals are
> treated - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
>
>
> "Plant a Tree, Go Green"
>
>
>
> Make In India : http://www.makeinindia.com/home
>
>
>
>
>
> On Wed, Sep 7, 2022 at 3:08 PM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
> Hi Laurence,
>
>
>
> It's hard to say just by seeing the graphs. Moreover Hive 2.3.1 is a quite
> old version so there are many things that may go wrong.
>
>
>
> I would suggest checking the logs and taking jstacks overtime and/or use a
> profiler (such as async-profiler[1]) to see what HS2 is actually doing
> while CPU usage grows.
>
>
>
> Best,
>
> Stamatis
>
>
>
> [1] https://github.com/jvm-profiling-tools/async-profiler
>
>
>
> On Mon, Sep 5, 2022 at 2:17 PM Laurence Brown via user <
> user@hive.apache.org> wrote:
>
>
>
> Hi
>
> We’re using Hive 2.3.1, we recently migrated our production amazon EC2
> instance types from r5.24xlarge to r6i.32xlarge
>
> on the r6 instance we have seen steady cpu usage growth that can all be
> attributed to our org.apache.hive.service.server.HiveServer2
>
> Even when this change is unreleased and this process is going effectively
> unused the CPU usage grows slowly until we restart that process
>
>
>
> In the attached graph you can see that CPU usage grows until we restart
> HiveServer2 after that it remains stable for a while and then usage starts
> growing on HiveServer2 .
> After we restarted that process we failed back to our previous server
> (leaving this server unused) but the CPU usage on HiveServer2 on this
> server continue to grow
>
>
>
> We’ve since built instances in dev with both r5 and r6i and all the r6i
> instances have the above problem and all the r5 do not…..
>
> Does anyone have any idea why this might be?
>
>
>
>
>
>
>
>
> ------------------------------
>
> This email and any attachment is confidential. If you are not the intended
> recipient, please delete this message. Macquarie does not guarantee the
> integrity of any emails or attachments. For important disclosures and
> information about the incorporation and regulated status of Macquarie Group
> entities please see: www.macquarie.com/disclosures
>
>
RE: HiveServer2 slowly increaseing background CPU usage until restated
Posted by Laurence Brown via user <us...@hive.apache.org>.
Hi
Thanks for your replies
We don’t have specific metrics for that processes heap but the memory utilization on the box was stable while the CPU usage was growing.
Correspondingly this is the cpu usage of the same software on an r5 instance
[cid:image001.png@01D8C2B2.56B02020]
From: gwenael le barzic <gl...@gmail.com>
Sent: 07 September 2022 10:57
To: user@hive.apache.org
Subject: Re: HiveServer2 slowly increaseing background CPU usage until restated
External Communication
Hello!
In addition to the CPU, did you see if the heap usage of the HS2 was increasing, please ?
Le mer. 7 sept. 2022 à 11:48, Deepak Goel <de...@gmail.com>> a écrit :
Can you please post the utilisation graph of r5.24xlarge too?
Deepak
"The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com<ma...@gmail.com>
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool<http://www.linkedin.com/in/deicool>
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Wed, Sep 7, 2022 at 3:08 PM Stamatis Zampetakis <za...@gmail.com>> wrote:
Hi Laurence,
It's hard to say just by seeing the graphs. Moreover Hive 2.3.1 is a quite old version so there are many things that may go wrong.
I would suggest checking the logs and taking jstacks overtime and/or use a profiler (such as async-profiler[1]) to see what HS2 is actually doing while CPU usage grows.
Best,
Stamatis
[1] https://github.com/jvm-profiling-tools/async-profiler
On Mon, Sep 5, 2022 at 2:17 PM Laurence Brown via user <us...@hive.apache.org>> wrote:
Hi
We’re using Hive 2.3.1, we recently migrated our production amazon EC2 instance types from r5.24xlarge to r6i.32xlarge
on the r6 instance we have seen steady cpu usage growth that can all be attributed to our org.apache.hive.service.server.HiveServer2
Even when this change is unreleased and this process is going effectively unused the CPU usage grows slowly until we restart that process
In the attached graph you can see that CPU usage grows until we restart HiveServer2 after that it remains stable for a while and then usage starts growing on HiveServer2 .
After we restarted that process we failed back to our previous server (leaving this server unused) but the CPU usage on HiveServer2 on this server continue to grow
[cid:image002.png@01D8C2B2.56B02020]
We’ve since built instances in dev with both r5 and r6i and all the r6i instances have the above problem and all the r5 do not…..
Does anyone have any idea why this might be?
________________________________
This email and any attachment is confidential. If you are not the intended recipient, please delete this message. Macquarie does not guarantee the integrity of any emails or attachments. For important disclosures and information about the incorporation and regulated status of Macquarie Group entities please see: www.macquarie.com/disclosures<http://www.macquarie.com/disclosures>
Re: HiveServer2 slowly increaseing background CPU usage until restated
Posted by gwenael le barzic <gl...@gmail.com>.
Hello!
In addition to the CPU, did you see if the heap usage of the HS2 was
increasing, please ?
Le mer. 7 sept. 2022 à 11:48, Deepak Goel <de...@gmail.com> a écrit :
> Can you please post the utilisation graph of r5.24xlarge too?
>
>
> Deepak
> "The greatness of a nation can be judged by the way its animals are
> treated - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> Make In India : http://www.makeinindia.com/home
>
>
> On Wed, Sep 7, 2022 at 3:08 PM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
>> Hi Laurence,
>>
>> It's hard to say just by seeing the graphs. Moreover Hive 2.3.1 is a
>> quite old version so there are many things that may go wrong.
>>
>> I would suggest checking the logs and taking jstacks overtime and/or use
>> a profiler (such as async-profiler[1]) to see what HS2 is actually doing
>> while CPU usage grows.
>>
>> Best,
>> Stamatis
>>
>> [1] https://github.com/jvm-profiling-tools/async-profiler
>>
>> On Mon, Sep 5, 2022 at 2:17 PM Laurence Brown via user <
>> user@hive.apache.org> wrote:
>>
>>>
>>>
>>> Hi
>>>
>>> We’re using Hive 2.3.1, we recently migrated our production amazon EC2
>>> instance types from r5.24xlarge to r6i.32xlarge
>>>
>>> on the r6 instance we have seen steady cpu usage growth that can all be
>>> attributed to our org.apache.hive.service.server.HiveServer2
>>>
>>> Even when this change is unreleased and this process is going
>>> effectively unused the CPU usage grows slowly until we restart that process
>>>
>>>
>>>
>>> In the attached graph you can see that CPU usage grows until we restart
>>> HiveServer2 after that it remains stable for a while and then usage starts
>>> growing on HiveServer2 .
>>> After we restarted that process we failed back to our previous server
>>> (leaving this server unused) but the CPU usage on HiveServer2 on this
>>> server continue to grow
>>>
>>>
>>>
>>> We’ve since built instances in dev with both r5 and r6i and all the r6i
>>> instances have the above problem and all the r5 do not…..
>>>
>>> Does anyone have any idea why this might be?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------
>>> This email and any attachment is confidential. If you are not the
>>> intended recipient, please delete this message. Macquarie does not
>>> guarantee the integrity of any emails or attachments. For important
>>> disclosures and information about the incorporation and regulated status of
>>> Macquarie Group entities please see: www.macquarie.com/disclosures
>>>
>>
Re: HiveServer2 slowly increaseing background CPU usage until restated
Posted by Deepak Goel <de...@gmail.com>.
Can you please post the utilisation graph of r5.24xlarge too?
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Wed, Sep 7, 2022 at 3:08 PM Stamatis Zampetakis <za...@gmail.com>
wrote:
> Hi Laurence,
>
> It's hard to say just by seeing the graphs. Moreover Hive 2.3.1 is a quite
> old version so there are many things that may go wrong.
>
> I would suggest checking the logs and taking jstacks overtime and/or use a
> profiler (such as async-profiler[1]) to see what HS2 is actually doing
> while CPU usage grows.
>
> Best,
> Stamatis
>
> [1] https://github.com/jvm-profiling-tools/async-profiler
>
> On Mon, Sep 5, 2022 at 2:17 PM Laurence Brown via user <
> user@hive.apache.org> wrote:
>
>>
>>
>> Hi
>>
>> We’re using Hive 2.3.1, we recently migrated our production amazon EC2
>> instance types from r5.24xlarge to r6i.32xlarge
>>
>> on the r6 instance we have seen steady cpu usage growth that can all be
>> attributed to our org.apache.hive.service.server.HiveServer2
>>
>> Even when this change is unreleased and this process is going effectively
>> unused the CPU usage grows slowly until we restart that process
>>
>>
>>
>> In the attached graph you can see that CPU usage grows until we restart
>> HiveServer2 after that it remains stable for a while and then usage starts
>> growing on HiveServer2 .
>> After we restarted that process we failed back to our previous server
>> (leaving this server unused) but the CPU usage on HiveServer2 on this
>> server continue to grow
>>
>>
>>
>> We’ve since built instances in dev with both r5 and r6i and all the r6i
>> instances have the above problem and all the r5 do not…..
>>
>> Does anyone have any idea why this might be?
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------
>> This email and any attachment is confidential. If you are not the
>> intended recipient, please delete this message. Macquarie does not
>> guarantee the integrity of any emails or attachments. For important
>> disclosures and information about the incorporation and regulated status of
>> Macquarie Group entities please see: www.macquarie.com/disclosures
>>
>
Re: HiveServer2 slowly increaseing background CPU usage until restated
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hi Laurence,
It's hard to say just by seeing the graphs. Moreover Hive 2.3.1 is a quite
old version so there are many things that may go wrong.
I would suggest checking the logs and taking jstacks overtime and/or use a
profiler (such as async-profiler[1]) to see what HS2 is actually doing
while CPU usage grows.
Best,
Stamatis
[1] https://github.com/jvm-profiling-tools/async-profiler
On Mon, Sep 5, 2022 at 2:17 PM Laurence Brown via user <us...@hive.apache.org>
wrote:
>
>
> Hi
>
> We’re using Hive 2.3.1, we recently migrated our production amazon EC2
> instance types from r5.24xlarge to r6i.32xlarge
>
> on the r6 instance we have seen steady cpu usage growth that can all be
> attributed to our org.apache.hive.service.server.HiveServer2
>
> Even when this change is unreleased and this process is going effectively
> unused the CPU usage grows slowly until we restart that process
>
>
>
> In the attached graph you can see that CPU usage grows until we restart
> HiveServer2 after that it remains stable for a while and then usage starts
> growing on HiveServer2 .
> After we restarted that process we failed back to our previous server
> (leaving this server unused) but the CPU usage on HiveServer2 on this
> server continue to grow
>
>
>
> We’ve since built instances in dev with both r5 and r6i and all the r6i
> instances have the above problem and all the r5 do not…..
>
> Does anyone have any idea why this might be?
>
>
>
>
>
>
>
>
> ------------------------------
> This email and any attachment is confidential. If you are not the intended
> recipient, please delete this message. Macquarie does not guarantee the
> integrity of any emails or attachments. For important disclosures and
> information about the incorporation and regulated status of Macquarie Group
> entities please see: www.macquarie.com/disclosures
>