You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Stephen Sprague <sp...@gmail.com> on 2013/11/18 21:59:21 UTC
HiveServer2
A word of warning for users of HiveServer2 - version 0.11 at least. This
puppy has the ability crash and/or hang your server with a memory leak.
Apparently its not new since googling shows this discussed before and i see
reference to a workaround here:
https://cwiki.apache.org/confluence/display/Hive/Setting+up+HiveServer2
Anyhoo. Consider this a Public Service Announcement. Take heed.
Regards,
Stephen.
Re: HiveServer2
Posted by Roberto Congiu <ro...@openx.com>.
Yeah, we had issues with zk locking too....but this workaround works pretty
well and our ops won't get paged at 4am :)
On Mon, Nov 18, 2013 at 2:16 PM, Edward Capriolo <ed...@gmail.com>wrote:
> Yikes: I personally found that the most problematic thing was hiveserver +
> zk locking, if you do not need that turn it off. Other then that we just
> wrote a good nagios check..it runs a query (one that does not invoke a map
> reduce job). That seems to spot the problems quickly and allow our ops to
> restart the bad instance.
>
>
> On Mon, Nov 18, 2013 at 5:11 PM, Roberto Congiu <ro...@openx.com>wrote:
>
>> We've also had issues with both hiveserver1 and 2 crashing because of
>> heap exhaustion, but instead of restarting it periodically we took a
>> different approach, that is, abstracting the part of the interface we
>> needed, and implemented an adapter that implements the same method as
>> thrift, but forking a shell, sending commands to it, and parsing the
>> results.
>> It is slow, but it's fast enough for our hourly process that loads data
>> in hive EXTERNAL tables for which we need extra reliability.
>>
>> R.
>>
>>
>> On Mon, Nov 18, 2013 at 1:24 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>
>>> Thanks for pointing out any issue. HiveServer1 is significantly less
>>> robust. We have run HS1 behind a load balancer/proxy and rotated/restarted
>>> "angry" instances.
>>>
>>>
>>> On Mon, Nov 18, 2013 at 3:59 PM, Stephen Sprague <sp...@gmail.com>wrote:
>>>
>>>> A word of warning for users of HiveServer2 - version 0.11 at least.
>>>> This puppy has the ability crash and/or hang your server with a memory leak.
>>>>
>>>> Apparently its not new since googling shows this discussed before and i
>>>> see reference to a workaround here:
>>>>
>>>> https://cwiki.apache.org/confluence/display/Hive/Setting+up+HiveServer2
>>>>
>>>> Anyhoo. Consider this a Public Service Announcement. Take heed.
>>>>
>>>> Regards,
>>>> Stephen.
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> ----------------------------------------------------------
>> Good judgement comes with experience.
>> Experience comes with bad judgement.
>> ----------------------------------------------------------
>> Roberto Congiu - Data Engineer - OpenX
>> tel: +1 626 466 1141
>>
>
>
--
----------------------------------------------------------
Good judgement comes with experience.
Experience comes with bad judgement.
----------------------------------------------------------
Roberto Congiu - Data Engineer - OpenX
tel: +1 626 466 1141
Re: HiveServer2
Posted by Edward Capriolo <ed...@gmail.com>.
Yikes: I personally found that the most problematic thing was hiveserver +
zk locking, if you do not need that turn it off. Other then that we just
wrote a good nagios check..it runs a query (one that does not invoke a map
reduce job). That seems to spot the problems quickly and allow our ops to
restart the bad instance.
On Mon, Nov 18, 2013 at 5:11 PM, Roberto Congiu <ro...@openx.com>wrote:
> We've also had issues with both hiveserver1 and 2 crashing because of heap
> exhaustion, but instead of restarting it periodically we took a different
> approach, that is, abstracting the part of the interface we needed, and
> implemented an adapter that implements the same method as thrift, but
> forking a shell, sending commands to it, and parsing the results.
> It is slow, but it's fast enough for our hourly process that loads data in
> hive EXTERNAL tables for which we need extra reliability.
>
> R.
>
>
> On Mon, Nov 18, 2013 at 1:24 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> Thanks for pointing out any issue. HiveServer1 is significantly less
>> robust. We have run HS1 behind a load balancer/proxy and rotated/restarted
>> "angry" instances.
>>
>>
>> On Mon, Nov 18, 2013 at 3:59 PM, Stephen Sprague <sp...@gmail.com>wrote:
>>
>>> A word of warning for users of HiveServer2 - version 0.11 at least. This
>>> puppy has the ability crash and/or hang your server with a memory leak.
>>>
>>> Apparently its not new since googling shows this discussed before and i
>>> see reference to a workaround here:
>>>
>>> https://cwiki.apache.org/confluence/display/Hive/Setting+up+HiveServer2
>>>
>>> Anyhoo. Consider this a Public Service Announcement. Take heed.
>>>
>>> Regards,
>>> Stephen.
>>>
>>>
>>>
>>>
>>
>
>
> --
> ----------------------------------------------------------
> Good judgement comes with experience.
> Experience comes with bad judgement.
> ----------------------------------------------------------
> Roberto Congiu - Data Engineer - OpenX
> tel: +1 626 466 1141
>
Re: HiveServer2
Posted by Roberto Congiu <ro...@openx.com>.
We've also had issues with both hiveserver1 and 2 crashing because of heap
exhaustion, but instead of restarting it periodically we took a different
approach, that is, abstracting the part of the interface we needed, and
implemented an adapter that implements the same method as thrift, but
forking a shell, sending commands to it, and parsing the results.
It is slow, but it's fast enough for our hourly process that loads data in
hive EXTERNAL tables for which we need extra reliability.
R.
On Mon, Nov 18, 2013 at 1:24 PM, Edward Capriolo <ed...@gmail.com>wrote:
> Thanks for pointing out any issue. HiveServer1 is significantly less
> robust. We have run HS1 behind a load balancer/proxy and rotated/restarted
> "angry" instances.
>
>
> On Mon, Nov 18, 2013 at 3:59 PM, Stephen Sprague <sp...@gmail.com>wrote:
>
>> A word of warning for users of HiveServer2 - version 0.11 at least. This
>> puppy has the ability crash and/or hang your server with a memory leak.
>>
>> Apparently its not new since googling shows this discussed before and i
>> see reference to a workaround here:
>>
>> https://cwiki.apache.org/confluence/display/Hive/Setting+up+HiveServer2
>>
>> Anyhoo. Consider this a Public Service Announcement. Take heed.
>>
>> Regards,
>> Stephen.
>>
>>
>>
>>
>
--
----------------------------------------------------------
Good judgement comes with experience.
Experience comes with bad judgement.
----------------------------------------------------------
Roberto Congiu - Data Engineer - OpenX
tel: +1 626 466 1141
Re: HiveServer2
Posted by Edward Capriolo <ed...@gmail.com>.
Thanks for pointing out any issue. HiveServer1 is significantly less
robust. We have run HS1 behind a load balancer/proxy and rotated/restarted
"angry" instances.
On Mon, Nov 18, 2013 at 3:59 PM, Stephen Sprague <sp...@gmail.com> wrote:
> A word of warning for users of HiveServer2 - version 0.11 at least. This
> puppy has the ability crash and/or hang your server with a memory leak.
>
> Apparently its not new since googling shows this discussed before and i
> see reference to a workaround here:
>
> https://cwiki.apache.org/confluence/display/Hive/Setting+up+HiveServer2
>
> Anyhoo. Consider this a Public Service Announcement. Take heed.
>
> Regards,
> Stephen.
>
>
>
>
Re: HiveServer2
Posted by Navis류승우 <na...@nexr.com>.
I've booked on https://issues.apache.org/jira/browse/HIVE-5858 for the
ALTER TABLE issue, mentioned by David Morel (Thanks).
2013/11/20 David Morel <dm...@gmail.com>
> On 18 Nov 2013, at 21:59, Stephen Sprague wrote:
>
> > A word of warning for users of HiveServer2 - version 0.11 at least. This
> > puppy has the ability crash and/or hang your server with a memory leak.
> >
> > Apparently its not new since googling shows this discussed before and i
> see
> > reference to a workaround here:
> >
> > https://cwiki.apache.org/confluence/display/Hive/Setting+up+HiveServer2
> >
> > Anyhoo. Consider this a Public Service Announcement. Take heed.
> >
> > Regards,
> > Stephen.
>
> When setting fs.hdfs.impl.disable.cache to false I have all my ALTER TABLE
> statements involving managed tables throw an Error 1 in Hive (nothing
> more).
> Can anyone confirm that behaviour?
>
> David
>
Re: HiveServer2
Posted by David Morel <dm...@gmail.com>.
On 18 Nov 2013, at 21:59, Stephen Sprague wrote:
> A word of warning for users of HiveServer2 - version 0.11 at least. This
> puppy has the ability crash and/or hang your server with a memory leak.
>
> Apparently its not new since googling shows this discussed before and i see
> reference to a workaround here:
>
> https://cwiki.apache.org/confluence/display/Hive/Setting+up+HiveServer2
>
> Anyhoo. Consider this a Public Service Announcement. Take heed.
>
> Regards,
> Stephen.
When setting fs.hdfs.impl.disable.cache to false I have all my ALTER TABLE
statements involving managed tables throw an Error 1 in Hive (nothing more).
Can anyone confirm that behaviour?
David