You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by Thomas Larsson <th...@klarna.com> on 2014/04/23 15:51:05 UTC

Setting limits for hive users

Hello.

We recently had a user that ran an ad-hoc hive query with a JOIN clause
without an ON-predicate, resulting in a huge resultset that then resulted
in our hdfs storage becoming full.

I am wondering what support and strategies there are to help limit the
damage that ad-hoc queries like this can do. I have looked at HDFS quotas
which might be of some help. Is there anything else.

Any tips and good links would be appreciated.

Best Regards
Thomas

Re: Setting limits for hive users

Posted by Thomas Larsson <th...@klarna.com>.

Thanks Matt. Good stuff.


On Wed, Apr 23, 2014 at 4:43 PM, Matt Tucker <ma...@gmail.com> wrote:

> I would recommend looking at Hive Strict Mode first. It helps users think
> about their queries by throwing errors for certain operations that may be
> unexpectedly bad, like a full table scan over a partitioned fact table,
> when only a certain subset may be needed.
>
>
> http://my.safaribooksonline.com/book/databases/hadoop/9781449326944/10dot-tuning/strict_mode_tuning_html
>  On Apr 23, 2014 9:51 AM, "Thomas Larsson" <th...@klarna.com>
> wrote:
>
>> Hello.
>>
>> We recently had a user that ran an ad-hoc hive query with a JOIN clause
>> without an ON-predicate, resulting in a huge resultset that then resulted
>> in our hdfs storage becoming full.
>>
>> I am wondering what support and strategies there are to help limit the
>> damage that ad-hoc queries like this can do. I have looked at HDFS quotas
>> which might be of some help. Is there anything else.
>>
>>  Any tips and good links would be appreciated.
>>
>> Best Regards
>> Thomas
>>
>>

Re: Setting limits for hive users

Posted by Matt Tucker <ma...@gmail.com>.

I would recommend looking at Hive Strict Mode first. It helps users think
about their queries by throwing errors for certain operations that may be
unexpectedly bad, like a full table scan over a partitioned fact table,
when only a certain subset may be needed.

http://my.safaribooksonline.com/book/databases/hadoop/9781449326944/10dot-tuning/strict_mode_tuning_html
 On Apr 23, 2014 9:51 AM, "Thomas Larsson" <th...@klarna.com>
wrote:

> Hello.
>
> We recently had a user that ran an ad-hoc hive query with a JOIN clause
> without an ON-predicate, resulting in a huge resultset that then resulted
> in our hdfs storage becoming full.
>
> I am wondering what support and strategies there are to help limit the
> damage that ad-hoc queries like this can do. I have looked at HDFS quotas
> which might be of some help. Is there anything else.
>
> Any tips and good links would be appreciated.
>
> Best Regards
> Thomas
>
>