You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by Jose Rozanec <jo...@mercadolibre.com> on 2016/09/27 22:52:08 UTC

Query consuming all resources

Hi,

We have a Hive cluster. We notice that some queries consume all resources,
which is not desirable to us, since we want to grant some degree of
parallelism to incoming ones: any incoming query should be able to do at
least some progress, not just wait the big one finish.

Is there way to do so? We use Hive 2.1.0 with Tez engine.

Thank you in advance,

Joze.

Re: Query consuming all resources

Posted by Per Ullberg <pe...@klarna.com>.

What Jörn said. We use the capacity scheduler to be able to give priority
to some user groups over others.

Regards
/Pelle

On Wednesday, September 28, 2016, Jörn Franke <jo...@gmail.com> wrote:

> You need to configure queues in yarn and use the fairscheduler. From your
> use case it looks like you need to also configure pre-emption
>
> > On 28 Sep 2016, at 00:52, Jose Rozanec <jose.rozanec@mercadolibre.com
> <javascript:;>> wrote:
> >
> > Hi,
> >
> > We have a Hive cluster. We notice that some queries consume all
> resources, which is not desirable to us, since we want to grant some degree
> of parallelism to incoming ones: any incoming query should be able to do at
> least some progress, not just wait the big one finish.
> >
> > Is there way to do so? We use Hive 2.1.0 with Tez engine.
> >
> > Thank you in advance,
> >
> > Joze.
>


-- 

*Per Ullberg*
Data Vault Tech Lead
Odin Uppsala
+46 701612693 <+46+701612693>

Klarna AB (publ)
Sveavägen 46, 111 34 Stockholm
Tel: +46 8 120 120 00 <+46812012000>
Reg no: 556737-0431
klarna.com

Re: Query consuming all resources

Posted by Jörn Franke <jo...@gmail.com>.

You need to configure queues in yarn and use the fairscheduler. From your use case it looks like you need to also configure pre-emption 

> On 28 Sep 2016, at 00:52, Jose Rozanec <jo...@mercadolibre.com> wrote:
> 
> Hi, 
> 
> We have a Hive cluster. We notice that some queries consume all resources, which is not desirable to us, since we want to grant some degree of parallelism to incoming ones: any incoming query should be able to do at least some progress, not just wait the big one finish.
> 
> Is there way to do so? We use Hive 2.1.0 with Tez engine.
> 
> Thank you in advance,
> 
> Joze.