submit job to fair scheduler pool

[Jonathan Cao <jo...@rockyou.com>]

I am trying to set up the hive to work with facebook fair scheduler. Does
hive provide way to allow user to submit the hive job to a particular fair
scheduler pool? If I set mapred.fairscheduler.poolnameproperty to "user.name"
and hive jobs submitted by different users will be sent to the right pools
of the users. Is there any other way to
allow the hive user to submit job to a particular pool?

Thanks,
Jonathan

Re: submit job to fair scheduler pool

[Jonathan Cao <jo...@rockyou.com>]

Hey, Matei,
Thanks for your email. It looks that with your suggested config and passing
"-hiveconf pool.name=HIVE_POOL" solve the problem.

Jonathan

On Tue, Mar 31, 2009 at 8:51 AM, Matei Zaharia <ma...@cloudera.com> wrote:

> Hi Jonathan,
>
> To have user.name be the default pool name property but allow changing
> pool name from the command line too, you can use the following
> configuration:
>
>   <property>
>     <name>mapred.fairscheduler.poolnameproperty</name>
>     <value>pool.name</value>
>   </property>
>
>   <property>
>     <name>pool.name</name>
>     <value>${user.name}</value>
>   </property>
>
> This sets the pool name property to "pool.name" (which is something I made
> up) but sets the default value of pool.name to user.name. You can then
> submit a job to a specific pool by either giving it the option -D
> pool.name=foo if you use bin/hadoop jar or setting the pool.name property
> in the JobConf when creating the job.
>
> Matei
>
>
>> From: Jonathan Cao <jo...@rockyou.com>
>> Date: Mon, Mar 30, 2009 at 7:10 PM
>> Subject: submit job to fair scheduler pool
>> To: hive-user@hadoop.apache.org
>> Cc: james warren <ja...@rockyou.com>
>>
>>
>>
>> I am trying to set up the hive to work with facebook fair scheduler. Does
>> hive provide way to allow user to submit the hive job to a particular fair
>> scheduler pool? If I set mapred.fairscheduler.poolnameproperty to "
>> user.name" and hive jobs submitted by different users will be sent to the
>> right pools of the users. Is there any other way to
>> allow the hive user to submit job to a particular pool?
>>
>> Thanks,
>> Jonathan
>>
>>
>

Re: submit job to fair scheduler pool

[Matei Zaharia <ma...@cloudera.com>]

Hi Jonathan,

To have user.name be the default pool name property but allow changing pool
name from the command line too, you can use the following configuration:

  <property>
    <name>mapred.fairscheduler.poolnameproperty</name>
    <value>pool.name</value>
  </property>

  <property>
    <name>pool.name</name>
    <value>${user.name}</value>
  </property>

This sets the pool name property to "pool.name" (which is something I made
up) but sets the default value of pool.name to user.name. You can then
submit a job to a specific pool by either giving it the option -D pool.name=foo
if you use bin/hadoop jar or setting the pool.name property in the JobConf
when creating the job.

Matei


> From: Jonathan Cao <jo...@rockyou.com>
> Date: Mon, Mar 30, 2009 at 7:10 PM
> Subject: submit job to fair scheduler pool
> To: hive-user@hadoop.apache.org
> Cc: james warren <ja...@rockyou.com>
>
>
> I am trying to set up the hive to work with facebook fair scheduler. Does
> hive provide way to allow user to submit the hive job to a particular fair
> scheduler pool? If I set mapred.fairscheduler.poolnameproperty to "
> user.name" and hive jobs submitted by different users will be sent to the
> right pools of the users. Is there any other way to
> allow the hive user to submit job to a particular pool?
>
> Thanks,
> Jonathan
>
>