You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Xiaoyu Wang <wa...@gmail.com> on 2014/12/31 08:55:49 UTC
How to set local property in beeline connect to the spark thrift server
Hi all!
I use Spark SQL1.2 start the thrift server on yarn.
I want to use fair scheduler in the thrift server.
I set the properties in spark-defaults.conf like this:
spark.scheduler.mode FAIR
spark.scheduler.allocation.file
/opt/spark-1.2.0-bin-2.4.1/conf/fairscheduler.xml
In the thrift server UI can see the scheduler pool is ok.
[image: 内嵌图片 1]
How can I specify one sql job to the "test" pool.
By default the sql job run in the "default" pool.
In the http://spark.apache.org/docs/latest/job-scheduling.html document
I see sc.setLocalProperty("spark.scheduler.pool", "pool1") can be set in
the code.
In the beeline I execute "set spark.scheduler.pool=test", but no use.
But how can I set the local property in the beeline?
Re: How to set local property in beeline connect to the spark thrift server
Posted by Xiaoyu Wang <wa...@gmail.com>.
Yi Tian, Thanks a lot!
It works!
I find the variable set guide in
http://spark.apache.org/docs/latest/sql-programming-guide.html
2014-12-31 17:12 GMT+08:00 田毅 <ti...@gmail.com>:
> Hi, Xiaoyu!
>
> You can use `spark.sql.thriftserver.scheduler.pool` instead of
> `spark.scheduler.pool` only in spark thrift server.
>
>
>
> On Wed, Dec 31, 2014 at 3:55 PM, Xiaoyu Wang <wa...@gmail.com> wrote:
>
>> Hi all!
>>
>> I use Spark SQL1.2 start the thrift server on yarn.
>>
>> I want to use fair scheduler in the thrift server.
>>
>> I set the properties in spark-defaults.conf like this:
>> spark.scheduler.mode FAIR
>> spark.scheduler.allocation.file
>> /opt/spark-1.2.0-bin-2.4.1/conf/fairscheduler.xml
>>
>> In the thrift server UI can see the scheduler pool is ok.
>> [image: 内嵌图片 1]
>>
>> How can I specify one sql job to the "test" pool.
>> By default the sql job run in the "default" pool.
>>
>> In the http://spark.apache.org/docs/latest/job-scheduling.html document
>> I see sc.setLocalProperty("spark.scheduler.pool", "pool1") can be set in
>> the code.
>>
>> In the beeline I execute "set spark.scheduler.pool=test", but no use.
>> But how can I set the local property in the beeline?
>>
>>
>
Re: How to set local property in beeline connect to the spark thrift server
Posted by 田毅 <ti...@gmail.com>.
Hi, Xiaoyu!
You can use `spark.sql.thriftserver.scheduler.pool` instead of
`spark.scheduler.pool` only in spark thrift server.
On Wed, Dec 31, 2014 at 3:55 PM, Xiaoyu Wang <wa...@gmail.com> wrote:
> Hi all!
>
> I use Spark SQL1.2 start the thrift server on yarn.
>
> I want to use fair scheduler in the thrift server.
>
> I set the properties in spark-defaults.conf like this:
> spark.scheduler.mode FAIR
> spark.scheduler.allocation.file
> /opt/spark-1.2.0-bin-2.4.1/conf/fairscheduler.xml
>
> In the thrift server UI can see the scheduler pool is ok.
> [image: 内嵌图片 1]
>
> How can I specify one sql job to the "test" pool.
> By default the sql job run in the "default" pool.
>
> In the http://spark.apache.org/docs/latest/job-scheduling.html document
> I see sc.setLocalProperty("spark.scheduler.pool", "pool1") can be set in
> the code.
>
> In the beeline I execute "set spark.scheduler.pool=test", but no use.
> But how can I set the local property in the beeline?
>
>