You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by tian boxiu <bo...@gmail.com> on 2019/07/31 10:21:53 UTC
Support priority of the Flink YARN application in Flink 1.9
To: user@flink.apache.org
Title: Support priority of the Flink YARN application in Flink 1.9
Hello everyone,
Many of our batch jobs have changed the execution engine from spark to
flink. The flink is deployed on the yarn cluster. In some scenarios,
high-priority core tasks need to be submitted on the preemptive queue.
Previously, when running in spark, users configured the spark.yarn.priority
to control the priority, ranging from 1 to 9.
Currently, flink 1.9 does not support yarn priority submission. The default
priority of yarn submission jobs is 1. Based on this, we implemented the
submission of the priority range from 1 to 9(from low to high priority).
When the user does not configure the parameters, the priority is still 1.
After we push the new feature online in June, The system has been tested to
run properly.
So I want to create an issue and push the improvement back to community.
Hope someone can review it, thanks~
Boxiu
Re: Support priority of the Flink YARN application in Flink 1.9
Posted by tian boxiu <bo...@gmail.com>.
Hello everyone,
I have created a Jira issue in:
https://issues.apache.org/jira/browse/FLINK-13548. Thanks for reviewing it.
Thank you~
Boxiu
Fabian Hueske <fh...@gmail.com> 于2019年8月2日周五 下午4:09写道：
> Hi Boxiu,
>
> This sounds like a good feature.
>
> Please have a look at our contribution guidelines [1].
> To propose a feature, you should open a Jira issue [2] and start a
> discussion there.
>
> Please note that the feature freeze for the Flink 1.9 release happened a
> few weeks ago.
> The community is currently working on fixing blockers and testing the
> release and won't accept new features.
> A new feature would go into the next release.
>
> Best, Fabian
>
> [1] https://flink.apache.org/contributing/contribute-code.html
> [2] https://issues.apache.org/jira/projects/FLINK/summary
>
> Am Mi., 31. Juli 2019 um 13:24 Uhr schrieb Xintong Song <
> tonysong820@gmail.com>:
>
>> Thanks for bringing this up, Boxiu.
>>
>>
>> The problem make sense to me. For me the concern is should we limit the
>> priorities to 1-9 or not.
>>
>>
>> I think it would be good to open a jira issue and have the discussion
>> there.
>>
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>> On Wed, Jul 31, 2019 at 12:22 PM tian boxiu <bo...@gmail.com>
>> wrote:
>>
>>> To: user@flink.apache.org
>>> Title: Support priority of the Flink YARN application in Flink 1.9
>>>
>>> Hello everyone,
>>>
>>> Many of our batch jobs have changed the execution engine from spark to
>>> flink. The flink is deployed on the yarn cluster. In some scenarios,
>>> high-priority core tasks need to be submitted on the preemptive queue.
>>> Previously, when running in spark, users configured the spark.yarn.priority
>>> to control the priority, ranging from 1 to 9.
>>>
>>> Currently, flink 1.9 does not support yarn priority submission. The
>>> default priority of yarn submission jobs is 1. Based on this, we
>>> implemented the submission of the priority range from 1 to 9(from low to
>>> high priority). When the user does not configure the parameters, the
>>> priority is still 1. After we push the new feature online in June, The
>>> system has been tested to run properly.
>>>
>>> So I want to create an issue and push the improvement back to community.
>>> Hope someone can review it, thanks~
>>>
>>> Boxiu
>>>
>>
Re: Support priority of the Flink YARN application in Flink 1.9
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Boxiu,
This sounds like a good feature.
Please have a look at our contribution guidelines [1].
To propose a feature, you should open a Jira issue [2] and start a
discussion there.
Please note that the feature freeze for the Flink 1.9 release happened a
few weeks ago.
The community is currently working on fixing blockers and testing the
release and won't accept new features.
A new feature would go into the next release.
Best, Fabian
[1] https://flink.apache.org/contributing/contribute-code.html
[2] https://issues.apache.org/jira/projects/FLINK/summary
Am Mi., 31. Juli 2019 um 13:24 Uhr schrieb Xintong Song <
tonysong820@gmail.com>:
> Thanks for bringing this up, Boxiu.
>
>
> The problem make sense to me. For me the concern is should we limit the
> priorities to 1-9 or not.
>
>
> I think it would be good to open a jira issue and have the discussion
> there.
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Wed, Jul 31, 2019 at 12:22 PM tian boxiu <bo...@gmail.com> wrote:
>
>> To: user@flink.apache.org
>> Title: Support priority of the Flink YARN application in Flink 1.9
>>
>> Hello everyone,
>>
>> Many of our batch jobs have changed the execution engine from spark to
>> flink. The flink is deployed on the yarn cluster. In some scenarios,
>> high-priority core tasks need to be submitted on the preemptive queue.
>> Previously, when running in spark, users configured the spark.yarn.priority
>> to control the priority, ranging from 1 to 9.
>>
>> Currently, flink 1.9 does not support yarn priority submission. The
>> default priority of yarn submission jobs is 1. Based on this, we
>> implemented the submission of the priority range from 1 to 9(from low to
>> high priority). When the user does not configure the parameters, the
>> priority is still 1. After we push the new feature online in June, The
>> system has been tested to run properly.
>>
>> So I want to create an issue and push the improvement back to community.
>> Hope someone can review it, thanks~
>>
>> Boxiu
>>
>
Re: Support priority of the Flink YARN application in Flink 1.9
Posted by Xintong Song <to...@gmail.com>.
Thanks for bringing this up, Boxiu.
The problem make sense to me. For me the concern is should we limit the
priorities to 1-9 or not.
I think it would be good to open a jira issue and have the discussion
there.
Thank you~
Xintong Song
On Wed, Jul 31, 2019 at 12:22 PM tian boxiu <bo...@gmail.com> wrote:
> To: user@flink.apache.org
> Title: Support priority of the Flink YARN application in Flink 1.9
>
> Hello everyone,
>
> Many of our batch jobs have changed the execution engine from spark to
> flink. The flink is deployed on the yarn cluster. In some scenarios,
> high-priority core tasks need to be submitted on the preemptive queue.
> Previously, when running in spark, users configured the spark.yarn.priority
> to control the priority, ranging from 1 to 9.
>
> Currently, flink 1.9 does not support yarn priority submission. The
> default priority of yarn submission jobs is 1. Based on this, we
> implemented the submission of the priority range from 1 to 9(from low to
> high priority). When the user does not configure the parameters, the
> priority is still 1. After we push the new feature online in June, The
> system has been tested to run properly.
>
> So I want to create an issue and push the improvement back to community.
> Hope someone can review it, thanks~
>
> Boxiu
>