You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2023/03/09 03:30:00 UTC
[jira] [Resolved] (SPARK-42703) How to use Fair Scheduler Pools
[ https://issues.apache.org/jira/browse/SPARK-42703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-42703.
----------------------------------
Resolution: Invalid
> How to use Fair Scheduler Pools
> -------------------------------
>
> Key: SPARK-42703
> URL: https://issues.apache.org/jira/browse/SPARK-42703
> Project: Spark
> Issue Type: Question
> Components: Scheduler
> Affects Versions: 3.2.3
> Reporter: LiJie2023
> Priority: Major
> Attachments: image-2023-03-08-09-53-35-867.png
>
>
> I have two questions to ask:
> # I wrote a demo referring to the official website, but it didn't meet my expectations. I don't know if there was a problem with my writing.I hope that when I use the following fairscheduler.xml, pool1 always performs tasks before pool2
> # What is the relationship between "spark.scheduler.mode" and "{{{}schedulingMode{}}}" in fairscheduler.xml？
>
>
> {code:java}
> object MultiJobTest {
> def main(args: Array[String]): Unit = {
> val conf = new SparkConf()
> conf.setAppName("test-pool").setMaster("local[1]")
> conf.set("spark.scheduler.mode", "FAIR")
> conf.set("spark.scheduler.allocation.file", "file:///D:/tmp/input/fairscheduler.xml")
> val sparkContext = new SparkContext(conf)
> val data: RDD[String] = sparkContext.textFile("file:///D:/tmp/input/input.txt")
> val rdd = data.flatMap(_.split(","))
> .map(x => (x(0), x(0)))
> new Thread(() => {
> sparkContext.setLocalProperty("spark.scheduler.pool", "pool1")
> rdd.foreachAsync(x => {
> println("1======start==" + new SimpleDateFormat("HH:mm:ss").format(new Date()))
> Thread.sleep(10000)
> println("1======end==" + new SimpleDateFormat("HH:mm:ss").format(new Date()))
> })
> }).start()
> new Thread(() => {
> sparkContext.setLocalProperty("spark.scheduler.pool", "pool2")
> rdd.foreachAsync(x => {
> println("2======start==" + new SimpleDateFormat("HH:mm:ss").format(new Date()))
> Thread.sleep(10000)
> println("2======end==" + new SimpleDateFormat("HH:mm:ss").format(new Date()))
> })
> }).start()
> TimeUnit.MINUTES.sleep(2)
> sparkContext.stop()
> }
> } {code}
>
> fairscheduler.xml
>
> {code:java}
> <?xml version="1.0"?><!--
> Licensed to the Apache Software Foundation (ASF) under one or more
> contributor license agreements. See the NOTICE file distributed with
> this work for additional information regarding copyright ownership.
> The ASF licenses this file to You under the Apache License, Version 2.0
> (the "License"); you may not use this file except in compliance with
> the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License.
> --><allocations>
> <pool name="pool1">
> <schedulingMode>FAIR</schedulingMode>
> <weight>100</weight>
> <minShare>0</minShare>
> </pool>
> <pool name="pool2">
> <schedulingMode>FAIR</schedulingMode>
> <weight>1</weight>
> <minShare>0</minShare>
> </pool>
> </allocations> {code}
>
>
> input.txt
>
> {code:java}
> aa bb {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org