You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by li-zhihui <gi...@git.apache.org> on 2014/06/19 10:32:08 UTC
[GitHub] spark pull request: [SPARK-2193] Improve tasks preferrd locality b...
GitHub user li-zhihui opened a pull request:
https://github.com/apache/spark/pull/1131
[SPARK-2193] Improve tasks preferrd locality by sorting tasks partial or...
Now, the last executor(s) maybe not get it’s preferred task(s), although these tasks have build in pendingTasksForHosts map. Because executers pick up tasks sequential, their preferred task(s) maybe picked up by other executors.
This appearance can be eliminated by sorting tasks partial ordering. Executor pick up task by host’s order of task’s preferredLocation, that mean, executor firstly pick up all tasks which task.preferredLocations.1 = executor.hostName, then secondly…
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/li-zhihui/spark sorttasks
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1131.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1131
----
commit a38d5f5baaae54d6e6d9a9e463c265ef4e5fc968
Author: li-zhihui <zh...@intel.com>
Date: 2014-06-19T08:09:09Z
[SPARK-2193] Improve tasks preferrd locality by sorting tasks partial ordering.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2193] Improve tasks preferrd locality b...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1131#issuecomment-46535871
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2193] Improve tasks preferrd locality b...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1131#issuecomment-54694656
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2193] Improve tasks preferrd locality b...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1131
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2193] Improve tasks preferrd locality b...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/1131#issuecomment-46537074
A preferred task for one worker might be picked up by another worker on process/node/rack only if they are at the same locality level : in which case, it is irrelevant which worker picks it up since both are at same locality level. I am probably missing why this is required ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2193] Improve tasks preferrd locality b...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1131#issuecomment-54372737
Hi @li-zhihui,
Sorry for allowing this to sit unreviewed for so long.
To check my understanding, the original issue was that an individual executor's pending task queue might have non-preferred tasks that appear ahead of preferred ones in the queue? It looks like this might have been partially addressed by #1313, which modified TaskSetManage to maintain a separate list of pending tasks without locality preferences: https://github.com/apache/spark/commit/63bdb1f41b4895e3a9444f7938094438a94d3007#diff-bad3987c83bd22d46416d3dd9d208e76R193.
Since we now maintain separate lists to track pending tasks for executors, hosts, and racks, I don't think that we need this sorting. If you agree, do you mind closing this pull request? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2193] Improve tasks preferrd locality b...
Posted by li-zhihui <gi...@git.apache.org>.
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1131#issuecomment-46540777
@mridulm
for example:
2 tasks(task_x, task_y), 2 executors(host1, host2)
task_x.preferredLocations = [host2, host3, host1]
task_y.preferredLocations = [host1, host3, host4]
the task_x exist in penddingTasks array of host1, host2, host3
the task_y exist in pendinngTasks arrya of host1, host3, host4
if host1 pick up task_x & task_x is the last task of host2
then host2 can't get any host_preferred_task(and it must wait 3 seconds to get task_y)
if host1 pick up task_y and leave task_x to host2
then both of host1 and host2 can get preferred task
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---