You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Yufei Gu (JIRA)" <ji...@apache.org> on 2017/05/01 18:28:04 UTC
[jira] [Comment Edited] (YARN-6522) Make SLS JSON input file format
simple and scalable
[ https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989320#comment-15989320 ]
Yufei Gu edited comment on YARN-6522 at 5/1/17 6:27 PM:
--------------------------------------------------------
Uploaded patch v1. It largely simplifies the job configuration and be compatible with the existing format.
This is a job configuration for FS minshare preemption test. Here is what it looks like before patch:
{code}
{
"am.type" : "mapreduce",
"job.start.ms" : 0,
"job.end.ms" : 60000,
"job.queue.name" : "default",
"job.id" : "job_1",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6664,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
} ]
}
{
"am.type" : "mapreduce",
"job.start.ms" : 10000,
"job.end.ms" : 60000,
"job.queue.name" : "min-wait",
"job.id" : "job_2",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 10000,
"container.end.ms" : 40000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 10000,
"container.end.ms" : 40000,
"container.priority" : 20,
"container.type" : "map"
} ]
}
{code}
and after patch:
{code}
{ "num_nodes":3}
{ "job.start.ms" : 0, "job.queue.name": "default", "job.tasks": [ {"count":29, "duration.ms": 50000}]}
{ "job.start.ms" : 10000, "job.queue.name": "min-wait", "job.tasks":[{"count":2, "duration.ms": 30000}]}
{code}
was (Author: yufeigu):
Uploaded patch v1. It largely simplifies the job configuration and be compatible with the existing format.
This is a job configuration for FS minshare preemption test. Here is what it looks like before patch:
{code}
{
"am.type" : "mapreduce",
"job.start.ms" : 0,
"job.end.ms" : 60000,
"job.queue.name" : "default",
"job.id" : "job_1",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6664,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
} ]
}
{
"am.type" : "mapreduce",
"job.start.ms" : 0,
"job.end.ms" : 60000,
"job.queue.name" : "default",
"job.id" : "job_11",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6664,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 60000,
"container.priority" : 20,
"container.type" : "map"
} ]
}
{
"am.type" : "mapreduce",
"job.start.ms" : 10000,
"job.end.ms" : 60000,
"job.queue.name" : "min-wait",
"job.id" : "job_2",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 10000,
"container.end.ms" : 40000,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 10000,
"container.end.ms" : 40000,
"container.priority" : 20,
"container.type" : "map"
} ]
}
{code}
and after patch:
{code}
{ "num_nodes":3}
{ "job.start.ms" : 0, "job.queue.name": "default", "job.tasks": [ {"count":29, "duration.ms": 50000}]}
{ "job.start.ms" : 10000, "job.queue.name": "min-wait", "job.tasks":[{"count":2, "duration.ms": 30000}]}
{code}
> Make SLS JSON input file format simple and scalable
> ---------------------------------------------------
>
> Key: YARN-6522
> URL: https://issues.apache.org/jira/browse/YARN-6522
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler-load-simulator
> Affects Versions: 3.0.0-alpha2
> Reporter: Yufei Gu
> Assignee: Yufei Gu
> Attachments: YARN-6522.001.patch
>
>
> SLS input format is verbose, and it doesn't scale out. We can improve it in these ways:
> # We need to configure tasks one by one if there are more than one task in a job, which means the job configuration usually includes lots of redundant items. To specify the number of task for task configuration will solve this issue.
> # Container host is useful for locality testing. It is obnoxious to specify container host for each task for tests unrelated to locality. We would like to make it optional.
> # For most tests, we don't care about job.id. Make it optional and generated automatically by default.
> # job.finish.ms doesn't make sense, just remove it.
> # container type and container priority should be optional as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org