You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Michel Tourn (JIRA)" <ji...@apache.org> on 2006/06/22 01:31:29 UTC
[jira] Created: (HADOOP-313) running a reduce task standalone
running a reduce task standalone
--------------------------------
Key: HADOOP-313
URL: http://issues.apache.org/jira/browse/HADOOP-313
Project: Hadoop
Type: Bug
Reporter: Michel Tourn
Attachments: sareduce.patch
This is a tool to reproduce problems and to run unit tests involving a reduce task.
You just give it a reduce directory on the command line.
Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
taskdir contains job.xml and one or more input files named: map_<dddd>.out
You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-313) A stand alone driver for individual
tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=all ]
Owen O'Malley updated HADOOP-313:
---------------------------------
Attachment: stand-alone-task.patch
Here is the updated patch that includes the IsolationRunner.
> A stand alone driver for individual tasks
> -----------------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Attachments: sareduce.patch, stand-alone-task.patch, stand-alone-task.patch
>
> This is a tool to reproduce problems and to run unit tests involving either a map or reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-313) A stand alone driver for individual
tasks
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=comments#action_12420028 ]
Doug Cutting commented on HADOOP-313:
-------------------------------------
This looks good, except I think you forgot to add IsolationRunner.java.
> A stand alone driver for individual tasks
> -----------------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Attachments: sareduce.patch, stand-alone-task.patch
>
> This is a tool to reproduce problems and to run unit tests involving either a map or reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-313) running a reduce task standalone
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=all ]
Owen O'Malley updated HADOOP-313:
---------------------------------
Attachment: stand-alone-task.patch
This patch does a lot more:
1. It moves the jobId and partition fields from ReduceTask up into Task.
2. It replaces Michel's stand-alone reduce runner, with a more general map or reduce runner.
3. It adds new task localizations into the job.xml.
For all tasks:
a. map.task.id = the task id
b. mapred.task.is.map = is this a map
c. mapred.task.partition = numeric task id
d. mapred.job.id = job id
For maps:
a. map.input.file = the file that we are reading
b. map.input.start = the offset in the file to start at
c. map.input.length = the number of bytes in the split
For reduces:
a. mapred.map.tasks = correct number of maps
These new attributes allow me to reconstruct the MapTask or ReduceTask with just the job.xml.
4. A new configuration variable keep.failed.task.files that tells the system to keep files/directories for tasks that fail. This attribute can be set on a particular JobConf.
> running a reduce task standalone
> --------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Attachments: sareduce.patch, stand-alone-task.patch
>
> This is a tool to reproduce problems and to run unit tests involving a reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-313) running a reduce task standalone
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=comments#action_12417837 ]
Owen O'Malley commented on HADOOP-313:
--------------------------------------
An option to keep failed reduces would be useful, but I've certainly seen cases before where I wanted to run a particular fragment in the debugger, even if it didn't crash.
> running a reduce task standalone
> --------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Attachments: sareduce.patch
>
> This is a tool to reproduce problems and to run unit tests involving a reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-313) running a reduce task standalone
Posted by "Michel Tourn (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=all ]
Michel Tourn updated HADOOP-313:
--------------------------------
Attachment: sareduce.patch
> running a reduce task standalone
> --------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Attachments: sareduce.patch
>
> This is a tool to reproduce problems and to run unit tests involving a reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-313) A stand alone driver for individual
tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=comments#action_12418710 ]
Owen O'Malley commented on HADOOP-313:
--------------------------------------
One more comment, the usage is:
bin/hadoop org.apache.hadoop.mapred.IsolationRunner <local-dir>/taskTracker/<taskid>/job.xml
> A stand alone driver for individual tasks
> -----------------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Attachments: sareduce.patch, stand-alone-task.patch
>
> This is a tool to reproduce problems and to run unit tests involving either a map or reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-313) A stand alone driver for individual
tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=all ]
Owen O'Malley updated HADOOP-313:
---------------------------------
Summary: A stand alone driver for individual tasks (was: running a reduce task standalone)
Description:
This is a tool to reproduce problems and to run unit tests involving either a map or reduce task.
You just give it a reduce directory on the command line.
Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
taskdir contains job.xml and one or more input files named: map_<dddd>.out
You should run with the same -Xmx option as the TaskTracker child JVM
was:
This is a tool to reproduce problems and to run unit tests involving a reduce task.
You just give it a reduce directory on the command line.
Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
taskdir contains job.xml and one or more input files named: map_<dddd>.out
You should run with the same -Xmx option as the TaskTracker child JVM
> A stand alone driver for individual tasks
> -----------------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Attachments: sareduce.patch, stand-alone-task.patch
>
> This is a tool to reproduce problems and to run unit tests involving either a map or reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-313) running a reduce task standalone
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=comments#action_12417708 ]
eric baldeschwieler commented on HADOOP-313:
--------------------------------------------
Why not an option to keep inputs for failed reduces?
Also recall HADOOP-91.
> running a reduce task standalone
> --------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Attachments: sareduce.patch
>
> This is a tool to reproduce problems and to run unit tests involving a reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-313) running a reduce task standalone
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=comments#action_12417531 ]
Owen O'Malley commented on HADOOP-313:
--------------------------------------
Actually, it isn't for testing at all. It is for debugging a reduce crash in place on the production machines. Michel manually caught the reduce inputs, but in general we also need to add a configuration setting that lets a job request that some reduce task's inputs be kept. Maybe something like:
mapred.keep.reduce.inputs=...
Where the legal values are "all", "none", or a list of integers of which reduce tasks to keep the inputs for.
> running a reduce task standalone
> --------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Attachments: sareduce.patch
>
> This is a tool to reproduce problems and to run unit tests involving a reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-313) A stand alone driver for individual
tasks
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=all ]
Doug Cutting resolved HADOOP-313:
---------------------------------
Fix Version: 0.5.0
Resolution: Fixed
I just committed this. Thanks, Owen.
> A stand alone driver for individual tasks
> -----------------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Fix For: 0.5.0
> Attachments: sareduce.patch, stand-alone-task.patch, stand-alone-task.patch
>
> This is a tool to reproduce problems and to run unit tests involving either a map or reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Assigned: (HADOOP-313) running a reduce task standalone
Posted by "Michel Tourn (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=all ]
Michel Tourn reassigned HADOOP-313:
-----------------------------------
Assign To: Michel Tourn
> running a reduce task standalone
> --------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Attachments: sareduce.patch
>
> This is a tool to reproduce problems and to run unit tests involving a reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-313) running a reduce task standalone
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-313?page=comments#action_12417356 ]
Doug Cutting commented on HADOOP-313:
-------------------------------------
Shouldn't this go into the "test" tree rather than the core?
> running a reduce task standalone
> --------------------------------
>
> Key: HADOOP-313
> URL: http://issues.apache.org/jira/browse/HADOOP-313
> Project: Hadoop
> Type: Bug
> Reporter: Michel Tourn
> Assignee: Michel Tourn
> Attachments: sareduce.patch
>
> This is a tool to reproduce problems and to run unit tests involving a reduce task.
> You just give it a reduce directory on the command line.
> Usage: java org.apache.hadoop.mapred.StandaloneReduceTask <taskdir> [<limitmaps>]
> taskdir name encodes: task_<jobid>_r_<partition>_<attempt>
> taskdir contains job.xml and one or more input files named: map_<dddd>.out
> You should run with the same -Xmx option as the TaskTracker child JVM
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira