You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Dick King (JIRA)" <ji...@apache.org> on 2009/12/15 01:48:18 UTC
[jira] Created: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
We need a job trace manipulator to build gridmix runs.
------------------------------------------------------
Key: MAPREDUCE-1295
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Dick King
Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1295) We need a job trace manipulator
to build gridmix runs.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792412#action_12792412 ]
Hadoop QA commented on MAPREDUCE-1295:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428351/mapreduce-1295--2009-12-17.patch
against trunk revision 892117.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 2335 javac compiler warnings (more than the trunk's current 2331 warnings).
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/216/console
This message is automatically generated.
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Status: Patch Available (was: Open)
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1295--2009-12-22.patch, mapreduce-1295--2009-12-23.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1295) We need a job trace manipulator
to build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791057#action_12791057 ]
Dick King commented on MAPREDUCE-1295:
--------------------------------------
I misnamed the patch.
I will need to fix it anyway; I noticed that it doesn't print out the needed skew buffer size. I will upload the new patch, with the correct name [and the then-correct date] after I see the hudson report and act on it if necessary.
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Attachment: mapreduce-1295--2009-12-22.patch
Fprgot that {{RuntimeException}} implements {{Serializable}} and therefore any subclasses I define should have a {{serialVersionUID}}
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1295--2009-12-22.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1295) We need a job trace manipulator
to build gridmix runs.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793893#action_12793893 ]
Hadoop QA commented on MAPREDUCE-1295:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428754/mapreduce-1295--2009-12-22.patch
against trunk revision 893055.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/335/console
This message is automatically generated.
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1295--2009-12-22.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Status: Patch Available (was: Open)
This is the folder.
The command line looks like this:
{noformat}
Folder [ -output-duration duration ] \
[ -input-cycle duration ] \
[ -concentration ratio ] \
[ -seed seed ] \
[ -temp temp-directory ] \
[ -debug ] \
[ -skew-buffer-length n ] \
[ -allow-missorting ] \
input-path output-path
{noformat}
All paths and directories are general {{Path}}s.
{{-output-duration}} is the length of the range of the submit times of the output trace.
{{-input-cycle}} is the length of the input job submit time cycle. For example, if {{-input-cycle}} is 2 hours, then 3PM and 1PM is treated alike.
{{-concentration}} is a double, a ratio of the density [number of jobs starting per hour] in the output over the input. This can be less than or greater than 1.0.
{{-seed}} is a random number generator seed, used to create repeatable runs if desired. If no {{-seed}} is provided, we state what the {{-seed}} should be if you want to repeat this run.
{{-temp}} is a temp directory, which must be able to hold about as much data as the input contains, compressed. Trace data compresses at about 17:1. This defaults to the directory of the output. The temporary files are erased whether the job succeeds or fails, unless {{-debug}} is coded.
{{-debug}} induces the tool to produce a lot of debugging output, and causes the itnermediate files to be retained after a run.
{{-skew-buffer-length}} describes the length of the skew buffer, which defaults to 0. The input to the folder should be sorted, and the job tracker log output is approximately sorted. However, there are occasional small glitches in the job tracker logs, jobs that come out a few places earlier than they should to be ordered by submit time. Code a {{-skew-buffer-length}} of {{i > 0}} to allow as many as {{i}} jobs to arrive earlier than they're supposed to and be buffered until they can be released.
{{-allow-missorting}} instructs the folder to just drop a job that arrives later than it should, if there's not enough room in the skew buffer for it. If {{-allow-missorting}} is not coded, we abend the run instead.
If the run is successful, either because {{-allow-missorting}} is coded or {{-skew-buffer-length}} is big enough, the folder tells the user what the smallest {{-skew-buffer-length}} could have been for the run to succeed without omitting any jobs.
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1295) We need a job trace manipulator
to build gridmix runs.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794309#action_12794309 ]
Hadoop QA commented on MAPREDUCE-1295:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428888/mapreduce-1295--2009-12-23.patch
against trunk revision 893469.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/242/console
This message is automatically generated.
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1295--2009-12-22.patch, mapreduce-1295--2009-12-23.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Status: Open (was: Patch Available)
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Status: Patch Available (was: Open)
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1295--2009-12-22.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1295:
-------------------------------------
Status: Open (was: Patch Available)
The patch doesn't apply; could you update it?
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1295:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
+1
I committed this. Thanks, Dick!
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Fix For: 0.22.0
>
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1295--2009-12-22.patch, mapreduce-1295--2009-12-23.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1295:
-------------------------------------
Status: Open (was: Patch Available)
The test failures are known (MAPREDUCE-1311, MAPREDUCE-1312).
Only a few minor nits:
* The two types of DeskewedJobTraceReader constructors could be combined by adding a private/protected cstr with a JobTraceReader formal
* Should the System.err messages in DJTR::nextJob be debug messages?
* The open/close idiom in {{run}} may be replaced by {{FileSystem::exists}}. Alternatively, require the user to provide a clean directory and use sequential segment numbering
* The first person is a little disorienting in the debug log messages- which could use log4j loggers- but whatever you prefer
* The {{deletees}} and similar accounting can be replace with {{FileSystem::deleteOnExit}}
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1295--2009-12-22.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Status: Patch Available (was: Open)
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Attachment: mapreduce-1295--2009-12-23.patch
This addresses the issues raised earlier today.
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1295--2009-12-22.patch, mapreduce-1295--2009-12-23.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1295) We need a job trace manipulator
to build gridmix runs.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791108#action_12791108 ]
Hadoop QA commented on MAPREDUCE-1295:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427990/mapreduce-1297--2009-12-14.patch
against trunk revision 890983.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/197/console
This message is automatically generated.
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1295) We need a job trace manipulator
to build gridmix runs.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793523#action_12793523 ]
Hadoop QA commented on MAPREDUCE-1295:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428686/mapreduce-1295--2009-12-21.patch
against trunk revision 893055.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 2331 javac compiler warnings (more than the trunk's current 2330 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/330/console
This message is automatically generated.
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Status: Patch Available (was: Open)
This patch fixes an applicability issue.
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1295) We need a job trace manipulator
to build gridmix runs.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798372#action_12798372 ]
Hudson commented on MAPREDUCE-1295:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #200 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/200/])
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Fix For: 0.22.0
>
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1295--2009-12-22.patch, mapreduce-1295--2009-12-23.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1295) We need a job trace manipulator
to build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King reassigned MAPREDUCE-1295:
------------------------------------
Assignee: Dick King
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Attachment: mapreduce-1295--2009-12-21.patch
This is the replacement for ...-17.patch
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1295) We need a job trace manipulator
to build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794280#action_12794280 ]
Dick King commented on MAPREDUCE-1295:
--------------------------------------
This is a comment on the previous comment timestamped 23/Dec/09 02:59AM.
* I will make all the {{DeskewedJobTraceReader}} constructors take a {{JobTraceReader}} . I see no reason why {{DeskewedJobTraceReader}} constructors shouldn't be public.
* I will use {{FileSystem::exists}} .
* I didn't notice that API when I wrote this code. My bad.
* I don't want to require an empty directory because we use the output directory as the temp directory if none is supplied, and requiring an empty directory would make it impossible to do two runs into the same output directory.
* I'll change the wording on some of the debug messages. I'll consider going to LOG4J .
* I stand by my decision to use {{deletees}} . I don't like the {{deleteOnExit}} idiom because it makes it hard to use a tool as a callable component.
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1295--2009-12-22.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated MAPREDUCE-1295:
----------------------------------
Component/s: tools/rumen
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: tools/rumen
> Reporter: Dick King
> Assignee: Dick King
> Fix For: 0.21.0
>
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1295--2009-12-21.patch, mapreduce-1295--2009-12-22.patch, mapreduce-1295--2009-12-23.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Attachment: mapreduce-1295--2009-12-17.patch
This patch applies on a direct download of Trunk, and replaces the previous patch
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Attachment: mapreduce-1297--2009-12-14.patch
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1295) We need a job trace manipulator to
build gridmix runs.
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1295:
---------------------------------
Status: Open (was: Patch Available)
> We need a job trace manipulator to build gridmix runs.
> ------------------------------------------------------
>
> Key: MAPREDUCE-1295
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1295
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Attachments: mapreduce-1295--2009-12-17.patch, mapreduce-1297--2009-12-14.patch
>
>
> Rumen produces "job traces", which are JSON format files describing important aspects of all jobs that are run [successfully or not] on a hadoop map/reduce cluster. There are two packages under development that will consume these trace files and produce actions in that cluster or another cluster: gridmix3 [see jira MAPREDUCE-1124 ] and Mumak [a simulator -- see MAPREDUCE-728 ].
> It would be useful to be able to do two things with job traces, so we can run experiments using these two tools: change the duration, and change the density. I would like to provide a "folder", a tool that can wrap a long-duration execution trace to redistribute its jobs over a shorter interval, and also change the density by duplicating or culling away jobs from the folded combined job trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.