You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2008/11/10 20:55:46 UTC
[jira] Created: (HADOOP-4627) gridmix version 2
gridmix version 2
-----------------
Key: HADOOP-4627
URL: https://issues.apache.org/jira/browse/HADOOP-4627
Project: Hadoop Core
Issue Type: New Feature
Reporter: Runping Qi
The new gridmix differs from the original gridmix in the following ways:
1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
2. Use JobControl to submit gridmix load, instead of shell script.
3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4627) gridmix version 2
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Runping Qi updated HADOOP-4627:
-------------------------------
Description:
The new gridmix differs from the original gridmix in the following ways:
1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
2. Use JobControl to submit gridmix load, instead of shell script.
3. Include Pig jobs.
was:
The new gridmix differs from the original gridmix in the following ways:
1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
2. Use JobControl to submit gridmix load, instead of shell script.
3. Include Pig jobs.
Fix Version/s: 0.20.0
0.19.1
This patch should be applied to trunk and 0.19 branch.
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Fix For: 0.19.1, 0.20.0
>
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4627) gridmix version 2
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647482#action_12647482 ]
Runping Qi commented on HADOOP-4627:
------------------------------------
The mail goal for gridmix is to allow a developer to create a reasonable mix jobs so that they can use that to establish a baseline for their cluster running a specific of Hadoop, and then test the potential impact of any changes made to hadoop and/or the clutser.
It can also be used to uncover potential performance issues of a specific Hadoop version and/or a specific cluster configuration.
It is believed to be generic enough such that a developer can easily tweak the scripts to generate specific job mixes that are most appropriate for their
specific need, but it cannot be tuned to be the best mix that fits all needs.
Gridmix version two is along the same theme, but with a more efficient job submission mechanism and provides more flexibility.
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4627) gridmix version 2
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647489#action_12647489 ]
Matei Zaharia commented on HADOOP-4627:
---------------------------------------
I don't think some of the issues I brought up are just about job mix. For example, having all small jobs use the same 3 files means that data locality for these jobs will be poor and any results from them won't be meaningful for a Hadoop cluster running small jobs on many different small files. Similarly, with smaller data sizes, the bottleneck in gridmix can be job submission, so the benchmark won't tell much about performance in a real cluster. Is the right thing to do to open separate JIRAs for these issues?
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4627) gridmix version 2
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647503#action_12647503 ]
Runping Qi commented on HADOOP-4627:
------------------------------------
Yes, that is the idea.
But we do not want to have too many different versions (or kinds).
We want each version to be flexible and generic so that a developer can get the right load mix for him by just changing the parameters, not the code.
Since we just started out for version two, it is a good time to make changes on the code part.
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4627) gridmix version 2
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Runping Qi updated HADOOP-4627:
-------------------------------
Status: Patch Available (was: Open)
The attached patch includes the new gridmix version
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4627) gridmix version 2
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647493#action_12647493 ]
Matei Zaharia commented on HADOOP-4627:
---------------------------------------
Thanks for explaining that Runping; backwards compatibility is indeed important. I think maybe the solution will be to create a separate benchmark for a more dynamic mix.
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4627) gridmix version 2
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4627:
----------------------------------
Resolution: Duplicate
Status: Resolved (was: Patch Available)
This is an updated patch for HADOOP-3770; it doesn't require a new issue.
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Fix For: 0.19.1, 0.20.0
>
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4627) gridmix version 2
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-4627:
--------------------------------
Fix Version/s: (was: 0.19.1)
(was: 0.20.0)
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4627) gridmix version 2
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647383#action_12647383 ]
Matei Zaharia commented on HADOOP-4627:
---------------------------------------
I'm not sure whether this is the right place for this, but I have run into several problems with gridmix that are not addressed by this JIRA:
1) Although the gridmix contains small, medium and large jobs, the small jobs all run first, and then the medium jobs, then the large jobs. In other words, it's not really testing a mix that would occur from users submitting jobs of various sizes at various times. I'm not sure whether this is intended but I think it's not representative of how most multi-user Hadoop clusters are used. The solution would be to submit the same numbers of jobs, but choose a random permutation of sizes.
2) The small jobs in gridmix all use the same input files - parts 0, 1 and 2 of the data set. This leads to a hotspot and to poor data locality. (The medium jobs also use the same input files, but there are 30 of those instead of 3 so maybe they have a higher chance of being spread-out).
3) For at least some job sizes, gridmix does not submit jobs fast enough to keep the cluster fully utilized. The sleep 10 in sleep_if_too_busy is part of the problem.
Do you want to fix these as part of this JIRA, or should I open another one? I have a fix for issue 2 that runs Ruby to select random input files for each job. In the long term it might be worthwhile to write other parts of the gridmix in a scripting language like Ruby or Python to make it easier to program.
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4627) gridmix version 2
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646408#action_12646408 ]
Hadoop QA commented on HADOOP-4627:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12393656/H-4627.txt
against trunk revision 712729.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3570/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3570/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3570/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3570/console
This message is automatically generated.
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Fix For: 0.19.1, 0.20.0
>
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4627) gridmix version 2
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647491#action_12647491 ]
Runping Qi commented on HADOOP-4627:
------------------------------------
Actually, in real life, it is very often for a lot of jobs to use some common hot data.
Again, I don't think there exists such thing as the best benchmark load.
Actually, the gridmix as a benchmark load is very different than the real load of a real cluster.
As I said, the purpose for us to use gridmix is to track the impacts of various changes over time and to uncover performance related problems.
So far, it has served us pretty well.
We want the load nature of the gridmix remains to be the same so that we can compare the running results over time.
If you change how a small/medium job gets its input, then it changes the gridmix load nature. It makes it is impossible to compare new runs with baselines.
Having said that, we know that the gridmix version one is not flexible enough and the job submission not efficient.
Then we create this version two. If you'd like to improve it, version two should be a good start.
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4627) gridmix version 2
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Runping Qi updated HADOOP-4627:
-------------------------------
Attachment: H-4627.txt
> gridmix version 2
> -----------------
>
> Key: HADOOP-4627
> URL: https://issues.apache.org/jira/browse/HADOOP-4627
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Runping Qi
> Attachments: H-4627.txt
>
>
> The new gridmix differs from the original gridmix in the following ways:
> 1. Use an xml config file to specify the types and sizes mix of a mix load. This provides better granularity control.
> 2. Use JobControl to submit gridmix load, instead of shell script.
> 3. Include Pig jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.