You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Mukund Madhugiri (JIRA)" <ji...@apache.org> on 2008/02/25 20:22:52 UTC
[jira] Created: (HADOOP-2888) Enhancements to gridmix scripts
Enhancements to gridmix scripts
-------------------------------
Key: HADOOP-2888
URL: https://issues.apache.org/jira/browse/HADOOP-2888
Project: Hadoop Core
Issue Type: Improvement
Components: test
Affects Versions: 0.17.0
Reporter: Mukund Madhugiri
Assignee: Mukund Madhugiri
Fix For: 0.17.0
Attachments: gridmix_patch
I would like to propose enhancements to the gridmix scripts to make it:
1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
Here are the details:
Ability to override these parameters in gridmix-env
* HADOOP_HOME
* GRID_MIX_HOME
* EXAMPLE_JAR
* APP_JAR
* STREAM_JAR
* GRID_MIX_DATA
* GRID_MIX_PROG
Ability to override these parameters in generateData.sh
* COMPRESSED_DATA_BYTES
* UNCOMPRESSED_DATA_BYTES
* INDIRECT_DATA_BYTES
Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
* submissionScripts/monsterQueriesToSameCluster
* submissionScripts/maxentToSameCluster
* submissionScripts/textSortToSameCluster
* submissionScripts/webdataScanToSameCluster
* submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2888:
-------------------------------------
Status: Patch Available (was: Open)
Missed the quotes. Attaching new patch.
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch, gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572866#action_12572866 ]
Hadoop QA commented on HADOOP-2888:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12376542/gridmix_patch
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 24 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1843/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1843/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1843/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1843/console
This message is automatically generated.
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2888:
-------------------------------------
Attachment: gridmix_patch
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2888:
-------------------------------------
Status: Patch Available (was: Open)
Changed the patch based on comments from Chris.
Changes are in:
* gridmix-env
* generateData.sh
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12578288#action_12578288 ]
Hudson commented on HADOOP-2888:
--------------------------------
Integrated in Hadoop-trunk #427 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/427/])
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch, gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2888:
-------------------------------------
Status: Open (was: Patch Available)
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch, gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2888:
-------------------------------------
Attachment: gridmix_patch
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch, gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2888:
-------------------------------------
Status: Open (was: Patch Available)
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575549#action_12575549 ]
Chris Douglas commented on HADOOP-2888:
---------------------------------------
While this is being updated, it might make sense to change the defaults away from 0.15.2-dev builds of the test, streaming, and example jars, probably to hadoop-*-test.jar, hadoop-*-examples.jar, etc. as in:
{noformat}
# set var only if it has not already been set externally
if [ -z "${EXAMPLE_JAR}" ] ; then
export EXAMPLE_JAR="${HADOOP_HOME}/hadoop-*-examples.jar"
fi
{noformat}
Also, since the USE_REAL_DATASET change, the comments describing the generated data no longer match the adjacent var; these might be more readable if each were moved within the "real" block.
+1 overall
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2888:
-------------------------------------
Status: Patch Available (was: Open)
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2888:
-------------------------------------
Status: Open (was: Patch Available)
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2888:
-------------------------------------
Status: Patch Available (was: Open)
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12577115#action_12577115 ]
Chris Douglas commented on HADOOP-2888:
---------------------------------------
In the section:
{noformat}
+if [ -z "${EXAMPLE_JAR}" ] ; then
+ export EXAMPLE_JAR=${HADOOP_HOME}/hadoop-*-examples.jar
+fi
{noformat}
The exported string should be in quotes so bash will interpret it.
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12577158#action_12577158 ]
Chris Douglas commented on HADOOP-2888:
---------------------------------------
+1
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch, gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2888:
-------------------------------------
Attachment: gridmix_patch
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2888:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Mukund!
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch, gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12577301#action_12577301 ]
Hadoop QA commented on HADOOP-2888:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12377543/gridmix_patch
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 24 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1932/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1932/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1932/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1932/console
This message is automatically generated.
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch, gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-2888) Enhancements to gridmix
scripts
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575549#action_12575549 ]
chris.douglas edited comment on HADOOP-2888 at 3/5/08 7:36 PM:
---------------------------------------------------------------
While this is being updated, it might make sense to change the defaults away from 0.15.2-dev builds of the test, streaming, and example jars, probably to {{hadoop-*-test.jar}}, {{hadoop-*-examples.jar}}, etc. as in:
{noformat}
# set var only if it has not already been set externally
if [ -z "${EXAMPLE_JAR}" ] ; then
export EXAMPLE_JAR="${HADOOP_HOME}/hadoop-*-examples.jar"
fi
{noformat}
Also, since the USE_REAL_DATASET change, the comments describing the generated data no longer match the adjacent var; these might be more readable if each were moved within the "real" block.
+1 overall
was (Author: chris.douglas):
While this is being updated, it might make sense to change the defaults away from 0.15.2-dev builds of the test, streaming, and example jars, probably to hadoop-*-test.jar, hadoop-*-examples.jar, etc. as in:
{noformat}
# set var only if it has not already been set externally
if [ -z "${EXAMPLE_JAR}" ] ; then
export EXAMPLE_JAR="${HADOOP_HOME}/hadoop-*-examples.jar"
fi
{noformat}
Also, since the USE_REAL_DATASET change, the comments describing the generated data no longer match the adjacent var; these might be more readable if each were moved within the "real" block.
+1 overall
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2888) Enhancements to gridmix scripts
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2888:
-------------------------------------
Attachment: gridmix_patch
> Enhancements to gridmix scripts
> -------------------------------
>
> Key: HADOOP-2888
> URL: https://issues.apache.org/jira/browse/HADOOP-2888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.17.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.17.0
>
> Attachments: gridmix_patch, gridmix_patch
>
>
> I would like to propose enhancements to the gridmix scripts to make it:
> 1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
> 2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)
> Here are the details:
> Ability to override these parameters in gridmix-env
> * HADOOP_HOME
> * GRID_MIX_HOME
> * EXAMPLE_JAR
> * APP_JAR
> * STREAM_JAR
> * GRID_MIX_DATA
> * GRID_MIX_PROG
> Ability to override these parameters in generateData.sh
> * COMPRESSED_DATA_BYTES
> * UNCOMPRESSED_DATA_BYTES
> * INDIRECT_DATA_BYTES
> Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:
> * submissionScripts/monsterQueriesToSameCluster
> * submissionScripts/maxentToSameCluster
> * submissionScripts/textSortToSameCluster
> * submissionScripts/webdataScanToSameCluster
> * submissionScripts/webdataSortToSameCluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.