You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Giridharan Kesavan (JIRA)" <ji...@apache.org> on 2010/10/20 22:54:26 UTC
[jira] Created: (HADOOP-7003) Fix hadoop patch testing using
jira_cli tool
Fix hadoop patch testing using jira_cli tool
--------------------------------------------
Key: HADOOP-7003
URL: https://issues.apache.org/jira/browse/HADOOP-7003
Project: Hadoop Common
Issue Type: New Feature
Components: build
Reporter: Giridharan Kesavan
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
jira_cli tool
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923798#action_12923798 ]
Hudson commented on HADOOP-7003:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #413 (See [https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/413/])
HADOOP-7003: New script for managing queue of pre-commit patches that need testing. Contributed by nigel.
> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
> Key: HADOOP-7003
> URL: https://issues.apache.org/jira/browse/HADOOP-7003
> Project: Hadoop Common
> Issue Type: New Feature
> Components: build
> Reporter: Giridharan Kesavan
> Assignee: Nigel Daley
> Fix For: hudson
>
> Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-7003) Fix hadoop patch testing using
jira_cli tool
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley resolved HADOOP-7003.
---------------------------------
Resolution: Fixed
Fix Version/s: hudson
Assignee: Nigel Daley
I just committed this.
> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
> Key: HADOOP-7003
> URL: https://issues.apache.org/jira/browse/HADOOP-7003
> Project: Hadoop Common
> Issue Type: New Feature
> Components: build
> Reporter: Giridharan Kesavan
> Assignee: Nigel Daley
> Fix For: hudson
>
> Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7003) Fix hadoop patch testing using
jira_cli tool
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-7003:
--------------------------------
Attachment: hudsonPatchQueueAdmin.sh
Attaching a replacement for nightly/hudsonPatchQueueAdmin.sh. This is not yet a patch file. Want to make this easier to review. The script is written for linux (tweaks are needed for MacOS) and requires xpath cmd line tool to be installed (sudo apt-get install libxml-xpath-perl).
> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
> Key: HADOOP-7003
> URL: https://issues.apache.org/jira/browse/HADOOP-7003
> Project: Hadoop Common
> Issue Type: New Feature
> Components: build
> Reporter: Giridharan Kesavan
> Attachments: hudsonPatchQueueAdmin.sh
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
jira_cli tool
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926127#action_12926127 ]
Hudson commented on HADOOP-7003:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
> Key: HADOOP-7003
> URL: https://issues.apache.org/jira/browse/HADOOP-7003
> Project: Hadoop Common
> Issue Type: New Feature
> Components: build
> Reporter: Giridharan Kesavan
> Assignee: Nigel Daley
> Fix For: hudson
>
> Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
jira_cli tool
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923345#action_12923345 ]
Nigel Daley commented on HADOOP-7003:
-------------------------------------
The current patch testing process has a number of flaws:
# reliance on email to determine when a patch is available
# a job per *slave* that you want to run the testing on
# because of 2, poor sharing of resources across many projects
# an admin job that tracks the queue and running jobs but can get confused if jobs fail or hudson is rebooted
Hudson has come a long way since I first set it up 4 years ago to run our patch testing. A recent Hudson feature allows the *same* job to be executed at the *same* time. A more established feature is the ability to pass parameters between jobs. These feature now enable a simpler architecture for patch testing.
Here are the new Hudson jobs needed:
*PreCommit-Admin*
This job replaces the current patch testing admin job and runs every 5 to 10 minutes. It runs a new version of the hudsonPatchQueueAdmin.sh script (attached). The new script roughly follows these steps:
# grabs the XML output of a Jira filter that contains all the issues that should be tested by the system. For instance, this filter contains all the HADOOP, HDFS, and MAPREDUCE issues that are in Patch Available state: https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-xml/12313474/SearchRequest-12313474.xml?tempMax=100 By changing the filter (eg. adding PIG and ZOOKEEPER for instance), this hudson job will run tests for those project issues as well.
# processes the XML into a list of <project-issue>, <attachment id> pairs, where attachment id is the id for the newest attachment
# grabs the one build artifact that this job keeps: a list of pairs: <project-issue>, <attachment id>. This file is called patch_tested.txt and contains the pairs already submitted for testing.
# for each pair from the XML, see if the pair is in patch_tested.txt. If not, kick a new project build (next job below) and append the pair to the file. When kicking off the new build, pass the issue number (and optional the attachment id) to the build. The build to kick off is computed as "PreCommit-<PROJECT NAME>-Build". This same project build can be kicked off many times by this execution of the admin job -- each instance sitting in the Hudson queue until a slave is available.
# when this admin job completes, archive the latest patch_tested.txt
*PreCommit-<PROJECT>-Build*
A job with this name (eg PreCommit-MAPREDUCE-Build) must exist for each Jira project administered by PreCommit-Admin. The job must enable concurrent builds in it's configuration. This job downloads the patch, does the actual testing and puts a comment in Jira. It does not communicate back to the admin job. The job is tied to slaves that are configured with a certain label. The label is the same as the project name.
*Slaves*
Slaves are configured to run a certain project job (like PreCommit-HDFS-Build) by being labeled with the project (eg HDFS). This means that any properly configured slave can be easily added to the build pool and start taking on load. It also means that most slaves should be able to run most patch tests for most projects.
> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
> Key: HADOOP-7003
> URL: https://issues.apache.org/jira/browse/HADOOP-7003
> Project: Hadoop Common
> Issue Type: New Feature
> Components: build
> Reporter: Giridharan Kesavan
> Attachments: hudsonPatchQueueAdmin.sh
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
jira_cli tool
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923815#action_12923815 ]
Hudson commented on HADOOP-7003:
--------------------------------
Integrated in ZooKeeper-trunk #976 (See [https://hudson.apache.org/hudson/job/ZooKeeper-trunk/976/])
HADOOP-7003: New script for managing queue of pre-commit patches that need testing. Contributed by nigel.
> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
> Key: HADOOP-7003
> URL: https://issues.apache.org/jira/browse/HADOOP-7003
> Project: Hadoop Common
> Issue Type: New Feature
> Components: build
> Reporter: Giridharan Kesavan
> Assignee: Nigel Daley
> Fix For: hudson
>
> Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
jira_cli tool
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923771#action_12923771 ]
Giridharan Kesavan commented on HADOOP-7003:
--------------------------------------------
PreCommit-Admin and the PreCommit-<PROJECT>-Build jobs are ready.
> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
> Key: HADOOP-7003
> URL: https://issues.apache.org/jira/browse/HADOOP-7003
> Project: Hadoop Common
> Issue Type: New Feature
> Components: build
> Reporter: Giridharan Kesavan
> Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7003) Fix hadoop patch testing using
jira_cli tool
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-7003:
--------------------------------
Attachment: hudsonPatchQueueAdmin.sh
Here's a new version with a few tweaks after talking with Giri. I'll go ahead and commit this. Giri agreed to setup new Hudson jobs based on this. Once it's up and running, we can delete the old jobs.
> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
> Key: HADOOP-7003
> URL: https://issues.apache.org/jira/browse/HADOOP-7003
> Project: Hadoop Common
> Issue Type: New Feature
> Components: build
> Reporter: Giridharan Kesavan
> Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
jira_cli tool
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923786#action_12923786 ]
Hudson commented on HADOOP-7003:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #396 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/396/])
HADOOP-7003: New script for managing queue of pre-commit patches that need testing. Contributed by nigel.
> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
> Key: HADOOP-7003
> URL: https://issues.apache.org/jira/browse/HADOOP-7003
> Project: Hadoop Common
> Issue Type: New Feature
> Components: build
> Reporter: Giridharan Kesavan
> Assignee: Nigel Daley
> Fix For: hudson
>
> Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
jira_cli tool
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923842#action_12923842 ]
Hudson commented on HADOOP-7003:
--------------------------------
Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
HADOOP-7003: New script for managing queue of pre-commit patches that need testing. Contributed by nigel.
> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
> Key: HADOOP-7003
> URL: https://issues.apache.org/jira/browse/HADOOP-7003
> Project: Hadoop Common
> Issue Type: New Feature
> Components: build
> Reporter: Giridharan Kesavan
> Assignee: Nigel Daley
> Fix For: hudson
>
> Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.