You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Sharad Agarwal (JIRA)" <ji...@apache.org> on 2009/01/28 09:22:59 UTC
[jira] Created: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Separate the core, hdfs and mapred junit tests
----------------------------------------------
Key: HADOOP-5135
URL: https://issues.apache.org/jira/browse/HADOOP-5135
Project: Hadoop Core
Issue Type: Improvement
Components: build
Reporter: Sharad Agarwal
To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal reassigned HADOOP-5135:
--------------------------------------
Assignee: Sharad Agarwal
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 5135_v1.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707322#action_12707322 ]
Sharad Agarwal commented on HADOOP-5135:
----------------------------------------
All earlier targets continue to work the same way. Newly added targets for each components are:
run-test-core
run-test-hdfs and run-test-hdfs-with-mr
run-test-mapred
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 5135_v1.tar, 5135_v2.tar, 5135_v3.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707496#action_12707496 ]
Hudson commented on HADOOP-5135:
--------------------------------
Integrated in Hadoop-trunk #830 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/830/])
. Trying to delete the empty src/test/org directory
. Splits the tests into different directories based on the package. Four new test targets have been defined - run-test-core, run-test-mapred, run-test-hdfs and run-test-hdfs-with-mr. Contributed by Sharad Agarwal.
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.21.0
>
> Attachments: 5135_v1.tar, 5135_v2.tar, 5135_v3.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-5135:
-----------------------------------
Attachment: 5135_v1.tar
untar the attached file in hadoop home dir. Run the 5135_script.sh. This script creates the new test source folders and does the svn mv of the respective files. It also applies the patch to build.xml in order to run the core, hdfs, mapred tests independently.
Things remaining:
There are couple of test classes which cover both mapred and hdfs functionality like TestCLI, AllTestDriver. These I think should be split.
The target for building test jar needs to be fixed. Separate test jars to be built.
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Attachments: 5135_v1.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707309#action_12707309 ]
Sharad Agarwal commented on HADOOP-5135:
----------------------------------------
All tests passed on my machine.
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 5135_v1.tar, 5135_v2.tar, 5135_v3.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "V.Karthikeyan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672917#action_12672917 ]
V.Karthikeyan commented on HADOOP-5135:
---------------------------------------
The patch is breaking the trunk and creates build.xml.reg file. Can you please update the patch?
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Attachments: 5135_v1.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667962#action_12667962 ]
Sharad Agarwal commented on HADOOP-5135:
----------------------------------------
In the current folder structure, the separation could look like as follows:
src/test/core - testcases only related to core apis
src/test/hdfs - testcases dependent on hdfs apis. May depend on core apis/tests as well.
src/test/mapred - testcases dependent on mapred apis. May depend on hdfs apis/tests as well.
src/test/hdfs-integration (better name ?)- hdfs tests which depend on mapred for execution. for example benchmark tests NNBench.
There could be separate targets for all the above say - test-core, test-hdfs, test-mapred, test-hdfs-integration in the current build.xml
When the project split, the hdfs subproject need not install mapred. It only needs mapred if wants to run test-hdfs-integration. mapred would have to install hdfs to run its tests as most of the tests run with hdfs.
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715375#action_12715375 ]
Sharad Agarwal commented on HADOOP-5135:
----------------------------------------
@Tsz I agree that tests need to be refactored as you have suggested. Actually the split has happened based on the code dependency and not on the package. The commit message perhaps is misleading. This issue mostly dealt with moving the tests into different categories. Refactoring individual tests to belong to right project can be subsequent issues.
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.21.0
>
> Attachments: 5135_v1.tar, 5135_v2.tar, 5135_v3.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715664#action_12715664 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5135:
------------------------------------------------
Sharad, thanks for your comment. I will file issues for the refactoring.
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.21.0
>
> Attachments: 5135_v1.tar, 5135_v2.tar, 5135_v3.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-5135:
-----------------------------------
Attachment: 5135_v2.tar
Updated to trunk.
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 5135_v1.tar, 5135_v2.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das resolved HADOOP-5135.
---------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
I just committed this. Thanks, Sharad!
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.21.0
>
> Attachments: 5135_v1.tar, 5135_v2.tar, 5135_v3.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707289#action_12707289 ]
Giridharan Kesavan commented on HADOOP-5135:
--------------------------------------------
build.xml changes in the patch looks good to me. Except for the test-core target.
test-patch.sh uses test-core and test-contrib target as the target for executing the junit test for the hadoop core and contrib resp.
Please change the ant target names in this patch accordingly.
Tnx!
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 5135_v1.tar, 5135_v2.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715321#action_12715321 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5135:
------------------------------------------------
The following tests are under the org.apache.hadoop.fs package but were moved to hdfs sub-directory:
{noformat}
./org/apache/hadoop/fs/ftp/TestFTPFileSystem.java
./org/apache/hadoop/fs/loadGenerator/TestLoadGenerator.java
./org/apache/hadoop/fs/permission/TestStickyBit.java
./org/apache/hadoop/fs/TestGlobPaths.java
./org/apache/hadoop/fs/TestUrlStreamHandler.java
{noformat}
- Some of them are not related to hdfs, e.g. TestFTPFileSystem. These files should be moved out from hdfs and should not use hdfs codes.
- Some of them are testing hdfs features, e.g. TestStickyBit. They should be defined under org.apache.hadoop.hdfs package.
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.21.0
>
> Attachments: 5135_v1.tar, 5135_v2.tar, 5135_v3.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5135) Separate the core, hdfs and mapred
junit tests
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-5135:
-----------------------------------
Attachment: 5135_v3.tar
Updated the patch incorporating Giri's comments.
Changed the target name to "run-test-core". "test-core" continues to run all non-contrib tests.
> Separate the core, hdfs and mapred junit tests
> ----------------------------------------------
>
> Key: HADOOP-5135
> URL: https://issues.apache.org/jira/browse/HADOOP-5135
> Project: Hadoop Core
> Issue Type: Improvement
> Components: build
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 5135_v1.tar, 5135_v2.tar, 5135_v3.tar
>
>
> To support splitting of projects, the tests should be separated into different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.