You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "alan wootton (JIRA)" <ji...@apache.org> on 2006/05/12 21:50:08 UTC
[jira] Created: (HADOOP-214) ClusterTestDFS fails
ClusterTestDFS fails
--------------------
Key: HADOOP-214
URL: http://issues.apache.org/jira/browse/HADOOP-214
Project: Hadoop
Type: Bug
Components: dfs
Versions: 0.2.1
Environment: local workstation (windows)
Reporter: alan wootton
Attachments: fix_clustertestdfs.patch
The dfs unit tests, from the ant target 'cluster' have been failing. (ClusterTestDFSNamespaceLogging, ClusterTestDFS). I don't know if anyone but me cares about these tests, but I do. I would like to write better tests for dns. I think we all need that.
They have been partially broken since "test.dfs.same.host.targets.allowed" went away and replication ceased for these tests.
They got really broken when NameNode stopped automatically formatting itself .
Since they seem to be ignored, I took the liberty of changing how they work.
The main thing is, you must put this into your hosts file:
127.0.0.1 localhost0
127.0.0.1 localhost1
127.0.0.1 localhost2
127.0.0.1 localhost3
127.0.0.1 localhost4
127.0.0.1 localhost5
127.0.0.1 localhost6
127.0.0.1 localhost7
127.0.0.1 localhost8
127.0.0.1 localhost9
127.0.0.1 localhost10
127.0.0.1 localhost11
127.0.0.1 localhost12
127.0.0.1 localhost13
127.0.0.1 localhost14
127.0.0.1 localhost15
This way you can start DataNodes, and TaskTrackers (up to 16 of them) with unique hostnames.
Also, I changed all the places that used to call InetAddress.getLocalHost().getHostName() to get it from a new method in Configuration (this issue is the same as http://issues.apache.org/jira/browse/HADOOP-197 ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-214) ClusterTestDFS fails
Posted by "alan wootton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-214?page=all ]
alan wootton updated HADOOP-214:
--------------------------------
Attachment: fix_clustertestdfs.patch
> ClusterTestDFS fails
> --------------------
>
> Key: HADOOP-214
> URL: http://issues.apache.org/jira/browse/HADOOP-214
> Project: Hadoop
> Type: Bug
> Components: dfs
> Versions: 0.2.1
> Environment: local workstation (windows)
> Reporter: alan wootton
> Attachments: fix_clustertestdfs.patch
>
> The dfs unit tests, from the ant target 'cluster' have been failing. (ClusterTestDFSNamespaceLogging, ClusterTestDFS). I don't know if anyone but me cares about these tests, but I do. I would like to write better tests for dns. I think we all need that.
> They have been partially broken since "test.dfs.same.host.targets.allowed" went away and replication ceased for these tests.
> They got really broken when NameNode stopped automatically formatting itself .
> Since they seem to be ignored, I took the liberty of changing how they work.
> The main thing is, you must put this into your hosts file:
> 127.0.0.1 localhost0
> 127.0.0.1 localhost1
> 127.0.0.1 localhost2
> 127.0.0.1 localhost3
> 127.0.0.1 localhost4
> 127.0.0.1 localhost5
> 127.0.0.1 localhost6
> 127.0.0.1 localhost7
> 127.0.0.1 localhost8
> 127.0.0.1 localhost9
> 127.0.0.1 localhost10
> 127.0.0.1 localhost11
> 127.0.0.1 localhost12
> 127.0.0.1 localhost13
> 127.0.0.1 localhost14
> 127.0.0.1 localhost15
> This way you can start DataNodes, and TaskTrackers (up to 16 of them) with unique hostnames.
> Also, I changed all the places that used to call InetAddress.getLocalHost().getHostName() to get it from a new method in Configuration (this issue is the same as http://issues.apache.org/jira/browse/HADOOP-197 ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-214) ClusterTestDFS fails
Posted by "alan wootton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-214?page=all ]
alan wootton updated HADOOP-214:
--------------------------------
Attachment: ClusterTestFixes.patch
patch to fix dfs cluster tests.
> ClusterTestDFS fails
> --------------------
>
> Key: HADOOP-214
> URL: http://issues.apache.org/jira/browse/HADOOP-214
> Project: Hadoop
> Type: Bug
> Components: dfs
> Versions: 0.2.1
> Environment: local workstation (windows)
> Reporter: alan wootton
> Attachments: ClusterTestFixes.patch, fix_clustertestdfs.patch
>
> The dfs unit tests, from the ant target 'cluster' have been failing. (ClusterTestDFSNamespaceLogging, ClusterTestDFS). I don't know if anyone but me cares about these tests, but I do. I would like to write better tests for dns. I think we all need that.
> They have been partially broken since "test.dfs.same.host.targets.allowed" went away and replication ceased for these tests.
> They got really broken when NameNode stopped automatically formatting itself .
> Since they seem to be ignored, I took the liberty of changing how they work.
> The main thing is, you must put this into your hosts file:
> 127.0.0.1 localhost0
> 127.0.0.1 localhost1
> 127.0.0.1 localhost2
> 127.0.0.1 localhost3
> 127.0.0.1 localhost4
> 127.0.0.1 localhost5
> 127.0.0.1 localhost6
> 127.0.0.1 localhost7
> 127.0.0.1 localhost8
> 127.0.0.1 localhost9
> 127.0.0.1 localhost10
> 127.0.0.1 localhost11
> 127.0.0.1 localhost12
> 127.0.0.1 localhost13
> 127.0.0.1 localhost14
> 127.0.0.1 localhost15
> This way you can start DataNodes, and TaskTrackers (up to 16 of them) with unique hostnames.
> Also, I changed all the places that used to call InetAddress.getLocalHost().getHostName() to get it from a new method in Configuration (this issue is the same as http://issues.apache.org/jira/browse/HADOOP-197 ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-214) ClusterTestDFS fails
Posted by "alan wootton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-214?page=comments#action_12383289 ]
alan wootton commented on HADOOP-214:
-------------------------------------
Well, THAT looks like a nice trick (hacking the name resolver in java). I'll have to try it. That would eliminate the need to change the hosts file.
Remember, these are long tests that nobody is currently running anyway. Running them is a deliberate act and you only need to change your hosts file once.
> ClusterTestDFS fails
> --------------------
>
> Key: HADOOP-214
> URL: http://issues.apache.org/jira/browse/HADOOP-214
> Project: Hadoop
> Type: Bug
> Components: dfs
> Versions: 0.2.1
> Environment: local workstation (windows)
> Reporter: alan wootton
> Attachments: fix_clustertestdfs.patch
>
> The dfs unit tests, from the ant target 'cluster' have been failing. (ClusterTestDFSNamespaceLogging, ClusterTestDFS). I don't know if anyone but me cares about these tests, but I do. I would like to write better tests for dns. I think we all need that.
> They have been partially broken since "test.dfs.same.host.targets.allowed" went away and replication ceased for these tests.
> They got really broken when NameNode stopped automatically formatting itself .
> Since they seem to be ignored, I took the liberty of changing how they work.
> The main thing is, you must put this into your hosts file:
> 127.0.0.1 localhost0
> 127.0.0.1 localhost1
> 127.0.0.1 localhost2
> 127.0.0.1 localhost3
> 127.0.0.1 localhost4
> 127.0.0.1 localhost5
> 127.0.0.1 localhost6
> 127.0.0.1 localhost7
> 127.0.0.1 localhost8
> 127.0.0.1 localhost9
> 127.0.0.1 localhost10
> 127.0.0.1 localhost11
> 127.0.0.1 localhost12
> 127.0.0.1 localhost13
> 127.0.0.1 localhost14
> 127.0.0.1 localhost15
> This way you can start DataNodes, and TaskTrackers (up to 16 of them) with unique hostnames.
> Also, I changed all the places that used to call InetAddress.getLocalHost().getHostName() to get it from a new method in Configuration (this issue is the same as http://issues.apache.org/jira/browse/HADOOP-197 ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-214) ClusterTestDFS fails
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-214?page=comments#action_12383263 ]
Doug Cutting commented on HADOOP-214:
-------------------------------------
Having to change the hosts file for a test is not great.
Perhaps we could instead just alter Java to resolve hostnames differently while testing?
http://forum.java.sun.com/thread.jspa?threadID=729300
> ClusterTestDFS fails
> --------------------
>
> Key: HADOOP-214
> URL: http://issues.apache.org/jira/browse/HADOOP-214
> Project: Hadoop
> Type: Bug
> Components: dfs
> Versions: 0.2.1
> Environment: local workstation (windows)
> Reporter: alan wootton
> Attachments: fix_clustertestdfs.patch
>
> The dfs unit tests, from the ant target 'cluster' have been failing. (ClusterTestDFSNamespaceLogging, ClusterTestDFS). I don't know if anyone but me cares about these tests, but I do. I would like to write better tests for dns. I think we all need that.
> They have been partially broken since "test.dfs.same.host.targets.allowed" went away and replication ceased for these tests.
> They got really broken when NameNode stopped automatically formatting itself .
> Since they seem to be ignored, I took the liberty of changing how they work.
> The main thing is, you must put this into your hosts file:
> 127.0.0.1 localhost0
> 127.0.0.1 localhost1
> 127.0.0.1 localhost2
> 127.0.0.1 localhost3
> 127.0.0.1 localhost4
> 127.0.0.1 localhost5
> 127.0.0.1 localhost6
> 127.0.0.1 localhost7
> 127.0.0.1 localhost8
> 127.0.0.1 localhost9
> 127.0.0.1 localhost10
> 127.0.0.1 localhost11
> 127.0.0.1 localhost12
> 127.0.0.1 localhost13
> 127.0.0.1 localhost14
> 127.0.0.1 localhost15
> This way you can start DataNodes, and TaskTrackers (up to 16 of them) with unique hostnames.
> Also, I changed all the places that used to call InetAddress.getLocalHost().getHostName() to get it from a new method in Configuration (this issue is the same as http://issues.apache.org/jira/browse/HADOOP-197 ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-214) ClusterTestDFS fails
Posted by "alan wootton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-214?page=comments#action_12413015 ]
alan wootton commented on HADOOP-214:
-------------------------------------
Here's the latest on this issue.
I have a new patch that fixes the two main DFS tests. I think it would be crazy for us to move onward without having good DFS tests.
There is a 'name server' hack in the patch now. The requires the system property "sun.net.spi.nameservice.provider" to be set to "dns,mine".
It is not good enough to just call:
System.setProperty("sun.net.spi.nameservice.provider.1","dns,mine");
in the code unless you can do it BEFORE anyone else initializes InetAddress.java. JUnit is bad this way so the property must be set in the ant script that calls JUnit. This is in the latest patch.
If you want to call the unit test from eclipse then you will need to pass
-Dsun.net.spi.nameservice.provider.1=dns,mine
to the jvm. Or, you might define "127.0.0.1 localhost0" through "127.0.0.1 localhost15" in your hosts file and get the same result.
Speaking of build.xml, The definition of the 'test-cluster' target tries to run ANYTHING in the test directory as a unit test - this generates lots of failure messages (e.g. RandomDatum is not a TestCase so it fails). I changed it so that the test-cluster target only runs files of the form "ClusterTest*.java". ToCpp.java, FromCpp.java, and DistributedFSCheck.java are NOT being run. To include them we should either make a new ant target (preferred), or change their name to start with ClusterTest. Or change the ant script.
I changed several constants in the ClusterTest*.java tests to make them run faster. I divided all the timeouts by 10. I removed the 'already existing' file creation test from ClusterTestDFSNamespaceLogging because it fails by timing out and that takes 3 minutes. When we can override LEASE_PERIOD then we can put that test back in. The whole cluster-test target now runs in 4 min.
ClusterTestDFSNamespaceLogging and ClusterTestDFS have VERY similar DFS simulations. As does MiniDfsCluster. I think someone needs to factor-out the dfs sim from all three of these tests.
> ClusterTestDFS fails
> --------------------
>
> Key: HADOOP-214
> URL: http://issues.apache.org/jira/browse/HADOOP-214
> Project: Hadoop
> Type: Bug
> Components: dfs
> Versions: 0.2.1
> Environment: local workstation (windows)
> Reporter: alan wootton
> Attachments: fix_clustertestdfs.patch
>
> The dfs unit tests, from the ant target 'cluster' have been failing. (ClusterTestDFSNamespaceLogging, ClusterTestDFS). I don't know if anyone but me cares about these tests, but I do. I would like to write better tests for dns. I think we all need that.
> They have been partially broken since "test.dfs.same.host.targets.allowed" went away and replication ceased for these tests.
> They got really broken when NameNode stopped automatically formatting itself .
> Since they seem to be ignored, I took the liberty of changing how they work.
> The main thing is, you must put this into your hosts file:
> 127.0.0.1 localhost0
> 127.0.0.1 localhost1
> 127.0.0.1 localhost2
> 127.0.0.1 localhost3
> 127.0.0.1 localhost4
> 127.0.0.1 localhost5
> 127.0.0.1 localhost6
> 127.0.0.1 localhost7
> 127.0.0.1 localhost8
> 127.0.0.1 localhost9
> 127.0.0.1 localhost10
> 127.0.0.1 localhost11
> 127.0.0.1 localhost12
> 127.0.0.1 localhost13
> 127.0.0.1 localhost14
> 127.0.0.1 localhost15
> This way you can start DataNodes, and TaskTrackers (up to 16 of them) with unique hostnames.
> Also, I changed all the places that used to call InetAddress.getLocalHost().getHostName() to get it from a new method in Configuration (this issue is the same as http://issues.apache.org/jira/browse/HADOOP-197 ).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira