You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Ion Barcan <io...@gmail.com> on 2012/10/23 16:05:05 UTC
Conflicting mkdirs() behavior in abstract test classes from Hadoop Common
Hi,
I have a question related to 2 tests classes found in Hadoop Common
(both 1.0.4 and 1.1.0): FileSystemContractBaseTest.java and
FSMainOperationsBaseTest.java - which (as I understand) one could use
to verify that a specific FileSystem implementation does what it's
supposed to do.
I'm a bit puzzled by the method
testMkdirsFailsForSubdirectoryOfExistingFile() that is defined in both
places. To me it looks like they're contradicting one another, i.e.
one expects FileSystem.mkdirs() to fail if it encounters an existing
file, the other expects FileSystem.mkdirs() to return false in the
same situation. Which one is right? FileSystemContractBaseTest has 8
implementors as opposed to FSMainOperationsBaseTest which only has 2 -
but I still think this should be sorted out. For reference I've pasted
the source code at the end of message.
Thanks,
Ion
* FileSystemContractBaseTest.java
118 public void testMkdirsFailsForSubdirectoryOfExistingFile()
throws Exception {
119 Path testDir = path("/test/hadoop");
120 assertFalse(fs.exists(testDir));
121 assertTrue(fs.mkdirs(testDir));
122 assertTrue(fs.exists(testDir));
123
124 createFile(path("/test/hadoop/file"));
125
126 Path testSubDir = path("/test/hadoop/file/subdir");
127 try {
128 fs.mkdirs(testSubDir);
129 fail("Should throw IOException.");
130 } catch (IOException e) {
131 // expected
132 }
133 assertFalse(fs.exists(testSubDir));
134
135 Path testDeepSubDir = path("/test/hadoop/file/deep/sub/dir");
136 try {
137 fs.mkdirs(testDeepSubDir);
138 fail("Should throw IOException.");
139 } catch (IOException e) {
140 // expected
141 }
142 assertFalse(fs.exists(testDeepSubDir));
143
144 }
* FSMainOperationsBaseTest.java
198 @Test
199 public void testMkdirsFailsForSubdirectoryOfExistingFile()
throws Exception {
200 Path testDir = getTestRootPath(fSys, "test/hadoop");
201 Assert.assertFalse(exists(fSys, testDir));
202 fSys.mkdirs(testDir);
203 Assert.assertTrue(exists(fSys, testDir));
204
205 createFile(getTestRootPath(fSys, "test/hadoop/file"));
206
207 Path testSubDir = getTestRootPath(fSys, "test/hadoop/file/subdir");
208 Assert.assertFalse(fSys.mkdirs(testSubDir));
209 Assert.assertFalse(exists(fSys, testSubDir));
210
211 Path testDeepSubDir = getTestRootPath(fSys,
"test/hadoop/file/deep/sub/dir");
212 Assert.assertFalse(exists(fSys, testSubDir));
213 Assert.assertFalse(fSys.mkdirs(testDeepSubDir));
214 Assert.assertFalse(exists(fSys, testDeepSubDir));
215
216 }
Re: Conflicting mkdirs() behavior in abstract test classes from
Hadoop Common
Posted by Harsh J <ha...@cloudera.com>.
Hi Ion,
If you're looking at developing something, I recommend doing it on trunk :)
Onto your confusion, there are two FS interaction APIs: FileSystem
(older, stable) and FileContext (new, supports a lot of the new HDFS
features, but not yet popular as of today).
The FileSystem.mkdirs, as seen at
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#mkdirs(org.apache.hadoop.fs.Path),
returns a boolean if it fails. Obviously, this was not very helpful in
knowing WHY the operation failed.
The FileContext API was derived keeping in mind the problems devs
faced with the older FileSystem and with support for new features of
HDFS such as symlinks. The FileContext.mkdir API, as seen at
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir(org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean),
throws appropriate exceptions for each kind of failure it runs into
when creating the directory.
On Tue, Oct 23, 2012 at 7:35 PM, Ion Barcan <io...@gmail.com> wrote:
> Hi,
>
> I have a question related to 2 tests classes found in Hadoop Common
> (both 1.0.4 and 1.1.0): FileSystemContractBaseTest.java and
> FSMainOperationsBaseTest.java - which (as I understand) one could use
> to verify that a specific FileSystem implementation does what it's
> supposed to do.
>
> I'm a bit puzzled by the method
> testMkdirsFailsForSubdirectoryOfExistingFile() that is defined in both
> places. To me it looks like they're contradicting one another, i.e.
> one expects FileSystem.mkdirs() to fail if it encounters an existing
> file, the other expects FileSystem.mkdirs() to return false in the
> same situation. Which one is right? FileSystemContractBaseTest has 8
> implementors as opposed to FSMainOperationsBaseTest which only has 2 -
> but I still think this should be sorted out. For reference I've pasted
> the source code at the end of message.
>
> Thanks,
> Ion
>
> * FileSystemContractBaseTest.java
> 118 public void testMkdirsFailsForSubdirectoryOfExistingFile()
> throws Exception {
> 119 Path testDir = path("/test/hadoop");
> 120 assertFalse(fs.exists(testDir));
> 121 assertTrue(fs.mkdirs(testDir));
> 122 assertTrue(fs.exists(testDir));
> 123
> 124 createFile(path("/test/hadoop/file"));
> 125
> 126 Path testSubDir = path("/test/hadoop/file/subdir");
> 127 try {
> 128 fs.mkdirs(testSubDir);
> 129 fail("Should throw IOException.");
> 130 } catch (IOException e) {
> 131 // expected
> 132 }
> 133 assertFalse(fs.exists(testSubDir));
> 134
> 135 Path testDeepSubDir = path("/test/hadoop/file/deep/sub/dir");
> 136 try {
> 137 fs.mkdirs(testDeepSubDir);
> 138 fail("Should throw IOException.");
> 139 } catch (IOException e) {
> 140 // expected
> 141 }
> 142 assertFalse(fs.exists(testDeepSubDir));
> 143
> 144 }
>
> * FSMainOperationsBaseTest.java
> 198 @Test
> 199 public void testMkdirsFailsForSubdirectoryOfExistingFile()
> throws Exception {
> 200 Path testDir = getTestRootPath(fSys, "test/hadoop");
> 201 Assert.assertFalse(exists(fSys, testDir));
> 202 fSys.mkdirs(testDir);
> 203 Assert.assertTrue(exists(fSys, testDir));
> 204
> 205 createFile(getTestRootPath(fSys, "test/hadoop/file"));
> 206
> 207 Path testSubDir = getTestRootPath(fSys, "test/hadoop/file/subdir");
> 208 Assert.assertFalse(fSys.mkdirs(testSubDir));
> 209 Assert.assertFalse(exists(fSys, testSubDir));
> 210
> 211 Path testDeepSubDir = getTestRootPath(fSys,
> "test/hadoop/file/deep/sub/dir");
> 212 Assert.assertFalse(exists(fSys, testSubDir));
> 213 Assert.assertFalse(fSys.mkdirs(testDeepSubDir));
> 214 Assert.assertFalse(exists(fSys, testDeepSubDir));
> 215
> 216 }
--
Harsh J
Re: Conflicting mkdirs() behavior in abstract test classes from
Hadoop Common
Posted by Harsh J <ha...@cloudera.com>.
Hi Ion,
If you're looking at developing something, I recommend doing it on trunk :)
Onto your confusion, there are two FS interaction APIs: FileSystem
(older, stable) and FileContext (new, supports a lot of the new HDFS
features, but not yet popular as of today).
The FileSystem.mkdirs, as seen at
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#mkdirs(org.apache.hadoop.fs.Path),
returns a boolean if it fails. Obviously, this was not very helpful in
knowing WHY the operation failed.
The FileContext API was derived keeping in mind the problems devs
faced with the older FileSystem and with support for new features of
HDFS such as symlinks. The FileContext.mkdir API, as seen at
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir(org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean),
throws appropriate exceptions for each kind of failure it runs into
when creating the directory.
On Tue, Oct 23, 2012 at 7:35 PM, Ion Barcan <io...@gmail.com> wrote:
> Hi,
>
> I have a question related to 2 tests classes found in Hadoop Common
> (both 1.0.4 and 1.1.0): FileSystemContractBaseTest.java and
> FSMainOperationsBaseTest.java - which (as I understand) one could use
> to verify that a specific FileSystem implementation does what it's
> supposed to do.
>
> I'm a bit puzzled by the method
> testMkdirsFailsForSubdirectoryOfExistingFile() that is defined in both
> places. To me it looks like they're contradicting one another, i.e.
> one expects FileSystem.mkdirs() to fail if it encounters an existing
> file, the other expects FileSystem.mkdirs() to return false in the
> same situation. Which one is right? FileSystemContractBaseTest has 8
> implementors as opposed to FSMainOperationsBaseTest which only has 2 -
> but I still think this should be sorted out. For reference I've pasted
> the source code at the end of message.
>
> Thanks,
> Ion
>
> * FileSystemContractBaseTest.java
> 118 public void testMkdirsFailsForSubdirectoryOfExistingFile()
> throws Exception {
> 119 Path testDir = path("/test/hadoop");
> 120 assertFalse(fs.exists(testDir));
> 121 assertTrue(fs.mkdirs(testDir));
> 122 assertTrue(fs.exists(testDir));
> 123
> 124 createFile(path("/test/hadoop/file"));
> 125
> 126 Path testSubDir = path("/test/hadoop/file/subdir");
> 127 try {
> 128 fs.mkdirs(testSubDir);
> 129 fail("Should throw IOException.");
> 130 } catch (IOException e) {
> 131 // expected
> 132 }
> 133 assertFalse(fs.exists(testSubDir));
> 134
> 135 Path testDeepSubDir = path("/test/hadoop/file/deep/sub/dir");
> 136 try {
> 137 fs.mkdirs(testDeepSubDir);
> 138 fail("Should throw IOException.");
> 139 } catch (IOException e) {
> 140 // expected
> 141 }
> 142 assertFalse(fs.exists(testDeepSubDir));
> 143
> 144 }
>
> * FSMainOperationsBaseTest.java
> 198 @Test
> 199 public void testMkdirsFailsForSubdirectoryOfExistingFile()
> throws Exception {
> 200 Path testDir = getTestRootPath(fSys, "test/hadoop");
> 201 Assert.assertFalse(exists(fSys, testDir));
> 202 fSys.mkdirs(testDir);
> 203 Assert.assertTrue(exists(fSys, testDir));
> 204
> 205 createFile(getTestRootPath(fSys, "test/hadoop/file"));
> 206
> 207 Path testSubDir = getTestRootPath(fSys, "test/hadoop/file/subdir");
> 208 Assert.assertFalse(fSys.mkdirs(testSubDir));
> 209 Assert.assertFalse(exists(fSys, testSubDir));
> 210
> 211 Path testDeepSubDir = getTestRootPath(fSys,
> "test/hadoop/file/deep/sub/dir");
> 212 Assert.assertFalse(exists(fSys, testSubDir));
> 213 Assert.assertFalse(fSys.mkdirs(testDeepSubDir));
> 214 Assert.assertFalse(exists(fSys, testDeepSubDir));
> 215
> 216 }
--
Harsh J
Re: Conflicting mkdirs() behavior in abstract test classes from
Hadoop Common
Posted by Harsh J <ha...@cloudera.com>.
Hi Ion,
If you're looking at developing something, I recommend doing it on trunk :)
Onto your confusion, there are two FS interaction APIs: FileSystem
(older, stable) and FileContext (new, supports a lot of the new HDFS
features, but not yet popular as of today).
The FileSystem.mkdirs, as seen at
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#mkdirs(org.apache.hadoop.fs.Path),
returns a boolean if it fails. Obviously, this was not very helpful in
knowing WHY the operation failed.
The FileContext API was derived keeping in mind the problems devs
faced with the older FileSystem and with support for new features of
HDFS such as symlinks. The FileContext.mkdir API, as seen at
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir(org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean),
throws appropriate exceptions for each kind of failure it runs into
when creating the directory.
On Tue, Oct 23, 2012 at 7:35 PM, Ion Barcan <io...@gmail.com> wrote:
> Hi,
>
> I have a question related to 2 tests classes found in Hadoop Common
> (both 1.0.4 and 1.1.0): FileSystemContractBaseTest.java and
> FSMainOperationsBaseTest.java - which (as I understand) one could use
> to verify that a specific FileSystem implementation does what it's
> supposed to do.
>
> I'm a bit puzzled by the method
> testMkdirsFailsForSubdirectoryOfExistingFile() that is defined in both
> places. To me it looks like they're contradicting one another, i.e.
> one expects FileSystem.mkdirs() to fail if it encounters an existing
> file, the other expects FileSystem.mkdirs() to return false in the
> same situation. Which one is right? FileSystemContractBaseTest has 8
> implementors as opposed to FSMainOperationsBaseTest which only has 2 -
> but I still think this should be sorted out. For reference I've pasted
> the source code at the end of message.
>
> Thanks,
> Ion
>
> * FileSystemContractBaseTest.java
> 118 public void testMkdirsFailsForSubdirectoryOfExistingFile()
> throws Exception {
> 119 Path testDir = path("/test/hadoop");
> 120 assertFalse(fs.exists(testDir));
> 121 assertTrue(fs.mkdirs(testDir));
> 122 assertTrue(fs.exists(testDir));
> 123
> 124 createFile(path("/test/hadoop/file"));
> 125
> 126 Path testSubDir = path("/test/hadoop/file/subdir");
> 127 try {
> 128 fs.mkdirs(testSubDir);
> 129 fail("Should throw IOException.");
> 130 } catch (IOException e) {
> 131 // expected
> 132 }
> 133 assertFalse(fs.exists(testSubDir));
> 134
> 135 Path testDeepSubDir = path("/test/hadoop/file/deep/sub/dir");
> 136 try {
> 137 fs.mkdirs(testDeepSubDir);
> 138 fail("Should throw IOException.");
> 139 } catch (IOException e) {
> 140 // expected
> 141 }
> 142 assertFalse(fs.exists(testDeepSubDir));
> 143
> 144 }
>
> * FSMainOperationsBaseTest.java
> 198 @Test
> 199 public void testMkdirsFailsForSubdirectoryOfExistingFile()
> throws Exception {
> 200 Path testDir = getTestRootPath(fSys, "test/hadoop");
> 201 Assert.assertFalse(exists(fSys, testDir));
> 202 fSys.mkdirs(testDir);
> 203 Assert.assertTrue(exists(fSys, testDir));
> 204
> 205 createFile(getTestRootPath(fSys, "test/hadoop/file"));
> 206
> 207 Path testSubDir = getTestRootPath(fSys, "test/hadoop/file/subdir");
> 208 Assert.assertFalse(fSys.mkdirs(testSubDir));
> 209 Assert.assertFalse(exists(fSys, testSubDir));
> 210
> 211 Path testDeepSubDir = getTestRootPath(fSys,
> "test/hadoop/file/deep/sub/dir");
> 212 Assert.assertFalse(exists(fSys, testSubDir));
> 213 Assert.assertFalse(fSys.mkdirs(testDeepSubDir));
> 214 Assert.assertFalse(exists(fSys, testDeepSubDir));
> 215
> 216 }
--
Harsh J
Re: Conflicting mkdirs() behavior in abstract test classes from
Hadoop Common
Posted by Harsh J <ha...@cloudera.com>.
Hi Ion,
If you're looking at developing something, I recommend doing it on trunk :)
Onto your confusion, there are two FS interaction APIs: FileSystem
(older, stable) and FileContext (new, supports a lot of the new HDFS
features, but not yet popular as of today).
The FileSystem.mkdirs, as seen at
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#mkdirs(org.apache.hadoop.fs.Path),
returns a boolean if it fails. Obviously, this was not very helpful in
knowing WHY the operation failed.
The FileContext API was derived keeping in mind the problems devs
faced with the older FileSystem and with support for new features of
HDFS such as symlinks. The FileContext.mkdir API, as seen at
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir(org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean),
throws appropriate exceptions for each kind of failure it runs into
when creating the directory.
On Tue, Oct 23, 2012 at 7:35 PM, Ion Barcan <io...@gmail.com> wrote:
> Hi,
>
> I have a question related to 2 tests classes found in Hadoop Common
> (both 1.0.4 and 1.1.0): FileSystemContractBaseTest.java and
> FSMainOperationsBaseTest.java - which (as I understand) one could use
> to verify that a specific FileSystem implementation does what it's
> supposed to do.
>
> I'm a bit puzzled by the method
> testMkdirsFailsForSubdirectoryOfExistingFile() that is defined in both
> places. To me it looks like they're contradicting one another, i.e.
> one expects FileSystem.mkdirs() to fail if it encounters an existing
> file, the other expects FileSystem.mkdirs() to return false in the
> same situation. Which one is right? FileSystemContractBaseTest has 8
> implementors as opposed to FSMainOperationsBaseTest which only has 2 -
> but I still think this should be sorted out. For reference I've pasted
> the source code at the end of message.
>
> Thanks,
> Ion
>
> * FileSystemContractBaseTest.java
> 118 public void testMkdirsFailsForSubdirectoryOfExistingFile()
> throws Exception {
> 119 Path testDir = path("/test/hadoop");
> 120 assertFalse(fs.exists(testDir));
> 121 assertTrue(fs.mkdirs(testDir));
> 122 assertTrue(fs.exists(testDir));
> 123
> 124 createFile(path("/test/hadoop/file"));
> 125
> 126 Path testSubDir = path("/test/hadoop/file/subdir");
> 127 try {
> 128 fs.mkdirs(testSubDir);
> 129 fail("Should throw IOException.");
> 130 } catch (IOException e) {
> 131 // expected
> 132 }
> 133 assertFalse(fs.exists(testSubDir));
> 134
> 135 Path testDeepSubDir = path("/test/hadoop/file/deep/sub/dir");
> 136 try {
> 137 fs.mkdirs(testDeepSubDir);
> 138 fail("Should throw IOException.");
> 139 } catch (IOException e) {
> 140 // expected
> 141 }
> 142 assertFalse(fs.exists(testDeepSubDir));
> 143
> 144 }
>
> * FSMainOperationsBaseTest.java
> 198 @Test
> 199 public void testMkdirsFailsForSubdirectoryOfExistingFile()
> throws Exception {
> 200 Path testDir = getTestRootPath(fSys, "test/hadoop");
> 201 Assert.assertFalse(exists(fSys, testDir));
> 202 fSys.mkdirs(testDir);
> 203 Assert.assertTrue(exists(fSys, testDir));
> 204
> 205 createFile(getTestRootPath(fSys, "test/hadoop/file"));
> 206
> 207 Path testSubDir = getTestRootPath(fSys, "test/hadoop/file/subdir");
> 208 Assert.assertFalse(fSys.mkdirs(testSubDir));
> 209 Assert.assertFalse(exists(fSys, testSubDir));
> 210
> 211 Path testDeepSubDir = getTestRootPath(fSys,
> "test/hadoop/file/deep/sub/dir");
> 212 Assert.assertFalse(exists(fSys, testSubDir));
> 213 Assert.assertFalse(fSys.mkdirs(testDeepSubDir));
> 214 Assert.assertFalse(exists(fSys, testDeepSubDir));
> 215
> 216 }
--
Harsh J