You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Dave Marion (JIRA)" <ji...@apache.org> on 2012/10/29 18:56:13 UTC
[jira] [Created] (VFS-442) HDFS FileSystem Provider
Dave Marion created VFS-442:
-------------------------------
Summary: HDFS FileSystem Provider
Key: VFS-442
URL: https://issues.apache.org/jira/browse/VFS-442
Project: Commons VFS
Issue Type: New Feature
Affects Versions: 2.0
Reporter: Dave Marion
Priority: Minor
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492865#comment-13492865 ]
Dave Marion commented on VFS-442:
---------------------------------
Gary,
When I get a few minutes I'll check out the patch. Regarding the build directory, its an HDFS store that is created by the embedded HDFS test cluster. I'd have to look to see if that directory location is configurable. Can we just add this directory to the svn-ignore list?
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486227#comment-13486227 ]
Gary Gregory commented on VFS-442:
----------------------------------
The first thing to check is that you can build with Maven, I use Maven 3.0.4. For example: "mvn clean site".
I get an error:
{noformat}
[INFO] --- maven-bundle-plugin:2.3.7:manifest (bundle-manifest) @ commons-vfs2 ---
[ERROR] Manifest org.apache.commons:commons-vfs2:jar:2.1-SNAPSHOT : The default package '.' is not permitted by the Import-Package syntax.
This can be caused by compile errors in Eclipse because Eclipse creates
valid class files regardless of compile errors.
The following package(s) import from the default package [org.apache.commons.vfs2.provider.hdfs]
[ERROR] Error(s) found in manifest configuration
{noformat}
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dave Marion updated VFS-442:
----------------------------
Attachment: VFS-442-1.patch
Provides a ReadOnlyHDFSFileSystem that I used in ACCUMULO-708. This is draft 1. I figure that some changes will need to be made before being accepted.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487159#comment-13487159 ]
Dave Marion commented on VFS-442:
---------------------------------
Before I start fixing the issues that you identified above, adding Hadoop does make this somewhat non-portable. Do you think this is still a good candidate for including in VFS?
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487167#comment-13487167 ]
Dave Marion commented on VFS-442:
---------------------------------
bq. Is there a reason the test does not live in the VFS test hierarchy?
Not sure I follow, the test class is in the correct package:
{noformat}
package org.apache.commons.vfs2.provider.hdfs.test;
import java.io.IOException;
...
public class ReadOnlyHdfsFileProviderTest {
{noformat}
I checked the patch that I submitted, and it also appears in the correct location:
{noformat}
Index: core/src/test/java/org/apache/commons/vfs2/provider/hdfs/test/ReadOnlyHdfsFileProviderTest.java
===================================================================
--- core/src/test/java/org/apache/commons/vfs2/provider/hdfs/test/ReadOnlyHdfsFileProviderTest.java (revision 0)
+++ core/src/test/java/org/apache/commons/vfs2/provider/hdfs/test/ReadOnlyHdfsFileProviderTest.java (revision 0)
@@ -0,0 +1,344 @@
{noformat}
What am I missing?
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502538#comment-13502538 ]
Gary Gregory commented on VFS-442:
----------------------------------
Ok, this is getting better :)
Your 1,2,3s are what should be done next in order of importance.
(1) will help you flush out pieces that may be missing for a full VFS implementation. Steps (2) and (3) can come once we see what happens with a test class that fits in the VFS framework.
Thank you and keep up the good work!
Gary
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492086#comment-13492086 ]
Gary Gregory commented on VFS-442:
----------------------------------
Sigh, Cygwin is no panacea here:
{noformat}
org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest Time elapsed: 1600 sec <<< ERROR!
java.lang.ExceptionInInitializerError
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:425)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:283)
at org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest.<clinit>(HdfsFileProviderTest.java:95)
... 22 more
org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest Time elapsed: 1600 sec <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{noformat}
What is your set up? What does "mvn -version" tell you?
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch, VFS-442-2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487204#comment-13487204 ]
Gary Gregory commented on VFS-442:
----------------------------------
If you subclass AbstractProviderTestConfig, your provider will run against some predefined tests. It's all twisty code (JUnit 3, not 4) but that how providers are tested, which avoids a lot of code duplication.
If you are getting 100% code coverage with your test, then good. But you might not find out what methods or behaviors your provider is supposed to implement unless you fit in the test framework.
I would not dump your existing test though.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492870#comment-13492870 ]
Gary Gregory commented on VFS-442:
----------------------------------
Yes, we could/should add "build" to the SVN ignore list. But the test should still clean up after itself. It would also help if the directory name were more obvious like "test-hdfs" but like you mentioned, you'll see if the location can be configured.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dave Marion updated VFS-442:
----------------------------
Attachment: VFS-442-4.patch
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487177#comment-13487177 ]
Gary Gregory edited comment on VFS-442 at 10/30/12 8:10 PM:
------------------------------------------------------------
Hello Dave,
Yes, an HDFS provider makes sense, if there is NO way to get Hadoop running on Windows (is there? Cygwin?), then the test MUST be coded such that it is a no-op on Windows. The test Javadoc MUST make a note of this and suggest that Windows users build it under VirtualBox/Linux/Ubuntu (if that works).
Did you look at the other provider test cases? Most extend {{AbstractProviderTestConfig}} and/or implement {{ProviderTestConfig}}.
was (Author: garydgregory):
Hello Dave,
Yes, a HDFS makes sense, if there is NO way to get Hadoop running on Windows (is there? Cygwin?), then the test MUST be coded such that it is a no-op on Windows. The test Javadoc MUST make a note of this and suggest that Windows users build it under VirtualBox/Linux/Ubuntu (if that works).
Did you look at the other provider test cases? Most extend {{AbstractProviderTestConfig}} and/or implement {{ProviderTestConfig}}.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dave Marion updated VFS-442:
----------------------------
Attachment: VFS-442-2.patch
Changes based on feedback.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch, VFS-442-2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487182#comment-13487182 ]
Dave Marion commented on VFS-442:
---------------------------------
Hey Gary,
I believe that you can run Hadoop on Windows with Cygwin. Programmatic access to the file system is typically done using Hadoop client software. It does all of the communication with the NameNode and DataNodes to access the blocks on the nodes that make up the file you are interacting with. I did look at the other test cases, but I didn't see a reason to extend AbstractProviderTestConfig. If it is required, then I will do it.
Dave Marion
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492485#comment-13492485 ]
Gary Gregory commented on VFS-442:
----------------------------------
The patch works for me on Linux. I'll have to disable these tests on Windows and review the patch again. Thank you.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch, VFS-442-2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502492#comment-13502492 ]
Dave Marion commented on VFS-442:
---------------------------------
I attached patch 4. It includes changes that we made in ACCUMULO-708. These changes include getting the umask and setting the MiniDFSCluster to use the target directory instead of the build directory. I also added an hdfs profile in the core/pom.xml file that is supposed to exclude the hdfs test on Windows. Maybe someone building on Windows can verify that this works.
Remaining TO-DOs:
1. It is still important to provide a test that fits in this framework in addition to the current test. The current test may then become obsolete.
2. Some Javadoc comments are missing for public members.
3. Only make public what must be public.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Gregory updated VFS-442:
-----------------------------
Attachment: vfs-422-3.diff
Version 3 of the patch.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486450#comment-13486450 ]
Dave Marion commented on VFS-442:
---------------------------------
mvn clean site completed successfully. I looked around the site and noticed that I have a bunch of issues with checkstyle. Is there a code formatter template somewhere?
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486562#comment-13486562 ]
Gary Gregory commented on VFS-442:
----------------------------------
I do not know of code format templates for different IDEs. You might want to google around, someone might have created one for the version of your IDE. Otherwise, look at the existing code and adjust your IDE formatter to match.
Even though it does not build for me with Maven. I can give you some pointers:
# Get coverage as close to 100% as possible.
# No need to have read-only in the class names. That's just the capabilities of the provider today. If someone makes it read-write then there is no need for a class rename or a new class name.
# Always use blocks, even for one-liners: if (expr) { stmnt; }
# Only make public what will be set in stone. You can make something public tomorrow, but you cannot make it have less visibility and keep binary compatibility.
# Javadoc everything public.
# Use Java 5 enhanced for loops where possible.
# Don't initialize ivars to their default values, for example do this: "private FileSystem fs;", not this: "private FileSystem fs = null;"
# Make ivars final where possible
# Don't make the code platform dependent, for example, on Windows 7, I get:
{noformat}
java.lang.ExceptionInInitializerError
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "ls": CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(Unknown Source)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:200)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:703)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:418)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:393)
at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1537)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1484)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
at org.apache.commons.vfs2.provider.hdfs.test.ReadOnlyHdfsFileProviderTest.<clinit>(ReadOnlyHdfsFileProviderTest.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 33 more
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:443)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:393)
at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1537)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1484)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
at org.apache.commons.vfs2.provider.hdfs.test.ReadOnlyHdfsFileProviderTest.<clinit>(ReadOnlyHdfsFileProviderTest.java:85)
... 16 more
java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.vfs2.provider.hdfs.test.ReadOnlyHdfsFileProviderTest
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
{noformat}
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487177#comment-13487177 ]
Gary Gregory commented on VFS-442:
----------------------------------
Hello Dave,
Yes, a HDFS makes sense, if there is NO way to get Hadoop running on Windows (is there? Cygwin?), then the test MUST be coded such that it is a no-op on Windows. The test Javadoc MUST make a note of this and suggest that Windows users build it under VirtualBox/Linux/Ubuntu (if that works).
Did you look at the other provider test cases? Most extend {{AbstractProviderTestConfig}} and/or implement {{ProviderTestConfig}}.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502559#comment-13502559 ]
Dave Marion commented on VFS-442:
---------------------------------
I think I found it, DefaultFileReplicator(String dir). Thanks anyway.
Dave
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486574#comment-13486574 ]
Gary Gregory edited comment on VFS-442 at 10/30/12 1:09 AM:
------------------------------------------------------------
Also:
# Fix Checkstyle issues as appropriate.
# Fix PMD issues as appropriate. The PMD plugin is broken for VFS until version 2.8 comes out shortly I hope. Until then you should run PMD from your IDE, there are plugins for many IDEs.
# Why not use the latest hadoop-core version (1.1.0)?
was (Author: garydgregory):
Also:
# Fix Checkstyle issues as appropriate.
# Fix PMD issues as appropriate. The PMD plugin is broken for VFS until version 2.8 comes out shortly I hope. Until then you should run PMD from your IDE, there are plugins for many IDEs.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502558#comment-13502558 ]
Dave Marion commented on VFS-442:
---------------------------------
Gary,
On a side note, is there a way to change the directory that is used for the cache? Some of the Accumulo folks were asking, we see it using /tmp/vfs_cache by default. Thanks,
Dave
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486574#comment-13486574 ]
Gary Gregory commented on VFS-442:
----------------------------------
Also:
# Fix Checkstyle issues as appropriate.
# Fix PMD issues as appropriate. The PMD plugin is broken for VFS until version 2.8 comes out shortly I hope. Until then you should run PMD from your IDE, there are plugins for many IDEs.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487178#comment-13487178 ]
Gary Gregory commented on VFS-442:
----------------------------------
Also, is it possible to talk to Hadoop remotely? Over HTTP, sockets or SSH?
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487205#comment-13487205 ]
Gary Gregory commented on VFS-442:
----------------------------------
If you subclass AbstractProviderTestConfig, your provider will run against some predefined tests. It's all twisty code (JUnit 3, not 4) but that how providers are tested, which avoids a lot of code duplication.
If you are getting 100% code coverage with your test, then good. But you might not find out what methods or behaviors your provider is supposed to implement unless you fit in the test framework.
I would not dump your existing test though.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486571#comment-13486571 ]
Gary Gregory commented on VFS-442:
----------------------------------
We can put my Maven build issue aside now. It seems to have resolved itself. Maven can be black magic.
I still suffer from what seems to be Hadoop's snobbish attitude towards Windows: java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "ls": CreateProcess error=2, The system cannot find the file specified
Unless that is configurable?
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486571#comment-13486571 ]
Gary Gregory edited comment on VFS-442 at 10/30/12 1:02 AM:
------------------------------------------------------------
We can put my Maven build issue aside now. It seems to have resolved itself. Maven can be black magic.
I still suffer from what seems to be Hadoop's snobbish attitude towards Windows: java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "ls": CreateProcess error=2, The system cannot find the file specified
Unless that is configurable? Otherwise the test will have to be disabled on Windows.
Is there a reason the test does not live in the VFS test hierarchy?
was (Author: garydgregory):
We can put my Maven build issue aside now. It seems to have resolved itself. Maven can be black magic.
I still suffer from what seems to be Hadoop's snobbish attitude towards Windows: java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "ls": CreateProcess error=2, The system cannot find the file specified
Unless that is configurable?
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Gary Gregory (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492536#comment-13492536 ]
Gary Gregory commented on VFS-442:
----------------------------------
Hello Dave,
I've attached version 3 of the patch that:
- IMPORTANT: adds HDFS as a provider in providers.xml.
- adds missing @Overrides annotations,
- makes some ivars final,
- makes some vars private,
- Edited the Javadoc on the test to say Linux is required. As my previous comment shows, Cygwin on Windows does not work.
I cannot run most of the VFS tests in Eclipse due to the way the VFS test framework was coded a long time ago. That's unfortunate, but that's what we have today. I often resort to debugging VFS from Eclipse by connecting to the JVM started by a local Maven run.
TO-DOs:
# It is still important to provide a test that fits in this framework in addition to the current test. The current test may then become obsolete.
# The test needs to be disabled on Windows. It might be best to do that from the POM, which I don't know how to do off the top of my head. JUnit's Assume may be the solution of last resort.
# The Maven build creates a "build" directory in the root of the project, that's not good because someone may end up committing it by mistake or including it in a patch (like I almost did). Is this an HDFS store? If so, put it in a folder where the system propoerty java.io.tmpdir points to and remove it at the end of the test.
# Some Javadoc comments are missing for public members.
# Only make public what must be public.
Thank you for your time and contribution!
Gary
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
Posted by "Dave Marion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492044#comment-13492044 ]
Dave Marion commented on VFS-442:
---------------------------------
bq. 1. Get coverage as close to 100% as possible.
I'm having an issue running code coverage in Eclipse. However, I think its close based on what I did in ACCUMULO-708
bq. 2. No need to have read-only in the class names. That's just the capabilities of the provider today. If someone makes it read-write then there is no need for a class rename or a new class name.
Done.
bq. 3. Always use blocks, even for one-liners: if (expr) { stmnt; }
Fixed most checkstyle issues that made sense.
bq. 4. Only make public what will be set in stone. You can make something public tomorrow, but you cannot make it have less visibility and keep binary compatibility.
bq. 5. Javadoc everything public.
Fixed most issues.
bq. 6. Use Java 5 enhanced for loops where possible.
bq. 7. Don't initialize ivars to their default values, for example do this: "private FileSystem fs;", not this: "private FileSystem fs = null;"
bq. 8. Make ivars final where possible
Fixed most issues.
bq. 9. Don't make the code platform dependent, for example, on Windows 7
Added comments in test that it needs to run on Hadoop compatible environments, need Cygwin for Windows.
Upgraded to Hadoop dependency to version 1.1.0. I will upload a new patch in a few moments to take the place of the first patch.
> HDFS FileSystem Provider
> ------------------------
>
> Key: VFS-442
> URL: https://issues.apache.org/jira/browse/VFS-442
> Project: Commons VFS
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Dave Marion
> Priority: Minor
> Labels: accumulo, hdfs
> Attachments: VFS-442-1.patch, VFS-442-2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira