You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Bernd Eckenfels <ec...@zusammenkunft.net> on 2015/01/09 15:05:55 UTC
Re: [VFS] Implementing custom hdfs file system using commons-vfs
2.0
Hello Dave,
for the download page (staged:
http://people.apache.org/~ecki/commons-vfs/download.html) I need a list
of libraries needed to run VFS to access hdfs. Could you maybe produce
an example VFS Shell session similiar to
https://wiki.apache.org/commons/VfsReleaseState
where you list the command line needed. It would be best if this is an
public HDFS instance (but I guess there is no such thing?)
BTW: I had asked in the VFS-530 bug about how commonplace the different
hdfs APIs are, and if it is really good if we bump the minimum version.
As I understand it. Will HDFS with 1.1.2 be able to communicate with
2.6 instances? If yes I would prefer to keep it at 1.2 in this release.
WDYT?
Gruss
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [VFS] Implementing custom hdfs file system using commons-vfs
2.0
Posted by Bernd Eckenfels <ec...@zusammenkunft.net>.
Hello,
with this commit I added a cleanup of the data dir before the
DfsMiniCluster is started. I also use absolute file names to make
debugging a bit easier and I moved initialisation code to the
setUp() method
http://svn.apache.org/r1650847 & http://svn.apache.org/r1650852
This way the test do not error out anymore. But I have no idea why this
was happening on one machine and not on others (maybe a race, the
failing machine had SSD?).
So this means, now I can concentrate on merging the new version.
Gruss
Bernd
Am Sun, 11 Jan 2015 01:25:48 +0100 schrieb Bernd Eckenfels
<ec...@zusammenkunft.net>:
> Hello,
>
> Am Sat, 10 Jan 2015 03:12:19 +0000 (UTC)
> schrieb dlmarion@comcast.net:
>
> > Bernd,
> >
> > Regarding the Hadoop version for VFS 2.1, why not use the latest on
> > the first release of the HDFS provider? The Hadoop 1.1.2 release was
> > released in Feb 2013.
>
> Yes, you are right. We dont need to care about 2.0 as this is a new
> provider. I will make the changes, just want to fix the current test
> failures I see first.
>
>
> > I just built 2.1-SNAPSHOT over the holidays with JDK 6, 7, and 8 on
> > Ubuntu. What type of test errors are you getting? Testing is
> > disabled on Windows unless you decide to pull in windows artifacts
> > attached to VFS-530. However, those artifacts are associated with
> > patch 3 and are for Hadoop 2.4.0. Updating to 2.4.0 would also be
> > sufficient in my opinion.
>
> Yes, what I mean is: I typically build under Windows so I would not
> notice if the test starts to fail. However it seems to pass on the
> integration build:
>
> https://continuum-ci.apache.org/continuum/projectView.action?projectId=129&projectGroupId=16
>
> Running
> org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
> Starting DataNode 0 with dfs.data.dir:
> target/build/test/data/dfs/data/data1,target/build/test/data/dfs/data/data2
> Cluster is active Cluster is active Tests run: 13, Failures: 0,
> Errors: 0, Skipped: 0, Time elapsed: 11.821 sec - in
> org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
> Running
> org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase
> Starting DataNode 0 with dfs.data.dir:
> target/build/test2/data/dfs/data/data1,target/build/test2/data/dfs/data/data2
> Cluster is active Cluster is active Tests run: 76, Failures: 0,
> Errors: 0, Skipped: 0, Time elapsed: 18.853 sec - in
> org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase
>
> Anyway, on a Ubuntu, I get this exception currently:
>
> Running
> org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase
> Starting DataNode 0 with dfs.data.dir:
> target/build/test/data/dfs/data/data1,tar
> get/build/test/data/dfs/data/data2 Cluster is active Cluster is
> active Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed: 1.486 sec <<< FA
> ILURE! - in
> org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase
> junit.framework.TestSuite@56c77035(org.apache.commons.vfs2.provider.hdfs.test.Hd
> fsFileProviderTestCase$HdfsProviderTestSuite) Time elapsed: 1.479
> sec <<< ERRO R!
> java.lang.RuntimeException: Error setting up mini cluster at
> org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase$H
> dfsProviderTestSuite.setUp(HdfsFileProviderTestCase.java:112) at
> org.apache.commons.vfs2.test.AbstractTestSuite$1.protect(AbstractTest
> Suite.java:148) at
> junit.framework.TestResult.runProtected(TestResult.java:142) at
> org.apache.commons.vfs2.test.AbstractTestSuite.run(AbstractTestSuite.
> java:154) at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.
> java:86) at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provide
> r.java:283) at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUni
> t4Provider.java:173) at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4
> Provider.java:153) at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider .java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameCla
> ssLoader(ForkedBooter.java:203) at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(Fork
> edBooter.java:155) at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:
> 103) Caused by: java.io.IOException: Cannot lock storage
> target/build/test/data/dfs/n
> ame1. The directory is already locked. at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(St
> orage.java:599) at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:13
> 27) at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:13
> 45) at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:
> 1207) at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:
> 187) at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:268)
> at
> org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase$H
> dfsProviderTestSuite.setUp(HdfsFileProviderTestCase.java:107) ... 11
> more
>
> Running
> org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest Tests
> run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.445 sec
> - in
>
> When I delete the core/target/build/test/data/dfs/ directory and then
> run the ProviderTest I can do that multiple times and it works:
>
> mvn surefire:test
> -Dtest=org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
>
> But when I run all tests or the HdfsFileProviderTestCase then it
> fails and afterwards not even the ProviderTest suceeds until I delete
> that dir.
>
> (I suspect the "locking" is a missleading error, looks more like the
> data pool has some kind of instance ID which it does not have at the
> next run)
>
> Looks like TestCase has a problem and ProviderTest does no proper
> pre-cleaning. Will check the source. More generally speaking it
> should not use a fixed working directory anyway.
>
>
> > I started up Hadoop 2.6.0 on my laptop, created a directory and
> > file, then used the VFS shell to list and view the contents
> > (remember, HDFS provider is read-only currently). Here is the what
> > I did:
>
> Looks good. I will shorten it a bit and add it to the wiki. BTW: the
> warning, is this something we can change?
>
> Gruss
> Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [VFS] Implementing custom hdfs file system using commons-vfs
2.0
Posted by Bernd Eckenfels <ec...@zusammenkunft.net>.
Hello,
Am Sat, 10 Jan 2015 03:12:19 +0000 (UTC)
schrieb dlmarion@comcast.net:
> Bernd,
>
> Regarding the Hadoop version for VFS 2.1, why not use the latest on
> the first release of the HDFS provider? The Hadoop 1.1.2 release was
> released in Feb 2013.
Yes, you are right. We dont need to care about 2.0 as this is a new
provider. I will make the changes, just want to fix the current test
failures I see first.
> I just built 2.1-SNAPSHOT over the holidays with JDK 6, 7, and 8 on
> Ubuntu. What type of test errors are you getting? Testing is disabled
> on Windows unless you decide to pull in windows artifacts attached to
> VFS-530. However, those artifacts are associated with patch 3 and are
> for Hadoop 2.4.0. Updating to 2.4.0 would also be sufficient in my
> opinion.
Yes, what I mean is: I typically build under Windows so I would not
notice if the test starts to fail. However it seems to pass on the
integration build:
https://continuum-ci.apache.org/continuum/projectView.action?projectId=129&projectGroupId=16
Running org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
Starting DataNode 0 with dfs.data.dir: target/build/test/data/dfs/data/data1,target/build/test/data/dfs/data/data2
Cluster is active
Cluster is active
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.821 sec - in org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
Running org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase
Starting DataNode 0 with dfs.data.dir: target/build/test2/data/dfs/data/data1,target/build/test2/data/dfs/data/data2
Cluster is active
Cluster is active
Tests run: 76, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.853 sec - in org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase
Anyway, on a Ubuntu, I get this exception currently:
Running org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase
Starting DataNode 0 with dfs.data.dir: target/build/test/data/dfs/data/data1,tar get/build/test/data/dfs/data/data2
Cluster is active
Cluster is active
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<< FA ILURE! - in org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase
junit.framework.TestSuite@56c77035(org.apache.commons.vfs2.provider.hdfs.test.Hd fsFileProviderTestCase$HdfsProviderTestSuite) Time elapsed: 1.479 sec <<< ERRO R!
java.lang.RuntimeException: Error setting up mini cluster
at org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase$H dfsProviderTestSuite.setUp(HdfsFileProviderTestCase.java:112)
at org.apache.commons.vfs2.test.AbstractTestSuite$1.protect(AbstractTest Suite.java:148)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at org.apache.commons.vfs2.test.AbstractTestSuite.run(AbstractTestSuite. java:154)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner. java:86)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provide r.java:283)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUni t4Provider.java:173)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4 Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider .java:128)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameCla ssLoader(ForkedBooter.java:203)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(Fork edBooter.java:155)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java: 103)
Caused by: java.io.IOException: Cannot lock storage target/build/test/data/dfs/n ame1. The directory is already locked.
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(St orage.java:599)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:13 27)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:13 45)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java: 1207)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java: 187)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:268)
at org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase$H dfsProviderTestSuite.setUp(HdfsFileProviderTestCase.java:107)
... 11 more
Running org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.445 sec - in
When I delete the core/target/build/test/data/dfs/ directory and then run the ProviderTest I can do that multiple times and it works:
mvn surefire:test -Dtest=org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
But when I run all tests or the HdfsFileProviderTestCase then it fails and afterwards not even the ProviderTest suceeds until I delete that dir.
(I suspect the "locking" is a missleading error, looks more like the data pool has some kind of instance ID which it does not have at the next run)
Looks like TestCase has a problem and ProviderTest does no proper pre-cleaning. Will check the source. More generally speaking it should not use a fixed working directory anyway.
> I started up Hadoop 2.6.0 on my laptop, created a directory and file,
> then used the VFS shell to list and view the contents (remember, HDFS
> provider is read-only currently). Here is the what I did:
Looks good. I will shorten it a bit and add it to the wiki. BTW: the warning, is this something we can change?
Gruss
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [VFS] Implementing custom hdfs file system using commons-vfs
2.0
Posted by dl...@comcast.net.
Bernd,
Regarding the Hadoop version for VFS 2.1, why not use the latest on the first release of the HDFS provider? The Hadoop 1.1.2 release was released in Feb 2013.
I just built 2.1-SNAPSHOT over the holidays with JDK 6, 7, and 8 on Ubuntu. What type of test errors are you getting? Testing is disabled on Windows unless you decide to pull in windows artifacts attached to VFS-530. However, those artifacts are associated with patch 3 and are for Hadoop 2.4.0. Updating to 2.4.0 would also be sufficient in my opinion.
I started up Hadoop 2.6.0 on my laptop, created a directory and file, then used the VFS shell to list and view the contents (remember, HDFS provider is read-only currently). Here is the what I did:
./hadoop fs -ls /
./hadoop fs -mkdir /vfs-test
./hadoop fs -ls /
Found 1 items
drwxr-xr-x - dave supergroup 0 2015-01-09 21:50 /vfs-test
echo "This is a test" > /tmp/test.txt
./hadoop fs -copyFromLocal /tmp/test.txt /vfs-test/text.txt
./hadoop fs -ls -R /
drwxr-xr-x - dave supergroup 0 2015-01-09 21:56 /vfs-test
-rw-r--r-- 3 dave supergroup 15 2015-01-09 21:56 /vfs-test/text.txt
./hadoop fs -cat /vfs-test/text.txt
This is a test
unset REP
unset HADOOP_HOME
unset HADOOP_CLASSPATH
unset LIBS
REP=~/.m2/repository
HADOOP_HOME=/home/dave/Software/hadoop-2.6.0
HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
LIBS=$REP/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
LIBS=$LIBS:core/target/commons-vfs2-2.1-SNAPSHOT.jar:examples/target/commons-vfs2-examples-2.1-SNAPSHOT.jar:sandbox/target/commons-VFS2-sandbox-2.1.jar
LIBS=$LIBS:$HADOOP_CLASSPATH
java -cp $LIBS org.apache.commons.vfs2.example.Shell
10:01:41 ~/EclipseWorkspace/commons-vfs2-project$ unset REP
10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ unset HADOOP_HOME
10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ unset HADOOP_CLASSPATH
10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ unset LIBS
10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ REP=~/.m2/repository
10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ HADOOP_HOME=/home/dave/Software/hadoop-2.6.0
10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$REP/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$LIBS:core/target/commons-vfs2-2.1-SNAPSHOT.jar:examples/target/commons-vfs2-examples-2.1-SNAPSHOT.jar:sandbox/target/commons-VFS2-sandbox-2.1.jar
10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$LIBS:$HADOOP_CLASSPATH
10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ java -cp $LIBS org.apache.commons.vfs2.example.Shell
15/01/09 22:01:44 INFO impl.StandardFileSystemManager: Using "/tmp/vfs_cache" as temporary files store.
VFS Shell 2.1-SNAPSHOT
> info
Default manager: "org.apache.commons.vfs2.impl.StandardFileSystemManager" version 2.1-SNAPSHOT
Provider Schemes: [https, res, gz, hdfs, sftp, ftps, ram, http, file, ftp, tmp, bz2]
Virtual Schemes: [zip, war, par, ear, jar, sar, ejb3, tar, tbz2, tgz]
> info hdfs
Provider Info for scheme "hdfs":
capabilities: [GET_TYPE, READ_CONTENT, URI, GET_LAST_MODIFIED, ATTRIBUTES, RANDOM_ACCESS_READ, DIRECTORY_READ_CONTENT, LIST_CHILDREN]
> ls hdfs://Dave-laptop:8020/
15/01/09 22:02:06 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
15/01/09 22:02:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Contents of hdfs://dave-laptop:8020/
vfs-test/
> ls hdfs://Dave-laptop:8020/vfs-test/
Contents of hdfs://dave-laptop:8020/vfs-test
text.txt
> cat hdfs://Dave-laptop:8020/vfs-test/text.txt
This is a test
----- Original Message -----
From: "Bernd Eckenfels" <ec...@zusammenkunft.net>
To: dlmarion@comcast.net
Cc: "Commons Developers List" <de...@commons.apache.org>
Sent: Friday, January 9, 2015 8:13:27 PM
Subject: Re: [VFS] Implementing custom hdfs file system using commons-vfs 2.0
Am Sat, 10 Jan 2015 01:04:56 +0000 (UTC)
schrieb dlmarion@comcast.net:
> To answer your question about the HDFS version, I went back and
> looked at the patches I supplied for VFS-530. In the patches that I
> supplied to bump the Hadoop version to 2.4.0 and 2.6.0 the only
> changes were in the pom. This should mean that, for the parts of the
> Hadoop client objects that I am using, they are API compatible.
Well, you never know. But if thats the case there is also no problem if
we stay at the older version for now.
> For the example, the list of jars depends on the Hadoop version. I
> think one way to handle that is to use the `hadoop classpath` command
> to create the classpath. The example would then look like:
Looks great, thanks.
> VFS Shell 2.1-SNAPSHOT
> > info
> Default manager:
> "org.apache.commons.vfs2.impl.StandardFileSystemManager" version
> 2.1-SNAPSHOT Provider Schemes: [https, res, gz, hdfs, sftp, ftps,
> ram, http, file, ftp, tmp, bz2] Virtual Schemes: [zip, war, par, ear,
> jar, sar, ejb3, tar, tbz2, tgz]
>
>
> Is this sufficient?
Did you try to use ls or another command to actually connect to a
hdfs name node? I think you should be able to provide
"user:password@host" with the Shell.
I have currently the problem, that the hdfs tests fail (under Linux)
with some form of directory lock problem. I am not sure if this worked
before (as I havent tested on my windows). I will see if the version
upgrade helps, if yes I will for sure include it in 2.1
Greetings
Bernd
>
> Dave
>
> ----- Original Message -----
>
> From: "Bernd Eckenfels" <ec...@zusammenkunft.net>
> To: dlmarion@comcast.net
> Cc: "Commons Developers List" <de...@commons.apache.org>
> Sent: Friday, January 9, 2015 9:05:55 AM
> Subject: Re: [VFS] Implementing custom hdfs file system using
> commons-vfs 2.0
>
> Hello Dave,
>
> for the download page (staged:
> http://people.apache.org/~ecki/commons-vfs/download.html) I need a
> list of libraries needed to run VFS to access hdfs. Could you maybe
> produce an example VFS Shell session similiar to
>
> https://wiki.apache.org/commons/VfsReleaseState
>
> where you list the command line needed. It would be best if this is
> an public HDFS instance (but I guess there is no such thing?)
>
> BTW: I had asked in the VFS-530 bug about how commonplace the
> different hdfs APIs are, and if it is really good if we bump the
> minimum version. As I understand it. Will HDFS with 1.1.2 be able to
> communicate with 2.6 instances? If yes I would prefer to keep it at
> 1.2 in this release. WDYT?
>
> Gruss
> Bernd
>
>
Re: [VFS] Implementing custom hdfs file system using commons-vfs
2.0
Posted by Bernd Eckenfels <ec...@zusammenkunft.net>.
Am Sat, 10 Jan 2015 01:04:56 +0000 (UTC)
schrieb dlmarion@comcast.net:
> To answer your question about the HDFS version, I went back and
> looked at the patches I supplied for VFS-530. In the patches that I
> supplied to bump the Hadoop version to 2.4.0 and 2.6.0 the only
> changes were in the pom. This should mean that, for the parts of the
> Hadoop client objects that I am using, they are API compatible.
Well, you never know. But if thats the case there is also no problem if
we stay at the older version for now.
> For the example, the list of jars depends on the Hadoop version. I
> think one way to handle that is to use the `hadoop classpath` command
> to create the classpath. The example would then look like:
Looks great, thanks.
> VFS Shell 2.1-SNAPSHOT
> > info
> Default manager:
> "org.apache.commons.vfs2.impl.StandardFileSystemManager" version
> 2.1-SNAPSHOT Provider Schemes: [https, res, gz, hdfs, sftp, ftps,
> ram, http, file, ftp, tmp, bz2] Virtual Schemes: [zip, war, par, ear,
> jar, sar, ejb3, tar, tbz2, tgz]
>
>
> Is this sufficient?
Did you try to use ls or another command to actually connect to a
hdfs name node? I think you should be able to provide
"user:password@host" with the Shell.
I have currently the problem, that the hdfs tests fail (under Linux)
with some form of directory lock problem. I am not sure if this worked
before (as I havent tested on my windows). I will see if the version
upgrade helps, if yes I will for sure include it in 2.1
Greetings
Bernd
>
> Dave
>
> ----- Original Message -----
>
> From: "Bernd Eckenfels" <ec...@zusammenkunft.net>
> To: dlmarion@comcast.net
> Cc: "Commons Developers List" <de...@commons.apache.org>
> Sent: Friday, January 9, 2015 9:05:55 AM
> Subject: Re: [VFS] Implementing custom hdfs file system using
> commons-vfs 2.0
>
> Hello Dave,
>
> for the download page (staged:
> http://people.apache.org/~ecki/commons-vfs/download.html) I need a
> list of libraries needed to run VFS to access hdfs. Could you maybe
> produce an example VFS Shell session similiar to
>
> https://wiki.apache.org/commons/VfsReleaseState
>
> where you list the command line needed. It would be best if this is
> an public HDFS instance (but I guess there is no such thing?)
>
> BTW: I had asked in the VFS-530 bug about how commonplace the
> different hdfs APIs are, and if it is really good if we bump the
> minimum version. As I understand it. Will HDFS with 1.1.2 be able to
> communicate with 2.6 instances? If yes I would prefer to keep it at
> 1.2 in this release. WDYT?
>
> Gruss
> Bernd
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [VFS] Implementing custom hdfs file system using commons-vfs
2.0
Posted by dl...@comcast.net.
Bernd,
To answer your question about the HDFS version, I went back and looked at the patches I supplied for VFS-530. In the patches that I supplied to bump the Hadoop version to 2.4.0 and 2.6.0 the only changes were in the pom. This should mean that, for the parts of the Hadoop client objects that I am using, they are API compatible.
For the example, the list of jars depends on the Hadoop version. I think one way to handle that is to use the `hadoop classpath` command to create the classpath. The example would then look like:
REP=~/.m2/repository
HADOOP_HOME=<PATH TO HADOOP>
HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
LIBS=$REP/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
LIBS=$LIBS:core/target/commons-vfs2-2.1-SNAPSHOT.jar:examples/target/commons-vfs2-examples-2.1-SNAPSHOT.jar:sandbox/target/commons-VFS2-sandbox-2.1.jar
LIBS=$LIBS:$HADOOP_CLASSPATH
java -cp $LIBS org.apache.commons.vfs2.example.Shell
On my local machine, this looks like:
08:03:17 ~/EclipseWorkspace/commons-vfs2-project$ unset REP
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ unset HADOOP_HOME
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ unset LIBS
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ REP=~/.m2/repository
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ HADOOP_HOME=/home/dave/Software/hadoop-2.6.0
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$REP/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$LIBS:core/target/commons-vfs2-2.1-SNAPSHOT.jar:examples/target/commons-vfs2-examples-2.1-SNAPSHOT.jar:sandbox/target/commons-VFS2-sandbox-2.1.jar
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$LIBS:$HADOOP_CLASSPATH
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ java -cp $LIBS org.apache.commons.vfs2.example.Shell
15/01/09 20:03:18 INFO impl.StandardFileSystemManager: Using "/tmp/vfs_cache" as temporary files store.
VFS Shell 2.1-SNAPSHOT
> info
Default manager: "org.apache.commons.vfs2.impl.StandardFileSystemManager" version 2.1-SNAPSHOT
Provider Schemes: [https, res, gz, hdfs, sftp, ftps, ram, http, file, ftp, tmp, bz2]
Virtual Schemes: [zip, war, par, ear, jar, sar, ejb3, tar, tbz2, tgz]
Is this sufficient?
Dave
----- Original Message -----
From: "Bernd Eckenfels" <ec...@zusammenkunft.net>
To: dlmarion@comcast.net
Cc: "Commons Developers List" <de...@commons.apache.org>
Sent: Friday, January 9, 2015 9:05:55 AM
Subject: Re: [VFS] Implementing custom hdfs file system using commons-vfs 2.0
Hello Dave,
for the download page (staged:
http://people.apache.org/~ecki/commons-vfs/download.html) I need a list
of libraries needed to run VFS to access hdfs. Could you maybe produce
an example VFS Shell session similiar to
https://wiki.apache.org/commons/VfsReleaseState
where you list the command line needed. It would be best if this is an
public HDFS instance (but I guess there is no such thing?)
BTW: I had asked in the VFS-530 bug about how commonplace the different
hdfs APIs are, and if it is really good if we bump the minimum version.
As I understand it. Will HDFS with 1.1.2 be able to communicate with
2.6 instances? If yes I would prefer to keep it at 1.2 in this release.
WDYT?
Gruss
Bernd
Re: [VFS] Implementing custom hdfs file system using commons-vfs
2.0
Posted by dl...@comcast.net.
Bernd,
Wanted to get back to you right away. I should be able to get the information to you in a day or so (hopefully tonight). I don't know of any public HDFS instances, but I will see what I can find. Regarding VFS-530, I just wanted to bump the version to the latest for the 2.1 release. I will have to ask on the hdfs-dev list if the Hadoop 1.1.2 and 2.6.0 are api compatible.
Dave
----- Original Message -----
From: "Bernd Eckenfels" <ec...@zusammenkunft.net>
To: dlmarion@comcast.net
Cc: "Commons Developers List" <de...@commons.apache.org>
Sent: Friday, January 9, 2015 9:05:55 AM
Subject: Re: [VFS] Implementing custom hdfs file system using commons-vfs 2.0
Hello Dave,
for the download page (staged:
http://people.apache.org/~ecki/commons-vfs/download.html) I need a list
of libraries needed to run VFS to access hdfs. Could you maybe produce
an example VFS Shell session similiar to
https://wiki.apache.org/commons/VfsReleaseState
where you list the command line needed. It would be best if this is an
public HDFS instance (but I guess there is no such thing?)
BTW: I had asked in the VFS-530 bug about how commonplace the different
hdfs APIs are, and if it is really good if we bump the minimum version.
As I understand it. Will HDFS with 1.1.2 be able to communicate with
2.6 instances? If yes I would prefer to keep it at 1.2 in this release.
WDYT?
Gruss
Bernd