You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Bernd Eckenfels <ec...@zusammenkunft.net> on 2014/06/18 04:46:56 UTC
MiniDFSCluster on Windows unfortunatelly requires native toolchain
for automated tests
Hello,
in the Commons VFS project we have a Provider for HDFS. This
Provider is unit-tested with the help of the excelent MiniDFSCluster
component.
A while back I enabled automated testing on Windows (cause it worked
for me), but I did not realize it only works with cygwin (namely
ls.exe) in the Path. This was some 1.2 version and it used the ls
command to read the owner.
We currently switch the dependency to 2.4.0 and had the hope that it better (i.e.
without binary dependencies) works in this scenario - as we want to run
the maven tests without pre-installing a tool chain on Windows.
However it got actually worse, it warns about a missing winutils.exe
and it requires the hadoop.dll. The first location where it fails is
testing writeability for the name node. And unlike all the comments in
FileUtil/NativeIO it does not cleanly fall back to java portable
methods.
I think the access/access0 method can easily be fixed, but before I work on this patch, does it make sense to go in this direction or will I have to fix some other, more complicated platform dependencies. My goal would be to start MiniDFSCluster inside surefire from maven without the need for complicated external toolchain setup.
Is actually anybody caring about making this possible?
org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest Time elapsed: 2.291 sec <<< ERROR!
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:292)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:879)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:455)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:511)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:670)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:655)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1304)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:975)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:856)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:640)
at org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest.<clinit>(HdfsFileProviderTest.java:95)
Greetings
Bernd
PS - in case you want to follow the VFS progress:
* https://issues.apache.org/jira/browse/VFS-530 - tracks 2.4 migration
* https://issues.apache.org/jira/browse/VFS-529 - the broken windows tests
* https://issues.apache.org/jira/browse/VFS-526 - failed enable of win tests
Re: MiniDFSCluster on Windows unfortunatelly requires native
toolchain for automated tests
Posted by Bernd Eckenfels <ec...@zusammenkunft.net>.
Hello Chris,
thanks for answering. I am not happy about the fact, that it might be
not possible to have generic fallbacks in all cases.
For the actual
method in question I think File#isWriteable is a perfectly fine
fallback especially as it is used in Posix systems anyway (and is never
really reliable). I think the method is also used without a real need
in this place. I hade the hope that other places look similiar.
Of course I can understand that you dont want to allow generic
fallbacks in production environments. Would you suggest to have a
system property?
I have seen you comments on the VFS bugs. I am not so sure about the
binaries policy for testing in apache commons, I will need to check
this. However I would simply disable the tests on Windows again.
There are some comments talking about the need for fallbacks in the
source. If this is not the actual desire it should be removed.
Gruss
Bernd
Am Fri, 20 Jun 2014 10:02:49 -0700
schrieb Chris Nauroth <cn...@hortonworks.com>:
> Hello Bernd,
>
> This is my first time looking at Commons VFS. It looks like a very
> useful project, and I'm sorry to hear that we've caused some grief
> for your tests.
>
> I can provide some background on these changes. Traditionally, 1.2
> and earlier versions didn't officially support Windows beyond a best
> effort to keep some approximation of the functionality running on
> developer desktops. As we worked on enabling real production
> deployments on Windows, we discovered multiple bugs caused by those
> approximations.
>
> In some cases, the Cygwin and GnuWin32 tools had incomplete
> implementations or bugs. Examples of the buggy areas include finding
> the groups for a user and process management. Basically, anything in
> org.apache.hadoop.util.Shell was problematic. For this category of
> bugs, the solution was winutils.exe, which delegates straight to the
> necessary Windows API calls.
>
> Other problems arose from bugs or missing functionality in the
> Windows JDK implementation. For example, the canRead/canWrite
> methods in java.io.File don't work correctly on Windows. (This is
> the code path shown in your stack trace.) For this category of bugs,
> the solution was hadoop.dll, which is a JNI layer that delegates to
> the Windows API.
>
> Without these workarounds, production deployment on Windows wouldn't
> have been feasible. There would have been too many dangerous bugs.
>
> Unfortunately, we have a weakness right now in that the Windows native
> build artifacts aren't bundled into the Apache releases. We also
> don't have a fallback mode that works without the native code. (Even
> if there was such a fallback mode, it would be very dangerous to run
> in this mode in production, so I'd want to treat it as a test-only
> setting.)
>
> I expect you'll run into a lot of challenges while fully implementing
> a Java-only fallback. I'd like to help by checking out the Commons
> VFS code and experimenting a bit. I'll try to put some time into
> this next week and then comment on the VFS jira issues with my
> findings. Does this sound good?
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Tue, Jun 17, 2014 at 7:46 PM, Bernd Eckenfels
> <ec...@zusammenkunft.net> wrote:
>
> > Hello,
> >
> > in the Commons VFS project we have a Provider for HDFS. This
> > Provider is unit-tested with the help of the excelent MiniDFSCluster
> > component.
> >
> > A while back I enabled automated testing on Windows (cause it worked
> > for me), but I did not realize it only works with cygwin (namely
> > ls.exe) in the Path. This was some 1.2 version and it used the ls
> > command to read the owner.
> >
> > We currently switch the dependency to 2.4.0 and had the hope that it
> > better (i.e.
> > without binary dependencies) works in this scenario - as we want to
> > run the maven tests without pre-installing a tool chain on Windows.
> >
> > However it got actually worse, it warns about a missing winutils.exe
> > and it requires the hadoop.dll. The first location where it fails is
> > testing writeability for the name node. And unlike all the comments
> > in FileUtil/NativeIO it does not cleanly fall back to java portable
> > methods.
> >
> > I think the access/access0 method can easily be fixed, but before I
> > work on this patch, does it make sense to go in this direction or
> > will I have to fix some other, more complicated platform
> > dependencies. My goal would be to start MiniDFSCluster inside
> > surefire from maven without the need for complicated external
> > toolchain setup.
> >
> > Is actually anybody caring about making this possible?
> >
> > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
> > Time elapsed: 2.291 sec <<< ERROR!
> > java.lang.UnsatisfiedLinkError:
> > org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
> > at
> > org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native
> > Method) at
> > org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
> > at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
> > at
> > org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:292)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:879)
> > at
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:455)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:511)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:670)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:655)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1304)
> > at
> > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:975)
> > at
> > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:856)
> > at
> > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
> > at
> > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:640)
> > at
> > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest.<clinit>(HdfsFileProviderTest.java:95)
> >
> > Greetings
> > Bernd
> >
> > PS - in case you want to follow the VFS progress:
> > * https://issues.apache.org/jira/browse/VFS-530 - tracks 2.4
> > migration
> > * https://issues.apache.org/jira/browse/VFS-529 - the broken
> > windows tests
> > * https://issues.apache.org/jira/browse/VFS-526 - failed enable of
> > win tests
> >
> >
>
Re: MiniDFSCluster on Windows unfortunatelly requires native
toolchain for automated tests
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hello Bernd,
This is my first time looking at Commons VFS. It looks like a very useful
project, and I'm sorry to hear that we've caused some grief for your tests.
I can provide some background on these changes. Traditionally, 1.2 and
earlier versions didn't officially support Windows beyond a best effort to
keep some approximation of the functionality running on developer desktops.
As we worked on enabling real production deployments on Windows, we
discovered multiple bugs caused by those approximations.
In some cases, the Cygwin and GnuWin32 tools had incomplete implementations
or bugs. Examples of the buggy areas include finding the groups for a user
and process management. Basically, anything in
org.apache.hadoop.util.Shell was problematic. For this category of bugs,
the solution was winutils.exe, which delegates straight to the necessary
Windows API calls.
Other problems arose from bugs or missing functionality in the Windows JDK
implementation. For example, the canRead/canWrite methods in java.io.File
don't work correctly on Windows. (This is the code path shown in your
stack trace.) For this category of bugs, the solution was hadoop.dll,
which is a JNI layer that delegates to the Windows API.
Without these workarounds, production deployment on Windows wouldn't have
been feasible. There would have been too many dangerous bugs.
Unfortunately, we have a weakness right now in that the Windows native
build artifacts aren't bundled into the Apache releases. We also don't
have a fallback mode that works without the native code. (Even if there
was such a fallback mode, it would be very dangerous to run in this mode in
production, so I'd want to treat it as a test-only setting.)
I expect you'll run into a lot of challenges while fully implementing a
Java-only fallback. I'd like to help by checking out the Commons VFS code
and experimenting a bit. I'll try to put some time into this next week and
then comment on the VFS jira issues with my findings. Does this sound good?
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Tue, Jun 17, 2014 at 7:46 PM, Bernd Eckenfels <ec...@zusammenkunft.net>
wrote:
> Hello,
>
> in the Commons VFS project we have a Provider for HDFS. This
> Provider is unit-tested with the help of the excelent MiniDFSCluster
> component.
>
> A while back I enabled automated testing on Windows (cause it worked
> for me), but I did not realize it only works with cygwin (namely
> ls.exe) in the Path. This was some 1.2 version and it used the ls
> command to read the owner.
>
> We currently switch the dependency to 2.4.0 and had the hope that it
> better (i.e.
> without binary dependencies) works in this scenario - as we want to run
> the maven tests without pre-installing a tool chain on Windows.
>
> However it got actually worse, it warns about a missing winutils.exe
> and it requires the hadoop.dll. The first location where it fails is
> testing writeability for the name node. And unlike all the comments in
> FileUtil/NativeIO it does not cleanly fall back to java portable
> methods.
>
> I think the access/access0 method can easily be fixed, but before I work
> on this patch, does it make sense to go in this direction or will I have to
> fix some other, more complicated platform dependencies. My goal would be to
> start MiniDFSCluster inside surefire from maven without the need for
> complicated external toolchain setup.
>
> Is actually anybody caring about making this possible?
>
> org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest Time
> elapsed: 2.291 sec <<< ERROR!
> java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
> at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native
> Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
> at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
> at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:292)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:879)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:455)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:511)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:670)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:655)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1304)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:975)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:856)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:640)
> at
> org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest.<clinit>(HdfsFileProviderTest.java:95)
>
> Greetings
> Bernd
>
> PS - in case you want to follow the VFS progress:
> * https://issues.apache.org/jira/browse/VFS-530 - tracks 2.4 migration
> * https://issues.apache.org/jira/browse/VFS-529 - the broken windows tests
> * https://issues.apache.org/jira/browse/VFS-526 - failed enable of win
> tests
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.