You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "NEKRASSOV, ALEXEI" <an...@att.com> on 2018/07/10 14:28:51 UTC
'mvn verify' fails at flink-hadoop-fs
I'm running 'mvn clean verify' on Windows with no Hadoop libraries installed, and the build fails (see below).
What's the solution? Is there a switch to skip Hadoop-related tests?
Or I need to install Hadoop libraries?
Thanks,
Alex
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.726 sec <<< FAILURE! - in org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest
org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest Time elapsed: 1.726 sec <<< ERROR!
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:293)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:996)
at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:867)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:374)
at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
at org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBehaviorTest.java:65)
Running org.apache.flink.runtime.fs.hdfs.HdfsKindTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.081 sec - in org.apache.flink.runtime.fs.hdfs.HdfsKindTest
Running org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec - in org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
Results :
Tests in error:
HdfsBehaviorTest.createHDFS:65 ▒ UnsatisfiedLink org.apache.hadoop.io.nativeio...
Tests run: 24, Failures: 0, Errors: 1, Skipped: 1
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] force-shading ...................................... SUCCESS [ 2.335 s]
[INFO] flink .............................................. SUCCESS [ 29.794 s]
[INFO] flink-annotations .................................. SUCCESS [ 2.198 s]
[INFO] flink-shaded-hadoop ................................ SUCCESS [ 0.226 s]
[INFO] flink-shaded-hadoop2 ............................... SUCCESS [ 11.015 s]
[INFO] flink-shaded-hadoop2-uber .......................... SUCCESS [ 16.343 s]
[INFO] flink-shaded-yarn-tests ............................ SUCCESS [ 13.653 s]
[INFO] flink-shaded-curator ............................... SUCCESS [ 1.386 s]
[INFO] flink-test-utils-parent ............................ SUCCESS [ 0.191 s]
[INFO] flink-test-utils-junit ............................. SUCCESS [ 3.318 s]
[INFO] flink-metrics ...................................... SUCCESS [ 0.212 s]
[INFO] flink-metrics-core ................................. SUCCESS [ 3.502 s]
[INFO] flink-core ......................................... SUCCESS [01:30 min]
[INFO] flink-java ......................................... SUCCESS [01:31 min]
[INFO] flink-queryable-state .............................. SUCCESS [ 0.186 s]
[INFO] flink-queryable-state-client-java .................. SUCCESS [ 4.099 s]
[INFO] flink-filesystems .................................. SUCCESS [ 0.198 s]
[INFO] flink-hadoop-fs .................................... FAILURE [ 8.672 s]
RE: 'mvn verify' fails at flink-hadoop-fs
Posted by "NEKRASSOV, ALEXEI" <an...@att.com>.
I now see that the test that fails for me already has this check. But I don't think it is done correctly.
https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/HdfsBehaviorTest.java has verifyOS() but a failed assumption in that method doesn't stop us from executing createHDFS() where there is no check, and thus the test fails.
I think we need to remove verifyOS(), and move the check directly to createHDFS() instead.
Similar problem may exist in:
flink-connectors\flink-connector-filesystem\src\test\java\org\apache\flink\streaming\connectors\fs\bucketing\BucketingSinkMigrationTest.java
flink-fs-tests\src\test\java\org\apache\flink\hdfstests\ContinuousFileProcessingMigrationTest.java
flink-fs-tests\src\test\java\org\apache\flink\hdfstests\HDFSTest.java
-----Original Message-----
From: Chesnay Schepler [mailto:chesnay@apache.org]
Sent: Tuesday, July 10, 2018 3:10 PM
To: dev@flink.apache.org; NEKRASSOV, ALEXEI <an...@att.com>
Subject: Re: 'mvn verify' fails at flink-hadoop-fs
That flat-out disables all tests in the module, even those that could run on Windows.
We commonly add an OS check to respective tests that skip the tests, with an "Assume.assumeTrue(os!=windows)" statement in a "@BeforeClass"
method.
On 10.07.2018 21:00, NEKRASSOV, ALEXEI wrote:
> I added lines below to flink-hadoop-fs/pom.xml, and that allowed me to turn off the tests that were failing for me.
> Do we want to add this change to master?
> If so, do I need to document this new switch somewhere?
>
> (
> the build then hang for me at flink-runtime, but that's a different
> issue Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 25.969 sec - in
> org.apache.flink.runtime.taskmanager.TaskManagerRegistrationTest
> )
>
> <build>
> <plugins>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <configuration>
> <skipTests>${skipHdfsTests}</skipTests>
> </configuration>
> </plugin>
> </plugins>
> </build>
>
> -----Original Message-----
> From: Chesnay Schepler [mailto:chesnay@apache.org]
> Sent: Tuesday, July 10, 2018 10:36 AM
> To: dev@flink.apache.org; NEKRASSOV, ALEXEI <an...@att.com>
> Subject: Re: 'mvn verify' fails at flink-hadoop-fs
>
> There's currently no workaround except going in and manually disabling them.
>
> On 10.07.2018 16:32, Chesnay Schepler wrote:
>> Generally, any test that uses HDFS will fail on Windows. We've
>> disabled most of them, but some slip through from time to time.
>>
>> Note that we do not provide any guarantees for all tests passing on
>> Windows.
>>
>> On 10.07.2018 16:28, NEKRASSOV, ALEXEI wrote:
>>> I'm running 'mvn clean verify' on Windows with no Hadoop libraries
>>> installed, and the build fails (see below).
>>> What's the solution? Is there a switch to skip Hadoop-related tests?
>>> Or I need to install Hadoop libraries?
>>>
>>> Thanks,
>>> Alex
>>>
>>>
>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
>>> 1.726 sec <<< FAILURE! - in
>>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest
>>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest Time elapsed:
>>> 1.726 sec <<< ERROR!
>>> java.lang.UnsatisfiedLinkError:
>>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
>>> at
>>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
>>> at
>>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
>>> at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
>>> at
>>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:293)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:996)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:867)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:374)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
>>> at
>>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBeh
>>> a
>>> viorTest.java:65)
>>>
>>> Running org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>>> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
>>> 0.081 sec - in org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>>> Running
>>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
>>> 0.017 sec - in
>>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>>>
>>> Results :
>>>
>>> Tests in error:
>>> HdfsBehaviorTest.createHDFS:65 ▒ UnsatisfiedLink
>>> org.apache.hadoop.io.nativeio...
>>>
>>> Tests run: 24, Failures: 0, Errors: 1, Skipped: 1
>>>
>>> [INFO]
>>> --------------------------------------------------------------------
>>> -
>>> ---
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] force-shading ...................................... SUCCESS
>>> [
>>> 2.335 s] [INFO] flink ..............................................
>>> SUCCESS [
>>> 29.794 s]
>>> [INFO] flink-annotations .................................. SUCCESS
>>> [
>>> 2.198 s] [INFO] flink-shaded-hadoop ................................
>>> SUCCESS [ 0.226 s] [INFO] flink-shaded-hadoop2
>>> ............................... SUCCESS [
>>> 11.015 s]
>>> [INFO] flink-shaded-hadoop2-uber .......................... SUCCESS
>>> [
>>> 16.343 s]
>>> [INFO] flink-shaded-yarn-tests ............................ SUCCESS
>>> [
>>> 13.653 s]
>>> [INFO] flink-shaded-curator ............................... SUCCESS
>>> [
>>> 1.386 s] [INFO] flink-test-utils-parent ............................
>>> SUCCESS [ 0.191 s] [INFO] flink-test-utils-junit
>>> ............................. SUCCESS [ 3.318 s] [INFO]
>>> flink-metrics ...................................... SUCCESS [
>>> 0.212 s] [INFO] flink-metrics-core .................................
>>> SUCCESS [ 3.502 s] [INFO] flink-core
>>> ......................................... SUCCESS
>>> [01:30 min]
>>> [INFO] flink-java ......................................... SUCCESS
>>> [01:31 min]
>>> [INFO] flink-queryable-state .............................. SUCCESS
>>> [
>>> 0.186 s] [INFO] flink-queryable-state-client-java ..................
>>> SUCCESS [ 4.099 s] [INFO] flink-filesystems
>>> .................................. SUCCESS [ 0.198 s] [INFO]
>>> flink-hadoop-fs .................................... FAILURE [
>>> 8.672 s]
>>>
>>>
>>>
>>>
>>
>
Re: 'mvn verify' fails at flink-hadoop-fs
Posted by Chesnay Schepler <ch...@apache.org>.
That flat-out disables all tests in the module, even those that could
run on Windows.
We commonly add an OS check to respective tests that skip the tests,
with an "Assume.assumeTrue(os!=windows)" statement in a "@BeforeClass"
method.
On 10.07.2018 21:00, NEKRASSOV, ALEXEI wrote:
> I added lines below to flink-hadoop-fs/pom.xml, and that allowed me to turn off the tests that were failing for me.
> Do we want to add this change to master?
> If so, do I need to document this new switch somewhere?
>
> (
> the build then hang for me at flink-runtime, but that's a different issue
> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.969 sec - in org.apache.flink.runtime.taskmanager.TaskManagerRegistrationTest
> )
>
> <build>
> <plugins>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <configuration>
> <skipTests>${skipHdfsTests}</skipTests>
> </configuration>
> </plugin>
> </plugins>
> </build>
>
> -----Original Message-----
> From: Chesnay Schepler [mailto:chesnay@apache.org]
> Sent: Tuesday, July 10, 2018 10:36 AM
> To: dev@flink.apache.org; NEKRASSOV, ALEXEI <an...@att.com>
> Subject: Re: 'mvn verify' fails at flink-hadoop-fs
>
> There's currently no workaround except going in and manually disabling them.
>
> On 10.07.2018 16:32, Chesnay Schepler wrote:
>> Generally, any test that uses HDFS will fail on Windows. We've
>> disabled most of them, but some slip through from time to time.
>>
>> Note that we do not provide any guarantees for all tests passing on
>> Windows.
>>
>> On 10.07.2018 16:28, NEKRASSOV, ALEXEI wrote:
>>> I'm running 'mvn clean verify' on Windows with no Hadoop libraries
>>> installed, and the build fails (see below).
>>> What's the solution? Is there a switch to skip Hadoop-related tests?
>>> Or I need to install Hadoop libraries?
>>>
>>> Thanks,
>>> Alex
>>>
>>>
>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.726
>>> sec <<< FAILURE! - in
>>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest
>>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest Time elapsed:
>>> 1.726 sec <<< ERROR!
>>> java.lang.UnsatisfiedLinkError:
>>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
>>> at
>>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
>>> at
>>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
>>> at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
>>> at
>>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:293)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:996)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:867)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:374)
>>> at
>>> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
>>> at
>>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBeha
>>> viorTest.java:65)
>>>
>>> Running org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>>> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.081
>>> sec - in org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>>> Running
>>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017
>>> sec - in
>>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>>>
>>> Results :
>>>
>>> Tests in error:
>>> HdfsBehaviorTest.createHDFS:65 ▒ UnsatisfiedLink
>>> org.apache.hadoop.io.nativeio...
>>>
>>> Tests run: 24, Failures: 0, Errors: 1, Skipped: 1
>>>
>>> [INFO]
>>> ---------------------------------------------------------------------
>>> ---
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] force-shading ...................................... SUCCESS [
>>> 2.335 s] [INFO] flink ..............................................
>>> SUCCESS [
>>> 29.794 s]
>>> [INFO] flink-annotations .................................. SUCCESS [
>>> 2.198 s] [INFO] flink-shaded-hadoop ................................
>>> SUCCESS [ 0.226 s] [INFO] flink-shaded-hadoop2
>>> ............................... SUCCESS [
>>> 11.015 s]
>>> [INFO] flink-shaded-hadoop2-uber .......................... SUCCESS [
>>> 16.343 s]
>>> [INFO] flink-shaded-yarn-tests ............................ SUCCESS [
>>> 13.653 s]
>>> [INFO] flink-shaded-curator ............................... SUCCESS [
>>> 1.386 s] [INFO] flink-test-utils-parent ............................
>>> SUCCESS [ 0.191 s] [INFO] flink-test-utils-junit
>>> ............................. SUCCESS [ 3.318 s] [INFO]
>>> flink-metrics ...................................... SUCCESS [ 0.212
>>> s] [INFO] flink-metrics-core .................................
>>> SUCCESS [ 3.502 s] [INFO] flink-core
>>> ......................................... SUCCESS
>>> [01:30 min]
>>> [INFO] flink-java ......................................... SUCCESS
>>> [01:31 min]
>>> [INFO] flink-queryable-state .............................. SUCCESS [
>>> 0.186 s] [INFO] flink-queryable-state-client-java ..................
>>> SUCCESS [ 4.099 s] [INFO] flink-filesystems
>>> .................................. SUCCESS [ 0.198 s] [INFO]
>>> flink-hadoop-fs .................................... FAILURE [ 8.672
>>> s]
>>>
>>>
>>>
>>>
>>
>
RE: 'mvn verify' fails at flink-hadoop-fs
Posted by "NEKRASSOV, ALEXEI" <an...@att.com>.
I added lines below to flink-hadoop-fs/pom.xml, and that allowed me to turn off the tests that were failing for me.
Do we want to add this change to master?
If so, do I need to document this new switch somewhere?
(
the build then hang for me at flink-runtime, but that's a different issue
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.969 sec - in org.apache.flink.runtime.taskmanager.TaskManagerRegistrationTest
)
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skipTests>${skipHdfsTests}</skipTests>
</configuration>
</plugin>
</plugins>
</build>
-----Original Message-----
From: Chesnay Schepler [mailto:chesnay@apache.org]
Sent: Tuesday, July 10, 2018 10:36 AM
To: dev@flink.apache.org; NEKRASSOV, ALEXEI <an...@att.com>
Subject: Re: 'mvn verify' fails at flink-hadoop-fs
There's currently no workaround except going in and manually disabling them.
On 10.07.2018 16:32, Chesnay Schepler wrote:
> Generally, any test that uses HDFS will fail on Windows. We've
> disabled most of them, but some slip through from time to time.
>
> Note that we do not provide any guarantees for all tests passing on
> Windows.
>
> On 10.07.2018 16:28, NEKRASSOV, ALEXEI wrote:
>> I'm running 'mvn clean verify' on Windows with no Hadoop libraries
>> installed, and the build fails (see below).
>> What's the solution? Is there a switch to skip Hadoop-related tests?
>> Or I need to install Hadoop libraries?
>>
>> Thanks,
>> Alex
>>
>>
>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.726
>> sec <<< FAILURE! - in
>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest
>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest Time elapsed:
>> 1.726 sec <<< ERROR!
>> java.lang.UnsatisfiedLinkError:
>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
>> at
>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
>> at
>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
>> at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
>> at
>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:293)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:996)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:867)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:374)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
>> at
>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBeha
>> viorTest.java:65)
>>
>> Running org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.081
>> sec - in org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>> Running
>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017
>> sec - in
>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>>
>> Results :
>>
>> Tests in error:
>> HdfsBehaviorTest.createHDFS:65 ▒ UnsatisfiedLink
>> org.apache.hadoop.io.nativeio...
>>
>> Tests run: 24, Failures: 0, Errors: 1, Skipped: 1
>>
>> [INFO]
>> ---------------------------------------------------------------------
>> ---
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] force-shading ...................................... SUCCESS [
>> 2.335 s] [INFO] flink ..............................................
>> SUCCESS [
>> 29.794 s]
>> [INFO] flink-annotations .................................. SUCCESS [
>> 2.198 s] [INFO] flink-shaded-hadoop ................................
>> SUCCESS [ 0.226 s] [INFO] flink-shaded-hadoop2
>> ............................... SUCCESS [
>> 11.015 s]
>> [INFO] flink-shaded-hadoop2-uber .......................... SUCCESS [
>> 16.343 s]
>> [INFO] flink-shaded-yarn-tests ............................ SUCCESS [
>> 13.653 s]
>> [INFO] flink-shaded-curator ............................... SUCCESS [
>> 1.386 s] [INFO] flink-test-utils-parent ............................
>> SUCCESS [ 0.191 s] [INFO] flink-test-utils-junit
>> ............................. SUCCESS [ 3.318 s] [INFO]
>> flink-metrics ...................................... SUCCESS [ 0.212
>> s] [INFO] flink-metrics-core .................................
>> SUCCESS [ 3.502 s] [INFO] flink-core
>> ......................................... SUCCESS
>> [01:30 min]
>> [INFO] flink-java ......................................... SUCCESS
>> [01:31 min]
>> [INFO] flink-queryable-state .............................. SUCCESS [
>> 0.186 s] [INFO] flink-queryable-state-client-java ..................
>> SUCCESS [ 4.099 s] [INFO] flink-filesystems
>> .................................. SUCCESS [ 0.198 s] [INFO]
>> flink-hadoop-fs .................................... FAILURE [ 8.672
>> s]
>>
>>
>>
>>
>
>
Re: 'mvn verify' fails at flink-hadoop-fs
Posted by Chesnay Schepler <ch...@apache.org>.
There's currently no workaround except going in and manually disabling them.
On 10.07.2018 16:32, Chesnay Schepler wrote:
> Generally, any test that uses HDFS will fail on Windows. We've
> disabled most of them, but some slip through from time to time.
>
> Note that we do not provide any guarantees for all tests passing on
> Windows.
>
> On 10.07.2018 16:28, NEKRASSOV, ALEXEI wrote:
>> I'm running 'mvn clean verify' on Windows with no Hadoop libraries
>> installed, and the build fails (see below).
>> What's the solution? Is there a switch to skip Hadoop-related tests?
>> Or I need to install Hadoop libraries?
>>
>> Thanks,
>> Alex
>>
>>
>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.726
>> sec <<< FAILURE! - in org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest
>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest Time elapsed:
>> 1.726 sec <<< ERROR!
>> java.lang.UnsatisfiedLinkError:
>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
>> at
>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
>> at
>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
>> at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
>> at
>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:293)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:996)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:867)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:374)
>> at
>> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
>> at
>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBehaviorTest.java:65)
>>
>> Running org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.081
>> sec - in org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>> Running
>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017
>> sec - in
>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>>
>> Results :
>>
>> Tests in error:
>> HdfsBehaviorTest.createHDFS:65 ▒ UnsatisfiedLink
>> org.apache.hadoop.io.nativeio...
>>
>> Tests run: 24, Failures: 0, Errors: 1, Skipped: 1
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] force-shading ...................................... SUCCESS
>> [ 2.335 s]
>> [INFO] flink .............................................. SUCCESS [
>> 29.794 s]
>> [INFO] flink-annotations .................................. SUCCESS
>> [ 2.198 s]
>> [INFO] flink-shaded-hadoop ................................ SUCCESS
>> [ 0.226 s]
>> [INFO] flink-shaded-hadoop2 ............................... SUCCESS [
>> 11.015 s]
>> [INFO] flink-shaded-hadoop2-uber .......................... SUCCESS [
>> 16.343 s]
>> [INFO] flink-shaded-yarn-tests ............................ SUCCESS [
>> 13.653 s]
>> [INFO] flink-shaded-curator ............................... SUCCESS
>> [ 1.386 s]
>> [INFO] flink-test-utils-parent ............................ SUCCESS
>> [ 0.191 s]
>> [INFO] flink-test-utils-junit ............................. SUCCESS
>> [ 3.318 s]
>> [INFO] flink-metrics ...................................... SUCCESS
>> [ 0.212 s]
>> [INFO] flink-metrics-core ................................. SUCCESS
>> [ 3.502 s]
>> [INFO] flink-core ......................................... SUCCESS
>> [01:30 min]
>> [INFO] flink-java ......................................... SUCCESS
>> [01:31 min]
>> [INFO] flink-queryable-state .............................. SUCCESS
>> [ 0.186 s]
>> [INFO] flink-queryable-state-client-java .................. SUCCESS
>> [ 4.099 s]
>> [INFO] flink-filesystems .................................. SUCCESS
>> [ 0.198 s]
>> [INFO] flink-hadoop-fs .................................... FAILURE
>> [ 8.672 s]
>>
>>
>>
>>
>
>
Re: 'mvn verify' fails at flink-hadoop-fs
Posted by Chesnay Schepler <ch...@apache.org>.
Generally, any test that uses HDFS will fail on Windows. We've disabled
most of them, but some slip through from time to time.
Note that we do not provide any guarantees for all tests passing on Windows.
On 10.07.2018 16:28, NEKRASSOV, ALEXEI wrote:
> I'm running 'mvn clean verify' on Windows with no Hadoop libraries installed, and the build fails (see below).
> What's the solution? Is there a switch to skip Hadoop-related tests?
> Or I need to install Hadoop libraries?
>
> Thanks,
> Alex
>
>
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.726 sec <<< FAILURE! - in org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest
> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest Time elapsed: 1.726 sec <<< ERROR!
> java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
> at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
> at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
> at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
> at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:293)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
> at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:996)
> at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:867)
> at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
> at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:374)
> at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
> at org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBehaviorTest.java:65)
>
> Running org.apache.flink.runtime.fs.hdfs.HdfsKindTest
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.081 sec - in org.apache.flink.runtime.fs.hdfs.HdfsKindTest
> Running org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec - in org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>
> Results :
>
> Tests in error:
> HdfsBehaviorTest.createHDFS:65 ▒ UnsatisfiedLink org.apache.hadoop.io.nativeio...
>
> Tests run: 24, Failures: 0, Errors: 1, Skipped: 1
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] force-shading ...................................... SUCCESS [ 2.335 s]
> [INFO] flink .............................................. SUCCESS [ 29.794 s]
> [INFO] flink-annotations .................................. SUCCESS [ 2.198 s]
> [INFO] flink-shaded-hadoop ................................ SUCCESS [ 0.226 s]
> [INFO] flink-shaded-hadoop2 ............................... SUCCESS [ 11.015 s]
> [INFO] flink-shaded-hadoop2-uber .......................... SUCCESS [ 16.343 s]
> [INFO] flink-shaded-yarn-tests ............................ SUCCESS [ 13.653 s]
> [INFO] flink-shaded-curator ............................... SUCCESS [ 1.386 s]
> [INFO] flink-test-utils-parent ............................ SUCCESS [ 0.191 s]
> [INFO] flink-test-utils-junit ............................. SUCCESS [ 3.318 s]
> [INFO] flink-metrics ...................................... SUCCESS [ 0.212 s]
> [INFO] flink-metrics-core ................................. SUCCESS [ 3.502 s]
> [INFO] flink-core ......................................... SUCCESS [01:30 min]
> [INFO] flink-java ......................................... SUCCESS [01:31 min]
> [INFO] flink-queryable-state .............................. SUCCESS [ 0.186 s]
> [INFO] flink-queryable-state-client-java .................. SUCCESS [ 4.099 s]
> [INFO] flink-filesystems .................................. SUCCESS [ 0.198 s]
> [INFO] flink-hadoop-fs .................................... FAILURE [ 8.672 s]
>
>
>
>