You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2019/04/01 23:59:03 UTC
[jira] [Comment Edited] (HDFS-14394) Add -std=c99 / -std=gnu99 to
libhdfs compile flags
[ https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807296#comment-16807296 ]
Eric Yang edited comment on HDFS-14394 at 4/1/19 11:58 PM:
-----------------------------------------------------------
[~stakiar] It seems a little strange that it failed the first time, then it worked the second time. The last section of the maven output looks like this:
{code:java}
[exec] /home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs.h:124: Failure
[exec] Expected: (nullptr) != (clusterInfo), actual: 8-byte object <00-00 00-00 00-00 00-00> vs NULL
[exec] #
[exec] # A fatal error has been detected by the Java Runtime Environment:
[exec] #
[exec] # SIGSEGV (0xb) at pc=0x0000000000783b7e, pid=28414, tid=0x00007efc91ac38c0
[exec] #
[exec] # JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build 1.8.0_151-b12)
[exec] # Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)
[exec] # Problematic frame:
[exec] nmdCreate: Builder#build error:
[exec] RuntimeException: Although a UNIX domain socket path is configured as /tmp/native_mini_dfs.sock.28414.846930886, we cannot start a localDataXceiverServer because libhadoop cannot be loaded.java.lang.RuntimeException: Although a UNIX domain socket path is configured as /tmp/native_mini_dfs.sock.28414.846930886, we cannot start a localDataXceiverServer because libhadoop cannot be loaded.
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1209)
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:1178)
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1433)
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:509)
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2827)
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2733)
[exec] at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1697)
[exec] at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:913)
[exec] at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:520)
[exec] at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:479)
[exec] # C [hdfs_ext_hdfspp_test_shim_static+0x383b7e]
[exec] #
[exec] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
[exec] #
[exec] # An error report file with more information is saved as:
[exec] # /home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/tests/hs_err_pid28414.log
[exec] #
[exec] # If you would like to submit a bug report, please visit:
[exec] # http://bugreport.java.com/bugreport/crash.jsp
[exec] # The crash happened outside the Java Virtual Machine in native code.
[exec] # See problematic frame for where to report the bug.
[exec] #
[exec]
[exec]
[exec] 85% tests passed, 6 tests failed out of 40
[exec]
[exec] Total Test time (real) = 96.60 sec
[exec]
[exec] The following tests FAILED:
[exec] 3 - test_test_libhdfs_zerocopy_hdfs_static (Failed)
[exec] 36 - test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static (OTHER_FAULT)
[exec] 37 - libhdfs_mini_stress_valgrind_hdfspp_test_static (Failed)
[exec] 38 - memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static (Failed)
[exec] 39 - test_libhdfs_mini_stress_hdfspp_test_shim_static (Failed)
[exec] 40 - test_hdfs_ext_hdfspp_test_shim_static (OTHER_FAULT)
[exec] Errors while running CTest{code}
It was not able to find libhadoop native library, even though I did ran with mvn clean install -Pnative in hadoop-common-project follow by the same compile command in hadoop-hdfs-native-client project.
There are some difference between [https://stackoverflow.com/questions/10461331/what-are-the-incompatible-differences-between-c99-and-c11|c++11 and c99]. I don't know if this will create instability in libhdfspp to use c99. Gcc 4.5+ can enforce c99 standard by passing -std=c99 -pedantic-errors -fextended-identifiers instead of gnu99 to prevent gnu features to be included.
was (Author: eyang):
[~stakiar] It seems a little strange that it failed the first time, then it worked the second time. The last section of the maven output looks like this:
{code:java}
[exec] /home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs.h:124: Failure
[exec] Expected: (nullptr) != (clusterInfo), actual: 8-byte object <00-00 00-00 00-00 00-00> vs NULL
[exec] #
[exec] # A fatal error has been detected by the Java Runtime Environment:
[exec] #
[exec] # SIGSEGV (0xb) at pc=0x0000000000783b7e, pid=28414, tid=0x00007efc91ac38c0
[exec] #
[exec] # JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build 1.8.0_151-b12)
[exec] # Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)
[exec] # Problematic frame:
[exec] nmdCreate: Builder#build error:
[exec] RuntimeException: Although a UNIX domain socket path is configured as /tmp/native_mini_dfs.sock.28414.846930886, we cannot start a localDataXceiverServer because libhadoop cannot be loaded.java.lang.RuntimeException: Although a UNIX domain socket path is configured as /tmp/native_mini_dfs.sock.28414.846930886, we cannot start a localDataXceiverServer because libhadoop cannot be loaded.
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1209)
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:1178)
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1433)
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:509)
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2827)
[exec] at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2733)
[exec] at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1697)
[exec] at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:913)
[exec] at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:520)
[exec] at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:479)
[exec] # C [hdfs_ext_hdfspp_test_shim_static+0x383b7e]
[exec] #
[exec] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
[exec] #
[exec] # An error report file with more information is saved as:
[exec] # /home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/tests/hs_err_pid28414.log
[exec] #
[exec] # If you would like to submit a bug report, please visit:
[exec] # http://bugreport.java.com/bugreport/crash.jsp
[exec] # The crash happened outside the Java Virtual Machine in native code.
[exec] # See problematic frame for where to report the bug.
[exec] #
[exec]
[exec]
[exec] 85% tests passed, 6 tests failed out of 40
[exec]
[exec] Total Test time (real) = 96.60 sec
[exec]
[exec] The following tests FAILED:
[exec] 3 - test_test_libhdfs_zerocopy_hdfs_static (Failed)
[exec] 36 - test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static (OTHER_FAULT)
[exec] 37 - libhdfs_mini_stress_valgrind_hdfspp_test_static (Failed)
[exec] 38 - memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static (Failed)
[exec] 39 - test_libhdfs_mini_stress_hdfspp_test_shim_static (Failed)
[exec] 40 - test_hdfs_ext_hdfspp_test_shim_static (OTHER_FAULT)
[exec] Errors while running CTest{code}
It was not able to find libhadoop native library, even though I did ran with mvn clean install -Pnative in hadoop-common-project follow by the same compile command in hadoop-hdfs-native-client project.
> Add -std=c99 / -std=gnu99 to libhdfs compile flags
> --------------------------------------------------
>
> Key: HDFS-14394
> URL: https://issues.apache.org/jira/browse/HDFS-14394
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs-client, libhdfs, native
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HDFS-14394.001.patch
>
>
> libhdfs compilation currently does not enforce a minimum required C version. As of today, the libhdfs build on Hadoop QA works, but when built on a machine with an outdated gcc / cc version where C89 is the default, compilation fails due to errors such as:
> {code}
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5: error: ‘for’ loop initial declarations are only allowed in C99 mode
> for (int i = 0; i < numCachedClasses; i++) {
> ^
> /build/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:106:5: note: use option -std=c99 or -std=gnu99 to compile your code
> {code}
> We should add the -std=c99 / -std=gnu99 flags to libhdfs compilation so that we can enforce C99 as the minimum required version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org