You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Allen Wittenauer (JIRA)" <ji...@apache.org> on 2016/01/11 23:20:40 UTC
[jira] [Comment Edited] (HADOOP-11505) Various native parts use
bswap incorrectly and unportably
[ https://issues.apache.org/jira/browse/HADOOP-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15092797#comment-15092797 ]
Allen Wittenauer edited comment on HADOOP-11505 at 1/11/16 10:20 PM:
---------------------------------------------------------------------
I'm not sure I agree with [~alanburlison]'s assessment, but I might be missing something. All of the cmake files appear to be building their own config.h files. Plus *zero* of our native builds are failing when we don't build hadoop-common prior to building anything else.
{code}
hadoop-common-project/hadoop-common/src/CMakeLists.txt:configure_file(${CMAKE_SOURCE_DIR}/config.h.cmake ${CMAKE_BINARY_DIR}/config.h)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt:configure_file(${CMAKE_SOURCE_DIR}/config.h.cmake ${CMAKE_BINARY_DIR}/config.h)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt:configure_file(${CMAKE_SOURCE_DIR}/config.h.cmake ${CMAKE_BINARY_DIR}/config.h)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt:configure_file(${CMAKE_SOURCE_DIR}/config.h.cmake ${CMAKE_BINARY_DIR}/config.h)
{code}
Additionally, for lz4, the pom.xml file does the magic to make it work:
{code}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml: <copy file="${basedir}/../../../hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.h"
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml: <copy file="${basedir}/../../../hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c"
{code}
So no, really, this is adding a new compile-time dependency that isn't there.
was (Author: aw):
I'm not sure I agree with [~alanburlison]'s assessment, but I might be missing something. All of the cmake files appear to be building their own config.h files. Plus *zero* of our native builds are failing when we don't build hadoop-common prior to building anything else.
{code}
hadoop-common-project/hadoop-common/src/CMakeLists.txt:configure_file(${CMAKE_SOURCE_DIR}/config.h.cmake ${CMAKE_BINARY_DIR}/config.h)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt:configure_file(${CMAKE_SOURCE_DIR}/config.h.cmake ${CMAKE_BINARY_DIR}/config.h)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt:configure_file(${CMAKE_SOURCE_DIR}/config.h.cmake ${CMAKE_BINARY_DIR}/config.h)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt:configure_file(${CMAKE_SOURCE_DIR}/config.h.cmake ${CMAKE_BINARY_DIR}/config.h)
{code}
Additionally, for lz4, the pom.xml file does the magic to make it work:
{code}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml: <copy file="${basedir}/../../../hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.h"
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml: <copy file="${basedir}/../../../hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c"
{code}
So no, really, this is adding a new dependency that isn't there.
> Various native parts use bswap incorrectly and unportably
> ---------------------------------------------------------
>
> Key: HADOOP-11505
> URL: https://issues.apache.org/jira/browse/HADOOP-11505
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Alan Burlison
> Fix For: 3.0.0
>
> Attachments: HADOOP-11505.001.patch, HADOOP-11505.003.patch, HADOOP-11505.004.patch, HADOOP-11505.005.patch, HADOOP-11505.006.patch, HADOOP-11505.007.patch
>
>
> hadoop-mapreduce-client-nativetask fails to use x86 optimizations in some cases. Also, on some alternate, non-x86, non-ARM architectures the generated code is incorrect. Thanks to Steve Loughran and Edward Nevill for finding this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)