You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "linyihang (Commented) (JIRA)" <ji...@apache.org> on 2011/10/09 10:20:32 UTC
[jira] [Commented] (MAPREDUCE-1270) Hadoop C++ Extention
[ https://issues.apache.org/jira/browse/MAPREDUCE-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123646#comment-13123646 ]
linyihang commented on MAPREDUCE-1270:
--------------------------------------
Hello Mr Yang,
I am new to HCE,when I download the HCE 1.0 and try to build it by "./build.sh",but failed.there is the errors:
"../hadoop_hce_v1/hadoop-0.20.3/../java6/jre/bin/java: 1: Syntax error: "(" unexpected
../hadoop_hce_v1/hadoop-0.20.3/../java6/jre/bin/java: 1: Syntax error: "(" unexpected
"
Then I modify the build.sh by comment the lines below,
" # prepare the java and ant ENV
#export JAVA_HOME=${workdir}/../java6
#export ANT_HOME=${workdir}/../ant
#export PATH=${JAVA_HOME}/bin:${ANT_HOME}/bin:$PATH",for my having install jdk1.6.0_21 and the ant1.8.2;
But there seems to be some errors like
"
[exec] /usr/include/linux/tcp.h:77: error: ‘__u32 __fswab32(__u32)’ cannot appear in a constant-expression
",and never meet "BUIED SUCCESSFULY" as the InstallMenu.pdf showing.My OS is UBUNTU 10.10 ,and GCC is version 4.4.5.
Is there any error I hava make?
Something further,then I try to fix the error as someguy strikes me on Google by replacing "#include <linux/tcp.h>" with "#include<netinet/tcp.h>" in the "../hadoop_hce_v1/hadoop-0.20.3/src/c++/hce/impl/Commo
n/Type.hh",also add "#include<stdint.h>" in the Type.hh. But there takes place to be a lot of mistakes(which I thought) like mistaking "printf("%lld")" by "printf("lld")",and one serious error as follow witch worry me a lot.
the serious error is ,
"
[exec] then mv -f ".deps/CompressionFactory.Tpo" ".deps/CompressionFactory.Po"; else rm -f ".deps/CompressionFactory.Tpo"; exit 1; fi
[exec] In file included from /usr/include/limits.h:153,
[exec] from /usr/lib/gcc/i686-linux-gnu/4.4.5/include-fixed/limits.h:122,
[exec] from /usr/lib/gcc/i686-linux-gnu/4.4.5/include-fixed/syslimits.h:7,
[exec] from /usr/lib/gcc/i686-linux-gnu/4.4.5/include-fixed/limits.h:11,
[exec] from /home/had/文档/HCE/bak/hadoop_hce_v1/hadoop-0.20.3/src/c++/hce/impl/../../../../nativelib/lzo/lzo/lzoconf.h:52,
[exec] from /home/had/文档/HCE/bak/hadoop_hce_v1/hadoop-0.20.3/src/c++/hce/impl/../../../../nativelib/lzo/lzo/lzo1.h:45,
[exec] from /home/had/文档/HCE/bak/hadoop_hce_v1/hadoop-0.20.3/src/c++/hce/impl/Compress/LzoCompressor.hh:23,
[exec] from /home/had/文档/HCE/bak/hadoop_hce_v1/hadoop-0.20.3/src/c++/hce/impl/Compress/LzoCodec.hh:27,
[exec] from /home/had/文档/HCE/bak/hadoop_hce_v1/hadoop-0.20.3/src/c++/hce/impl/Compress/CompressionFactory.cc:23:
[exec] /usr/include/bits/xopen_lim.h:95: error: missing binary operator before token "("
[exec] /usr/include/bits/xopen_lim.h:98: error: missing binary operator before token "("
[exec] /usr/include/bits/xopen_lim.h:122: error: missing binary operator before token "("
[exec] make[1]:正在离开目录 `/home/had/文档/HCE/bak/hadoop_hce_v1/hadoop-0.20.3/build/c++-build/Linux-i386-32/hce/impl/Compress'
[exec] make[1]: *** [CompressionFactory.o] 错误 1
[exec] make: *** [install-recursive] 错误 1
"
How can I fix this error?
> Hadoop C++ Extention
> --------------------
>
> Key: MAPREDUCE-1270
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1270
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.1
> Environment: hadoop linux
> Reporter: Wang Shouyan
> Attachments: HADOOP-HCE-1.0.0.patch, HCE InstallMenu.pdf, HCE Performance Report.pdf, HCE Tutorial.pdf, Overall Design of Hadoop C++ Extension.doc
>
>
> Hadoop C++ extension is an internal project in baidu, We start it for these reasons:
> 1 To provide C++ API. We mostly use Streaming before, and we also try to use PIPES, but we do not find PIPES is more efficient than Streaming. So we
> think a new C++ extention is needed for us.
> 2 Even using PIPES or Streaming, it is hard to control memory of hadoop map/reduce Child JVM.
> 3 It costs so much to read/write/sort TB/PB data by Java. When using PIPES or Streaming, pipe or socket is not efficient to carry so huge data.
> What we want to do:
> 1 We do not use map/reduce Child JVM to do any data processing, which just prepares environment, starts C++ mapper, tells mapper which split it should deal with, and reads report from mapper until that finished. The mapper will read record, ivoke user defined map, to do partition, write spill, combine and merge into file.out. We think these operations can be done by C++ code.
> 2 Reducer is similar to mapper, it was started after sort finished, it read from sorted files, ivoke user difined reduce, and write to user defined record writer.
> 3 We also intend to rewrite shuffle and sort with C++, for efficience and memory control.
> at first, 1 and 2, then 3.
> What's the difference with PIPES:
> 1 Yes, We will reuse most PIPES code.
> 2 And, We should do it more completely, nothing changed in scheduling and management, but everything in execution.
> *UPDATE:*
> Now you can get a test version of HCE from this link http://docs.google.com/leaf?id=0B5xhnqH1558YZjcxZmI0NzEtODczMy00NmZiLWFkNjAtZGM1MjZkMmNkNWFk&hl=zh_CN&pli=1
> This is a full package with all hadoop source code.
> Following document "HCE InstallMenu.pdf" in attachment, you will build and deploy it in your cluster.
> Attachment "HCE Tutorial.pdf" will lead you to write the first HCE program and give other specifications of the interface.
> Attachment "HCE Performance Report.pdf" gives a performance report of HCE compared to Java MapRed and Pipes.
> Any comments are welcomed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira