You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Craig Macdonald (JIRA)" <ji...@apache.org> on 2008/05/04 00:38:55 UTC
[jira] Created: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
libhdfs: always builds 32bit, even when x86_64 Java used
--------------------------------------------------------
Key: HADOOP-3344
URL: https://issues.apache.org/jira/browse/HADOOP-3344
Project: Hadoop Core
Issue Type: Bug
Components: libhdfs
Environment: x86_64 linux, x86_64 Java installed
Reporter: Craig Macdonald
The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
The relevant lines are:
LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
$OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
{noformat}
[exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
[exec] /usr/bin/ld: cannot find -ljvm
[exec] collect2: ld returned 1 exit status
[exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
{noformat}
The solution should be to specify -m32 or -m64 depending on the os.arch detected.
There are 3 cases to check:
* 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-3344:
------------------------------------
Attachment: HADOOP-3344.v0.patch
Ok, this is first attempt at an autotools build system.
Notes/issues:
* It doesn't yet work. I havent figured out how to get .so shared files built, so building the tests fail
* The autoconf macros came from Apache Commons, Daemon. Hope this is OK?
* I still need to test on other linux architectures. Will need other volunteers for alternative platforms.
* Requires autoconf 2.61, for the AC_TYPE_INT64_T and similar macros
To build configure & Makefile, do:
{noformat}
autoreconf; libtoolize && aclocal -I ../utils/m4/ && automake -a --foreign && autoconf
{noformat}
For normal building:
{noformat}
./configure && make
{noformat}
Can anyone provide assistance in completing this?
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Attachments: HADOOP-3344.v0.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3344) libhdfs: always builds
32bit, even when x86_64 Java used
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594750#action_12594750 ]
craigm edited comment on HADOOP-3344 at 5/6/08 6:03 PM:
-----------------------------------------------------------------
Ok, this is first attempt at an autotools build system.
Notes/issues:
* It doesn't yet work. I havent figured out how to get .so shared files built, so building the tests fail
* The Java-related autoconf macros came from Apache Commons, Daemon. Hope this is OK?
* I still need to test on other linux architectures. Will need other volunteers for alternative platforms.
* Requires autoconf 2.61, for the AC_TYPE_INT64_T and similar macros
To build configure & Makefile, do:
{noformat}
autoreconf; libtoolize && aclocal -I ../utils/m4/ && automake -a --foreign && autoconf
{noformat}
For normal building:
{noformat}
./configure && make
{noformat}
Can anyone provide assistance in completing this?
was (Author: craigm):
Ok, this is first attempt at an autotools build system.
Notes/issues:
* It doesn't yet work. I havent figured out how to get .so shared files built, so building the tests fail
* The autoconf macros came from Apache Commons, Daemon. Hope this is OK?
* I still need to test on other linux architectures. Will need other volunteers for alternative platforms.
* Requires autoconf 2.61, for the AC_TYPE_INT64_T and similar macros
To build configure & Makefile, do:
{noformat}
autoreconf; libtoolize && aclocal -I ../utils/m4/ && automake -a --foreign && autoconf
{noformat}
For normal building:
{noformat}
./configure && make
{noformat}
Can anyone provide assistance in completing this?
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Attachments: HADOOP-3344.v0.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645170#action_12645170 ]
Craig Macdonald commented on HADOOP-3344:
-----------------------------------------
I have tested on Mac OS X 32bit powerpc 10.4. Will test on Linux in due course (10.4 Mac OS X is now unsupported platform for trunk, as Java 6 is not provided by default).
Some minor comments:
1. Mac OS X does not have error.h, so my compile fails. hdfsJniHelper.c includes error.h, added by HADOOP-3549. As far as I can see, it is superfluous. At least on Linux, there are no functions in error.h which hdfsJniHelper.c uses. Should i reopen HADOOP-3549, or file a new issue? Note that error.h should not be confused with the standard errno.h
2. Will create-c++-configure be called for every compile? It customary to include the generated configure/Makefile, as people compiling need not have autoconf/automake.
3. (related to 2) Your patch includes configure/Makefile etc, which is great for testing the patch atm, but the version committed should NOT include these.
Craig
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594715#action_12594715 ]
Doug Cutting commented on HADOOP-3344:
--------------------------------------
Craig: it doesn't sound like anyone is yet working on an autoconf build for libhdfs, but, yes, this would be a welcome contribution for Hadoop 0.18.
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-3344:
---------------------------------------
Status: In Progress (was: Patch Available)
Looks like patch requires some merging , as there are some conflicts with the ivy porting
resubmitting the patch
-giri
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-3344:
---------------------------------------
Release Note:
To build libhdfs use the following command.
ant compile -Dcompile.c++=true -Dlibhdfs=true
But make sure you have "autoconf-2.61" installed
By using libhdfs=true flag we build libhdfs with other c++components.
and the resulting .so file will be installed in c++/<os_osarch_jvmdatamodel>/lib directory
This patch addresses all the three scenarios
*32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
was:
To build libhdfs use the following command.
ant compile -Dcompile.c++=true -Dlibhdfs=true
By using libhdfs=true flag we build libhdfs with other c++components.
This patch addresses all the three scenarios
*32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Fix For: 0.20.0
>
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3344) libhdfs: always builds
32bit, even when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649262#action_12649262 ]
gkesavan edited comment on HADOOP-3344 at 11/19/08 9:04 PM:
----------------------------------------------------------------------
Craig,
Any update on testing with linux? Based on that I can make the patch available..
thanks,
Giri
was (Author: gkesavan):
Criag,
Any update on testing with linux? Based on that I can make the patch available..
thanks,
Giri
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595725#action_12595725 ]
Doug Cutting commented on HADOOP-3344:
--------------------------------------
> Can anyone provide assistance in completing this?
Sorry, I have no experience with autotools. Anyone else?
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Attachments: HADOOP-3344.v0.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3344) libhdfs: always builds
32bit, even when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644201#action_12644201 ]
gkesavan edited comment on HADOOP-3344 at 11/3/08 8:41 PM:
---------------------------------------------------------------------
Here we have the v2 version of the patch available which as well *requires autoconf-2.61*. This is an improved version over v1 submited by Craig. Many thanks to Craig.
I 've used a small piece of java code along with the m4 macros to detect the jvm arch.
{code:title=getArch.java|borderStyle=solid}
class getArch {
public static void main(String []args) {
System.out.println(System.getProperty("sun.arch.data.model", "32"));
}
}
{code}
If somebody can suggest me of a better way I would be more than happy to implement it
This patch addresses all the three scenarios
* 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
{noformat}
To Build libhdfs.so use ant compile-c++-libhdfs -Dcompile.c++=true
To Test libhdfs.so use ant test-c++-libhdfs -Dcompile.c++=true
{noformat}
I have tested this patch on amd64 with 32 bit and 64bit jvm.
Please help me by testing in other platforms as necessary and let me know your comments.
Thanks
was (Author: gkesavan):
Here we have the v2 version of the patch available which as well *requires autoconf-2.61*. This is an improved version over v1 submited by Craig. Many thanks to Craig.
I 've used a small piece of java code along with the m4 macros to detect the jvm arch.
class getArch {
public static void main(String []args) {
System.out.println(System.getProperty("sun.arch.data.model", "32"));
}
}
If somebody can suggest me of a better way I would be more than happy to implement it
This patch addresses all the three scenarios
* 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
{quote}
To *Build* libhdfs.so use *ant compile-c++-libhdfs -Dcompile.c++=true*
To *Test* libhdfs.so use *ant test-c++-libhdfs -Dcompile.c++=true*
{quote}
I have tested this patch on amd64 with 32 bit and 64bit jvm.
Please help me by testing in other platforms as necessary and let me know your comments.
Thanks
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3344) libhdfs: always builds
32bit, even when x86_64 Java used
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594557#action_12594557 ]
craigm edited comment on HADOOP-3344 at 5/6/08 7:57 AM:
-----------------------------------------------------------------
This was an issue for me for testing fuse-dfs - FUSE is installed to match the kernel, so on a 64bit kernel, I have to use 64bit java and 64bit libhdfs. If an autoconf build system would be ready for 0.18 then perhaps we should try to move to one?
was (Author: craigm):
This was an issue for me for testing fuse-dfs - FUSE is installed to match the kernel, so on a 64bit kernel, I have to use 64bit java and 64bit libhdfs. If an autoconf build system would be ready for 0.18?
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald reassigned HADOOP-3344:
---------------------------------------
Assignee: Giridharan Kesavan
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344.v0.patch, HADOOP-3344.v1.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628925#action_12628925 ]
Pete Wyckoff commented on HADOOP-3344:
--------------------------------------
bq. What are the disadvantages to adding -fPIC (a) for amd64 only, (b) for all platforms?
Just the extra indirection required for being independent. I guess on another platform if you are going static it could cost something although not much - especially here/
I had to remove the -shared from the configure.ac, add -fPIC to the CPPFLAGS and also weirdly add hdfs_read_LDADD = libhdfs.la
for each of read, write and test. I thought it would be automatically added. other than that, everything else works for me on amd64.
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Attachments: HADOOP-3344.v0.patch, HADOOP-3344.v1.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-3344:
---------------------------------------
Attachment: HADOOP-3344.patch
With this new patch c++-create-configure target is added with the libhdfs flag.
By doing so it doesn't touch the libhdfs 's configure/compile/ or test target. (unless the flags are set
ie -Dcompile.c++=true & -Dlibhdfs=true)
-Giri
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3344:
------------------------------------
Release Note: Changed build procedure for libhdfs to build correctly for different platforms. Build instructions are in the Jira item. (was: Changed build procedure for libhdfs to always make the 32-bit version. Build instructions are in the Jira item.)
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
Thanks, Craig, for the correction!
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Fix For: 0.20.0
>
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-3344:
--------------------------------
Component/s: build
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644421#action_12644421 ]
Arun C Murthy commented on HADOOP-3344:
---------------------------------------
bq. I 've used a small piece of java code along with the m4 macros to detect the jvm arch.
Giridharan, you can pass the requisite java properties (sun.arch.data.model etc.) straight from build.xml without adding getArch. Please take a look at the compile-core-native target in hadoop/trunk/build.xml, we use that for compiling the native hadoop compression libraries (libhadoop.so)
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656250#action_12656250 ]
Hadoop QA commented on HADOOP-3344:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12395943/HADOOP-3344.patch
against trunk revision 726129.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3734/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3734/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3734/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3734/console
This message is automatically generated.
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594557#action_12594557 ]
Craig Macdonald commented on HADOOP-3344:
-----------------------------------------
This was an issue for me for testing fuse-dfs - FUSE is installed to match the kernel, so on a 64bit kernel, I have to use 64bit java and 64bit libhdfs. If an autoconf build system would be ready for 0.18?
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649262#action_12649262 ]
Giridharan Kesavan commented on HADOOP-3344:
--------------------------------------------
Criag,
Any update on testing with linux? Based on that I can make the patch available..
thanks,
Giri
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645174#action_12645174 ]
Giridharan Kesavan commented on HADOOP-3344:
--------------------------------------------
Thanks for your comments Craig.
{quote}
2. Will create-c++-configure be called for every compile? It customary to include the generated configure/Makefile, as people compiling need not have autoconf/automake.
{quote}
Nope - It doesn't gets called.
During the compilation we just call the configure and then the make install.
{quote}
3. (related to 2) Your patch includes configure/Makefile etc, which is great for testing the patch atm, but the version committed should NOT include these.
{quote}
I did "make distclean" before creating this patch. I understand this patch has an empty Makefile which we have to delete after submitting the patch to svn.
Please correct me if my understanding is wrong anywhere.
Thanks
Giri
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3344:
------------------------------------
Release Note: Changed build procedure for libhdfs to always make the 32-bit version. Build instructions are in the Jira item. (was: Changed build procedure for libhdfs to always make the 32-bit version. Build instructions are in the Jira item.
To build libhdfs use the following command.
ant compile -Dcompile.c++=true -Dlibhdfs=true
But make sure you have "autoconf-2.61" installed
By using libhdfs=true flag we build libhdfs with other c++components.
and the resulting .so file will be installed in c++/<os_osarch_jvmdatamodel>/lib directory
This patch addresses all the three scenarios
*32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64 )
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Fix For: 0.20.0
>
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3344) libhdfs: always builds
32bit, even when x86_64 Java used
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628916#action_12628916 ]
wyckoff edited comment on HADOOP-3344 at 9/6/08 3:15 PM:
--------------------------------------------------------------
I'm getting this error ---
relocation against `a local symbol' can not be used when making a shared object; recompile with -fPIC
This seems to have been a problem for others on amd64 too, but I don't know what the fix is other than adding -fPIC to the library file build, but that has bad implications on non amd64 platforms.
was (Author: wyckoff):
I tried using this on an amd64 machine and I have 2 problems. One is I only have autoconf 2.59 - is 2.61 really needed? When using with 2.59 I get the following error:
relocation against `a local symbol' can not be used when making a shared object; recompile with -fPIC
This seems to have been a problem for others on amd64 too, but I don't know what the fix is other than adding -fPIC to the library file build, but that has bad implications on non amd64 platforms.
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Attachments: HADOOP-3344.v0.patch, HADOOP-3344.v1.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656915#action_12656915 ]
Nigel Daley commented on HADOOP-3344:
-------------------------------------
Incompatible due to:
1) autconf-2.61 required to compile now
2) location of libhdfs is now c++/<os_osarch_jvmdatamodel>/lib
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Fix For: 0.20.0
>
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Macdonald updated HADOOP-3344:
------------------------------------
Attachment: HADOOP-3344.v1.patch
This patch is an improved patch - libhdfs now builds OK, but doesn't build as a .so file. Added doc and test targets, and updated build.xml to call configure.
Will test on 64bit platforms and improve. Work in progress.
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Attachments: HADOOP-3344.v0.patch, HADOOP-3344.v1.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-3344:
---------------------------------------
Release Note:
To build libhdfs use the following command.
ant compile -Dcompile.c++=true -Dlibhdfs=true
By using libhdfs=true flag we build libhdfs with other c++components.
This patch addresses all the three scenarios
*32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
Status: Patch Available (was: Open)
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3344:
------------------------------------
Release Note:
Changed build procedure for libhdfs to always make the 32-bit version. Build instructions are in the Jira item.
To build libhdfs use the following command.
ant compile -Dcompile.c++=true -Dlibhdfs=true
But make sure you have "autoconf-2.61" installed
By using libhdfs=true flag we build libhdfs with other c++components.
and the resulting .so file will be installed in c++/<os_osarch_jvmdatamodel>/lib directory
This patch addresses all the three scenarios
*32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
was:
To build libhdfs use the following command.
ant compile -Dcompile.c++=true -Dlibhdfs=true
But make sure you have "autoconf-2.61" installed
By using libhdfs=true flag we build libhdfs with other c++components.
and the resulting .so file will be installed in c++/<os_osarch_jvmdatamodel>/lib directory
This patch addresses all the three scenarios
*32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
Edit release note for publication.
To build libhdfs use the following command.
ant compile -Dcompile.c++=true -Dlibhdfs=true
But make sure you have "autoconf-2.61" installed
By using libhdfs=true flag we build libhdfs with other c++components.
and the resulting .so file will be installed in c++/<os_osarch_jvmdatamodel>/lib directory
This patch addresses all the three scenarios
*32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
*64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Fix For: 0.20.0
>
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-3344:
---------------------------------------
Attachment: HADOOP-3344-v2.patch
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley resolved HADOOP-3344.
---------------------------------
Resolution: Fixed
Fix Version/s: 0.20.0
Hadoop Flags: [Incompatible change, Reviewed]
I just committed this. Thanks Giri!
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Fix For: 0.20.0
>
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3344) libhdfs: always builds
32bit, even when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644201#action_12644201 ]
gkesavan edited comment on HADOOP-3344 at 10/30/08 8:41 PM:
----------------------------------------------------------------------
Here we have the v2 version of the patch available which as well *requires autoconf-2.61*. This is an improved version over v1 submited by Craig. Many thanks to Craig.
I 've used a small piece of java code along with the m4 macros to detect the jvm arch.
class getArch {
public static void main(String []args) {
System.out.println(System.getProperty("sun.arch.data.model", "32"));
}
}
If somebody can suggest me of a better way I would be more than happy to implement it
This patch addresses all the three scenarios
* 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
{quote}
To *Build* libhdfs.so use *ant compile-c++-libhdfs -Dcompile.c++=true*
To *Test* libhdfs.so use *ant test-c++-libhdfs -Dcompile.c++=true*
{quote}
I have tested this patch on amd64 with 32 bit and 64bit jvm.
Please help me by testing in other platforms as necessary and let me know your comments.
Thanks
was (Author: gkesavan):
Here we have the v2 version of the patch available which as well requires autoconf-2.6. This is an improved version over v1 submited by Craig.
Many thanks to Craig.
I 've used a small piece of java code along with the m4 macros to detect the jvm arch.
class getArch {
public static void main(String []args) {
System.out.println(System.getProperty("sun.arch.data.model", "32"));
}
}
If somebody can suggest me of a better way I would be more than happy to implement it
This patch addresses all the three scenarios
* 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
To Build libhdfs.so ant compile-c++-libhdfs -Dcompile.c++=true
To test libhdfs.so ant test-c++-libhdfs -Dcompile.c++=true
I have tested this patch on amd64 with 32 bit and 64bit jvm.
Please help me by testing in other platforms as necessary and let me know your comments.
Thanks
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344.v0.patch, HADOOP-3344.v1.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644201#action_12644201 ]
Giridharan Kesavan commented on HADOOP-3344:
--------------------------------------------
Here we have the v2 version of the patch available which as well requires autoconf-2.6. This is an improved version over v1 submited by Craig.
Many thanks to Craig.
I 've used a small piece of java code along with the m4 macros to detect the jvm arch.
class getArch {
public static void main(String []args) {
System.out.println(System.getProperty("sun.arch.data.model", "32"));
}
}
If somebody can suggest me of a better way I would be more than happy to implement it
This patch addresses all the three scenarios
* 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
To Build libhdfs.so ant compile-c++-libhdfs -Dcompile.c++=true
To test libhdfs.so ant test-c++-libhdfs -Dcompile.c++=true
I have tested this patch on amd64 with 32 bit and 64bit jvm.
Please help me by testing in other platforms as necessary and let me know your comments.
Thanks
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344.v0.patch, HADOOP-3344.v1.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628916#action_12628916 ]
Pete Wyckoff commented on HADOOP-3344:
--------------------------------------
I tried using this on an amd64 machine and I have 2 problems. One is I only have autoconf 2.59 - is 2.61 really needed? When using with 2.59 I get the following error:
relocation against `a local symbol' can not be used when making a shared object; recompile with -fPIC
This seems to have been a problem for others on amd64 too, but I don't know what the fix is other than adding -fPIC to the library file build, but that has bad implications on non amd64 platforms.
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Attachments: HADOOP-3344.v0.patch, HADOOP-3344.v1.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-3344:
---------------------------------------
Attachment: HADOOP-3344.v3.patch
version v3 has the changes incorporated. v3 doesn't have the java snippet anymore to find the JVM Arch
Thanks to Arun.
Please help me in testing this v3 patch; I 've the patch tested on amd64 with 32 and 64 bit java.
Thanks
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594296#action_12594296 ]
Doug Cutting commented on HADOOP-3344:
--------------------------------------
Owen has argued that libhdfs's build should be re-written to use autoconf.
https://issues.apache.org/jira/browse/HADOOP-1410?focusedCommentId=12497947#action_12497947
We probably need a separate issue for that. Should we fix this independently?
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-3344:
---------------------------------------
Status: Open (was: Patch Available)
The reason for the failure is the missing autoconf-2.61 on the build server.
Im going to resubmitting the patch
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-3344:
---------------------------------------
Attachment: HADOOP-3344.patch
thanks,
giri
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-3344:
---------------------------------------
Attachment: HADOOP-3344.patch
steps to build libhdfs
{noformat}
ant compile -Dcompile.c++=true -Dlibhdfs=true
By using libhdfs=true flag we build libhdfs with other c++components.
{noformat}
This patch addresses all the three scenarios
* 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
* 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
Here is the local test-patch result
[exec]
[exec]
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 10 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec]
[exec]
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================
[exec]
[exec]
Thanks,
Giri
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Craig Macdonald (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628919#action_12628919 ]
Craig Macdonald commented on HADOOP-3344:
-----------------------------------------
fPIC rings a bell on amd64. What are the disadvantages to adding -fPIC (a) for amd64 only, (b) for all platforms?
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Attachments: HADOOP-3344.v0.patch, HADOOP-3344.v1.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656579#action_12656579 ]
Hadoop QA commented on HADOOP-3344:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12396056/HADOOP-3344.patch
against trunk revision 726129.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3743/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3743/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3743/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3743/console
This message is automatically generated.
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3344) libhdfs: always builds 32bit, even
when x86_64 Java used
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-3344:
---------------------------------------
Status: Patch Available (was: Open)
Build doesn't do anything to the libhdfs target's unless the two flags are set. This is implemented b'coz this *libhdfs requires autconf-2.61* and the build server seems to have autoconf-2.59.
Thanks,
Giri
> libhdfs: always builds 32bit, even when x86_64 Java used
> --------------------------------------------------------
>
> Key: HADOOP-3344
> URL: https://issues.apache.org/jira/browse/HADOOP-3344
> Project: Hadoop Core
> Issue Type: Bug
> Components: build, libhdfs
> Environment: x86_64 linux, x86_64 Java installed
> Reporter: Craig Macdonald
> Assignee: Giridharan Kesavan
> Attachments: HADOOP-3344-v2.patch, HADOOP-3344.patch, HADOOP-3344.patch, HADOOP-3344.v0.patch, HADOOP-3344.v1.patch, HADOOP-3344.v3.patch
>
>
> The makefile for libhdfs is hard-coded to compile 32bit libraries. It should perhaps compile dependent on which Java is set.
> The relevant lines are:
> LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m32 -Wl,-x
> CPPFLAGS = -m32 -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)
> $OS_ARCH can be (e.g.) amd64 if you're using a 64bit java on the x86_64 platform. So while gcc will try to link against the correct libjvm.so, it will fail because libhdfs is to be built 32bit (because of -m32)
> {noformat}
> [exec] /usr/bin/ld: skipping incompatible /usr/java64/latest/jre/lib/amd64/server/libjvm.so when searching for -ljvm
> [exec] /usr/bin/ld: cannot find -ljvm
> [exec] collect2: ld returned 1 exit status
> [exec] make: *** [/root/def/hadoop-0.16.3/build/libhdfs/libhdfs.so.1] Error 1
> {noformat}
> The solution should be to specify -m32 or -m64 depending on the os.arch detected.
> There are 3 cases to check:
> * 32bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 32bit java => libhdfs should be built 32bit, specify -m32
> * 64bit OS, 64bit java => libhdfs should be built 64bit, specify -m64
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.