You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Matt Foley (Created) (JIRA)" <ji...@apache.org> on 2012/02/08 09:31:59 UTC
[jira] [Created] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds
Binary tarball does not preserve platform info for native builds
----------------------------------------------------------------
Key: HADOOP-8037
URL: https://issues.apache.org/jira/browse/HADOOP-8037
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 1.0.1
Reporter: Matt Foley
Assignee: Giridharan Kesavan
The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Giridharan Kesavan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207302#comment-13207302 ]
Giridharan Kesavan commented on HADOOP-8037:
--------------------------------------------
+1 Verified rpm's and deb's for both 32bit and 64bit arch. Thanks Matt
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Matt Foley
> Priority: Blocker
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037-2.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HADOOP-8037) Binary tarball does not
preserve platform info for native builds, and RPMs fail to provide needed
symlinks for libhadoop.so
Posted by "Matt Foley (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203973#comment-13203973 ]
Matt Foley edited comment on HADOOP-8037 at 2/8/12 8:52 PM:
------------------------------------------------------------
The RPMs now appear perfect. However, the bin-tarball still has a few 64-bit-specific files that shouldn't be there. See lines flagged with {{"<==="}} below: {code}
$ tar ztf hadoop-1.0.1.tar.gz|grep libhdfs|grep -v /src/ |grep -v /docs/
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhdfs
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.la <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so.0 <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so.0.0.0 <===
{code}
and {code}
$ tar ztf hadoop-1.0.1.tar.gz|grep libhadoop|grep -v /src/ |grep -v /docs/
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.la
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhadoop
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooppipes.a <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooputils.a <===
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.la
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0
{code}
was (Author: mattf):
The RPMs now appear perfect. However, the bin-tarball still has {{hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.*}} files, and {{hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop{pipes,utils}.a}} files, that shouldn't be there: {code}
$ tar ztf hadoop-1.0.1.tar.gz|grep libhdfs|grep -v /src/ |grep -v /docs/
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhdfs
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.la <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so.0 <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so.0.0.0 <===
{code}
and {code}
$ tar ztf hadoop-1.0.1.tar.gz|grep libhadoop|grep -v /src/ |grep -v /docs/
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.la
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhadoop
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooppipes.a <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooputils.a <===
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.la
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0
{code}
See lines flagged with {{"<==="}} above.
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8037:
-------------------------------
Priority: Blocker (was: Major)
Changing this to blocker, because it did in fact block the 1.0.1 RC for several days while we straightened it out.
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Priority: Blocker
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8037:
-------------------------------
Release Note: This fix is marked "incompatible" only because it changes the bin-tarball directory structure to be consistent with the source tarball directory structure. Everything else (in particular, the source tarball and rpm directory structures) are unchanged, except that the 64-bit rpms and debs now use lib64 instead of lib for native builds. (was: This fix is marked "incompatible" only because it changes the bin-tarball directory structure to be consistent with the source tarball directory structure. Everything else (in particular, the source tarball and rpm directory structures) are unchanged, except that the 64-bit rpms and debs use lib64 instead of lib for native builds.)
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Priority: Blocker
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037-2.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and fails to provide needed symlinks for
libhadoop.so
Posted by "Matt Foley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203442#comment-13203442 ]
Matt Foley commented on HADOOP-8037:
------------------------------------
The symlinks are now good! However, the rpms are retaining some paths they shouldn't: {code}
$ rpm -qlp hadoop-1.0.1-1.i386.rpm|grep libhadoop
/usr/c++/Linux-i386-32/lib/libhadooppipes.a
/usr/c++/Linux-i386-32/lib/libhadooputils.a
/usr/lib/libhadoop.a
/usr/lib/libhadoop.la
/usr/lib/libhadoop.so
/usr/lib/libhadoop.so.1
/usr/lib/libhadoop.so.1.0.0
/usr/lib/libhadooppipes.a
/usr/lib/libhadooputils.a
$ rpm -qlp hadoop-1.0.1-1.amd64.rpm|grep libhadoop
/usr/c++/Linux-amd64-64/lib/libhadooppipes.a
/usr/c++/Linux-amd64-64/lib/libhadooputils.a
/usr/c++/Linux-i386-32/lib/libhadooppipes.a
/usr/c++/Linux-i386-32/lib/libhadooputils.a
/usr/lib64/libhadoop.a
/usr/lib64/libhadoop.la
/usr/lib64/libhadoop.so
/usr/lib64/libhadoop.so.1
/usr/lib64/libhadoop.so.1.0.0
/usr/lib64/libhadooppipes.a
/usr/lib64/libhadooputils.a
{code}
I think the /usr/c++/<platform>/lib paths shouldn't be in the rpms, only in the bin-tarball.
Also, in the bin-tarball, the libhadooppipes and utils libs are handled inconsistently: {code}
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhadoop
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooppipes.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.la
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0
I think the files {code}
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooppipes.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooputils.a
{code}
shouldn't be there.
The source tarball looked fine.
Finally, please remove the commented-out line in build.xml: {code}
<!--<delete dir="${dist.dir}"/> --> {code}
Thanks.
> Binary tarball does not preserve platform info for native builds, and fails to provide needed symlinks for libhadoop.so
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Attachments: hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8037:
-------------------------------
Release Note: This fix is marked "incompatible" only because it changes the bin-tarball directory structure to be consistent with the source tarball directory structure. The source tarball is unchanged. RPMs and DEBs now use an intermediate bin-tarball with an "${os.arch}" tag (like the packages themselves). The un-tagged bin-tarball is now multi-platform and retains the structure of the source tarball; it is in fact generated by target "tar", not by target "binary". Finally, in the 64-bit RPMs and DEBs, the native libs go in the "lib64" directory instead of "lib". (was: This fix is marked "incompatible" only because it changes the bin-tarball directory structure to be consistent with the source tarball directory structure. Everything else (in particular, the source tarball and rpm directory structures) are unchanged, except that the 64-bit rpms and debs now use lib64 instead of lib for native libraries.)
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Matt Foley
> Priority: Blocker
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037-2.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8037:
-------------------------------
Release Note: This fix is marked "incompatible" only because it changes the bin-tarball directory structure to be consistent with the source tarball directory structure. Everything else (in particular, the source tarball and rpm directory structures) are unchanged, except that the 64-bit rpms and debs now use lib64 instead of lib for native libraries. (was: This fix is marked "incompatible" only because it changes the bin-tarball directory structure to be consistent with the source tarball directory structure. Everything else (in particular, the source tarball and rpm directory structures) are unchanged, except that the 64-bit rpms and debs now use lib64 instead of lib for native builds.)
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Priority: Blocker
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037-2.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley reopened HADOOP-8037:
--------------------------------
Regrettably, I have to revert this patch. I didn't notice it before, but the sizes of the resulting rpm and bin-tarball files are greatly increased due to duplicated files. We have to try again:
{code}
pre-patch sizes:
-rw-r--r-- 1 hortonfo users 31975332 Jan 31 02:42 hadoop_1.0.1-1_amd64.deb
-rw-r--r-- 1 hortonfo users 32039471 Jan 31 02:40 hadoop-1.0.1-1.amd64.rpm
-rw-r--r-- 1 hortonfo users 35909710 Jan 31 02:35 hadoop_1.0.1-1_i386.deb
-rw-r--r-- 1 hortonfo users 35974659 Jan 31 02:34 hadoop-1.0.1-1.i386.rpm
-rw-r--r-- 1 hortonfo users 31965140 Jan 31 02:41 hadoop-1.0.1-bin.tar.gz
-rw-r--r-- 1 hortonfo users 60322619 Jan 31 02:39 hadoop-1.0.1.tar.gz
post-patch sizes:
-rw-r--r-- 1 hortonfo users 93729906 Feb 9 03:38 hadoop_1.0.1-1_amd64.deb
-rw-r--r-- 1 hortonfo users 40754076 Feb 9 03:36 hadoop-1.0.1-1.amd64.rpm
-rw-r--r-- 1 hortonfo users 101357042 Feb 9 03:29 hadoop_1.0.1-1_i386.deb
-rw-r--r-- 1 hortonfo users 48735998 Feb 9 03:27 hadoop-1.0.1-1.i386.rpm
-rw-r--r-- 1 hortonfo users 66467789 Feb 9 03:38 hadoop-1.0.1-bin.tar.gz
-rw-r--r-- 1 hortonfo users 60782862 Feb 9 03:34 hadoop-1.0.1.tar.gz
pre-patch file counts:
445 hadoop-1.0.1-1.amd64.rpm
445 hadoop-1.0.1-1.i386.rpm
330 hadoop-1.0.1-bin.tar.gz
6974 hadoop-1.0.1.tar.gz
post-patch file counts:
789 hadoop-1.0.1-1.amd64.rpm
792 hadoop-1.0.1-1.i386.rpm
668 hadoop-1.0.1-bin.tar.gz
6978 hadoop-1.0.1.tar.gz
{code}
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Priority: Blocker
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8037:
-------------------------------
Attachment: hadoop-8037-2.patch
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Priority: Blocker
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037-2.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and fails to provide needed symlinks for
libhadoop.so
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8037:
-------------------------------
Summary: Binary tarball does not preserve platform info for native builds, and fails to provide needed symlinks for libhadoop.so (was: Binary tarball does not preserve platform info for native builds)
There's a secondary issue: The code fragment in bin-package that does the stripping of platform names from the directory paths is this: {code}
<move todir="${dist.dir}/lib" flatten="true">
<fileset dir="${dist.dir}/native">
<include name="*/*"/>
</fileset>
</move>
<delete dir="${dist.dir}/native"/>
{code}
This usage of the <move> ant task fails to grab the symlinks that are important for libhadoop. Prior to this, the platform-specific subdirectories contain, e.g., {code}
./hadoop-1.0.1/native/Linux-i386-32/libhadoop.so
./hadoop-1.0.1/native/Linux-i386-32/libhadoop.so.1
./hadoop-1.0.1/native/Linux-i386-32/libhadoop.so.1.0.0
{code}
However, after the projection is done by the <move> task, we are left with only {code}
hadoop-1.0.1/lib/libhadoop.so.1.0.0
{code}
The other two files were symlinks to libhadoop.so.1.0.0, and did not get moved.
As a result, the rpm with native build fails to provide the needed libhadoop.so.
This could be opened as a separate Jira, but both need to be solved in the same area of code, so I'm leaving them together. I'll add to the bug title.
> Binary tarball does not preserve platform info for native builds, and fails to provide needed symlinks for libhadoop.so
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and fails to provide needed symlinks for
libhadoop.so
Posted by "Giridharan Kesavan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-8037:
---------------------------------------
Attachment: hadoop-8037.patch
this patch takes care of preserving the architecture specific lib directories in case of bin-tar.gz and also takes care of retaining the symlinks for the so files in case of the rpm's
> Binary tarball does not preserve platform info for native builds, and fails to provide needed symlinks for libhadoop.so
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Attachments: hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley resolved HADOOP-8037.
--------------------------------
Resolution: Fixed
Fix Version/s: 1.0.1
Committed to branch-1 and branch-1.0.
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Matt Foley
> Priority: Blocker
> Fix For: 1.0.1
>
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037-2.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and fails to provide needed symlinks for
libhadoop.so
Posted by "Giridharan Kesavan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-8037:
---------------------------------------
Attachment: hadoop-8027-1.patch
hadoop-8027-1.patch address the issues mentioned above
> Binary tarball does not preserve platform info for native builds, and fails to provide needed symlinks for libhadoop.so
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Attachments: hadoop-8027-1.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206382#comment-13206382 ]
Matt Foley commented on HADOOP-8037:
------------------------------------
Please review the new patch attached. It seems to do all the right things. RPMs and DEBs now use an intermediate bin-tarball with an "${os.arch}" tag (like the packages themselves). The un-tagged bin-tarball is now multi-platform and retains the structure of the source tarball; it is in fact generated by target "tar", not by target "binary". Finally, in the 64-bit RPMs and DEBs, the native libs go in "lib64" instead of "lib", as suggested by Giri's previous patch.
Results of the checks are: {code}
file sizes:
-rw-r--r-- 1 hortonfo users 32507490 Feb 12 08:57 hadoop_1.0.1-1_amd64.deb
-rw-r--r-- 1 hortonfo users 32551430 Feb 12 08:55 hadoop-1.0.1-1.amd64.rpm
-rw-r--r-- 1 hortonfo users 36460606 Feb 12 08:50 hadoop_1.0.1-1_i386.deb
-rw-r--r-- 1 hortonfo users 36522879 Feb 12 08:49 hadoop-1.0.1-1.i386.rpm
-rw-r--r-- 1 hortonfo users 33695494 Feb 12 08:54 hadoop-1.0.1-bin.tar.gz
-rw-r--r-- 1 hortonfo users 60790076 Feb 12 08:54 hadoop-1.0.1.tar.gz
file counts:
$ rpm -lqp hadoop-1.0.1-1.amd64.rpm|wc -l ==> 448
$ rpm -lqp hadoop-1.0.1-1.i386.rpm |wc -l ==> 450
$ tar ztf hadoop-1.0.1-bin.tar.gz |wc -l ==> 383
$ tar ztf hadoop-1.0.1.tar.gz |wc -l ==> 6994
$ tar ztf hadoop-1.0.1.tar.gz|grep libhdfs|grep -v /src/ |grep -v /docs/
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhdfs
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
$ tar ztf hadoop-1.0.1.tar.gz|grep libhadoop|grep -v /src/ |grep -v /docs/
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.la
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhadoop
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.la
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0
$ rpm -qlp hadoop-1.0.1-1.i386.rpm|grep libhadoop
/usr/lib/libhadoop.a
/usr/lib/libhadoop.la
/usr/lib/libhadoop.so
/usr/lib/libhadoop.so.1
/usr/lib/libhadoop.so.1.0.0
/usr/lib/libhadooppipes.a
/usr/lib/libhadooputils.a
$ rpm -qlp hadoop-1.0.1-1.amd64.rpm|grep libhadoop
/usr/lib64/libhadoop.a
/usr/lib64/libhadoop.la
/usr/lib64/libhadoop.so
/usr/lib64/libhadoop.so.1
/usr/lib64/libhadoop.so.1.0.0
/usr/lib64/libhadooppipes.a
/usr/lib64/libhadooputils.a
$ rpm -qlp hadoop-1.0.1-1.i386.rpm|grep libhdfs
/usr/lib/libhdfs.la
/usr/lib/libhdfs.so
/usr/lib/libhdfs.so.0
/usr/lib/libhdfs.so.0.0.0
$ rpm -qlp hadoop-1.0.1-1.amd64.rpm|grep libhdfs
/usr/lib64/libhdfs.la
/usr/lib64/libhdfs.so
/usr/lib64/libhdfs.so.0
/usr/lib64/libhdfs.so.0.0.0
{code}
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Priority: Blocker
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037-2.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203973#comment-13203973 ]
Matt Foley commented on HADOOP-8037:
------------------------------------
The RPMs now appear perfect. However, the bin-tarball still has {{hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.*}} files, and {{hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop{pipes,utils}.a}} files, that shouldn't be there: {code}
$ tar ztf hadoop-1.0.1.tar.gz|grep libhdfs|grep -v /src/ |grep -v /docs/
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhdfs
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.la <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so.0 <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so.0.0.0 <===
{code}
and {code}
$ tar ztf hadoop-1.0.1.tar.gz|grep libhadoop|grep -v /src/ |grep -v /docs/
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.la
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhadoop
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooppipes.a <===
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooputils.a <===
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.la
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0
{code}
See lines flagged with {{"<==="}} above.
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and fails to provide needed symlinks for
libhadoop.so
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8037:
-------------------------------
Release Note: This fix is marked "incompatible" only because it changes the bin-tarball directory structure to be consistent with the source tarball directory structure. Everything else (in particular, the source tarball and rpm directory structures) are unchanged.
> Binary tarball does not preserve platform info for native builds, and fails to provide needed symlinks for libhadoop.so
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8037:
-------------------------------
Summary: Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so (was: Binary tarball does not preserve platform info for native builds, and fails to provide needed symlinks for libhadoop.so)
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley reassigned HADOOP-8037:
----------------------------------
Assignee: Matt Foley (was: Giridharan Kesavan)
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Matt Foley
> Priority: Blocker
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037-2.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8037:
-------------------------------
Fix Version/s: (was: 1.0.1)
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Priority: Blocker
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley resolved HADOOP-8037.
--------------------------------
Resolution: Fixed
Fix Version/s: 1.0.1
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Priority: Blocker
> Fix For: 1.0.1
>
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HADOOP-8037) Binary tarball does not
preserve platform info for native builds, and fails to provide needed
symlinks for libhadoop.so
Posted by "Matt Foley (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203442#comment-13203442 ]
Matt Foley edited comment on HADOOP-8037 at 2/8/12 10:26 AM:
-------------------------------------------------------------
The symlinks are now good! However, the rpms are retaining some paths they shouldn't: {code}
$ rpm -qlp hadoop-1.0.1-1.i386.rpm|grep libhadoop
/usr/c++/Linux-i386-32/lib/libhadooppipes.a
/usr/c++/Linux-i386-32/lib/libhadooputils.a
/usr/lib/libhadoop.a
/usr/lib/libhadoop.la
/usr/lib/libhadoop.so
/usr/lib/libhadoop.so.1
/usr/lib/libhadoop.so.1.0.0
/usr/lib/libhadooppipes.a
/usr/lib/libhadooputils.a
$ rpm -qlp hadoop-1.0.1-1.amd64.rpm|grep libhadoop
/usr/c++/Linux-amd64-64/lib/libhadooppipes.a
/usr/c++/Linux-amd64-64/lib/libhadooputils.a
/usr/c++/Linux-i386-32/lib/libhadooppipes.a
/usr/c++/Linux-i386-32/lib/libhadooputils.a
/usr/lib64/libhadoop.a
/usr/lib64/libhadoop.la
/usr/lib64/libhadoop.so
/usr/lib64/libhadoop.so.1
/usr/lib64/libhadoop.so.1.0.0
/usr/lib64/libhadooppipes.a
/usr/lib64/libhadooputils.a
{code}
I think the /usr/c++/<platform>/lib paths shouldn't be in the rpms, only in the bin-tarball.
Also, in the bin-tarball, the libhadooppipes and utils libs are handled inconsistently: {code}
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhadoop
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooppipes.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.la
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0
{code}
I think the files {code}
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooppipes.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooputils.a
{code}
shouldn't be there.
The source tarball looked fine.
Finally, please remove the commented-out line in build.xml: {code}
<!--<delete dir="${dist.dir}"/> --> {code}
Thanks.
was (Author: mattf):
The symlinks are now good! However, the rpms are retaining some paths they shouldn't: {code}
$ rpm -qlp hadoop-1.0.1-1.i386.rpm|grep libhadoop
/usr/c++/Linux-i386-32/lib/libhadooppipes.a
/usr/c++/Linux-i386-32/lib/libhadooputils.a
/usr/lib/libhadoop.a
/usr/lib/libhadoop.la
/usr/lib/libhadoop.so
/usr/lib/libhadoop.so.1
/usr/lib/libhadoop.so.1.0.0
/usr/lib/libhadooppipes.a
/usr/lib/libhadooputils.a
$ rpm -qlp hadoop-1.0.1-1.amd64.rpm|grep libhadoop
/usr/c++/Linux-amd64-64/lib/libhadooppipes.a
/usr/c++/Linux-amd64-64/lib/libhadooputils.a
/usr/c++/Linux-i386-32/lib/libhadooppipes.a
/usr/c++/Linux-i386-32/lib/libhadooputils.a
/usr/lib64/libhadoop.a
/usr/lib64/libhadoop.la
/usr/lib64/libhadoop.so
/usr/lib64/libhadoop.so.1
/usr/lib64/libhadoop.so.1.0.0
/usr/lib64/libhadooppipes.a
/usr/lib64/libhadooputils.a
{code}
I think the /usr/c++/<platform>/lib paths shouldn't be in the rpms, only in the bin-tarball.
Also, in the bin-tarball, the libhadooppipes and utils libs are handled inconsistently: {code}
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhadoop
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhadooputils.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooppipes.a
hadoop-1.0.1/c++/Linux-i386-32/lib/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooppipes.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooputils.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.a
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.la
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1
hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0
I think the files {code}
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooppipes.a
hadoop-1.0.1/lib/native/Linux-amd64-64/libhadooputils.a
{code}
shouldn't be there.
The source tarball looked fine.
Finally, please remove the commented-out line in build.xml: {code}
<!--<delete dir="${dist.dir}"/> --> {code}
Thanks.
> Binary tarball does not preserve platform info for native builds, and fails to provide needed symlinks for libhadoop.so
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Attachments: hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and fails to provide needed symlinks for
libhadoop.so
Posted by "Giridharan Kesavan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giridharan Kesavan updated HADOOP-8037:
---------------------------------------
Attachment: hadoop-8037-1.patch
uploadign the right patch
> Binary tarball does not preserve platform info for native builds, and fails to provide needed symlinks for libhadoop.so
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley closed HADOOP-8037.
------------------------------
Closed upon release of 1.0.1.
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Matt Foley
> Priority: Blocker
> Fix For: 1.0.1
>
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037-2.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8037:
-------------------------------
Release Note: This fix is marked "incompatible" only because it changes the bin-tarball directory structure to be consistent with the source tarball directory structure. Everything else (in particular, the source tarball and rpm directory structures) are unchanged, except that the 64-bit rpms and debs use lib64 instead of lib for native builds. (was: This fix is marked "incompatible" only because it changes the bin-tarball directory structure to be consistent with the source tarball directory structure. Everything else (in particular, the source tarball and rpm directory structures) are unchanged.)
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Priority: Blocker
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037-2.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and fails to provide needed symlinks for
libhadoop.so
Posted by "Matt Foley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203728#comment-13203728 ]
Matt Foley commented on HADOOP-8037:
------------------------------------
Same issue with 'libhdfs': {code}
$ rpm -qlp hadoop-1.0.1-1.i386.rpm|grep libhdfs
/usr/c++/Linux-i386-32/lib/libhdfs.la
/usr/c++/Linux-i386-32/lib/libhdfs.so
/usr/c++/Linux-i386-32/lib/libhdfs.so.0
/usr/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
/usr/lib/libhdfs.la
/usr/lib/libhdfs.so
/usr/lib/libhdfs.so.0
/usr/lib/libhdfs.so.0.0.0
$ rpm -qlp hadoop-1.0.1-1.amd64.rpm|grep libhdfs
/usr/c++/Linux-amd64-64/lib/libhdfs.la
/usr/c++/Linux-amd64-64/lib/libhdfs.so
/usr/c++/Linux-amd64-64/lib/libhdfs.so.0
/usr/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
/usr/c++/Linux-i386-32/lib/libhdfs.la
/usr/c++/Linux-i386-32/lib/libhdfs.so
/usr/c++/Linux-i386-32/lib/libhdfs.so.0
/usr/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
/usr/lib64/libhdfs.la
/usr/lib64/libhdfs.so
/usr/lib64/libhdfs.so.0
/usr/lib64/libhdfs.so.0.0.0
{code}
The architecture-specific paths shouldn't be in the rpms or debs.
And in the tarballs: {code}
$ tar ztf hadoop-1.0.1.tar.gz|grep libhdfs|grep -v /src/ |grep -v /docs/
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
$ tar ztf hadoop-1.0.1-bin.tar.gz|grep libhdfs
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.la
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0
hadoop-1.0.1/c++/Linux-i386-32/lib/libhdfs.so.0.0.0
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.la
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so.0
hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.so.0.0.0
{code}
To be consistent with the source tarball, the {{"hadoop-1.0.1/lib/native/Linux-amd64-64/libhdfs.*"}} files shouldn't be in the bin-tarball. The fact that the 32-bit equivalents aren't there might be a clue to the error.
Thanks for the hard work, Giri!
> Binary tarball does not preserve platform info for native builds, and fails to provide needed symlinks for libhadoop.so
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Attachments: hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8037) Binary tarball does not preserve
platform info for native builds, and RPMs fail to provide needed symlinks
for libhadoop.so
Posted by "Matt Foley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204209#comment-13204209 ]
Matt Foley commented on HADOOP-8037:
------------------------------------
I'm still seeing the above problem (extra copies of certain 64-bit native libs in the bin-tarball), but Giri doesn't see it in his environment.
This patch resolves the two dangerous problems, and leaves only a minor glitch in the bin-tarball, which is a new artifact that few people are known to use. I'm going to accept this patch, and move the remaining issue into a new bug, to be resolved in the future.
+1, thanks Giri!
> Binary tarball does not preserve platform info for native builds, and RPMs fail to provide needed symlinks for libhadoop.so
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8037
> URL: https://issues.apache.org/jira/browse/HADOOP-8037
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 1.0.1
> Reporter: Matt Foley
> Assignee: Giridharan Kesavan
> Attachments: hadoop-8027-1.patch, hadoop-8037-1.patch, hadoop-8037.patch
>
>
> The source tarball uses "package" ant target, which includes both sets of native builds (32 and 64 bit libraries), under subdirectories that are named for the supported platform, so you can tell what they are.
> The binary tarball uses the "bin-package" ant target, which projects both sets of native builds into a single directory, stripping out the platform names from the directory paths. Since the native built libraries have identical names, only one of each survives the process. Afterward, there is no way to know whether they are intended for 32 or 64 bit environments.
> It seems to be done this way as a step toward building the rpm and deb artifacts. But the rpms and debs are self-identifying as to the platform they were built for, and contain only one set of libs each, while the binary tarball isn't. The binary tarball should have the same platform-specific subdirectories that the full tarball does; but this means that the rpm and deb builds have to be more careful about include/exclude specs for what goes into those artifacts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira