You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Ivan Mitic (JIRA)" <ji...@apache.org> on 2012/11/03 00:40:12 UTC
[jira] [Created] (HADOOP-9008) Building hadoop tarball fails on
Windows
Ivan Mitic created HADOOP-9008:
----------------------------------
Summary: Building hadoop tarball fails on Windows
Key: HADOOP-9008
URL: https://issues.apache.org/jira/browse/HADOOP-9008
Project: Hadoop Common
Issue Type: Bug
Affects Versions: trunk-win
Reporter: Ivan Mitic
Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth reassigned HADOOP-9008:
-------------------------------------
Assignee: Chris Nauroth
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
> Assignee: Chris Nauroth
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth updated HADOOP-9008:
----------------------------------
Attachment: HADOOP-9008-branch-trunk-win.patch
I made one additional change to guarantee that winutils goes into the bin directory for the common and distribution tarballs.
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
> Assignee: Chris Nauroth
> Attachments: HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch
>
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth updated HADOOP-9008:
----------------------------------
Attachment: HADOOP-9008-branch-trunk-win.patch
Thank you, Raja. I think you're right. I have attached a new patch that simplifies the make_file_filter function and adds some comments to help prevent confusion around the Python tarfile.add API.
Also, I realized that my earlier patch did not include all of the files that I changed. Can you please take another look? There are smaller changes in a few other pom.xml files.
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
> Assignee: Chris Nauroth
> Attachments: HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch
>
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth updated HADOOP-9008:
----------------------------------
Attachment: HADOOP-9008-branch-trunk-win.patch
Sorry, uploading the patch one more time, back to including just hadoop-dist/pom.xml and hadoop-project-dist/pom.xml. I forgot that the other module-specific ones are tracked in separate jiras.
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
> Assignee: Chris Nauroth
> Attachments: HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch
>
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489926#comment-13489926 ]
Bikas Saha commented on HADOOP-9008:
------------------------------------
or maven plugins like the ones used to compile protobufs without sh scripts.
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491898#comment-13491898 ]
Chris Nauroth commented on HADOOP-9008:
---------------------------------------
It looks like this is going to have to be Python. Trying to do this purely with Maven plugins, I see that it drops the symlinks that were created as part of the Linux native builds. It's also cumbersome to set file permissions correctly with the Maven plugin.
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
> Assignee: Chris Nauroth
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth updated HADOOP-9008:
----------------------------------
Attachment: HADOOP-9008-branch-trunk-win.patch
The attached patch ports the sh scripting in the distribution build to Python. It wasn't possible to use only Maven plugins (like maven-antrun-plugin with a <tar> task), because they mishandled permissions and symlinks in the built tarballs.
I tested all of the following build variations:
Windows: mvn -Pnative-win -Pdist -Dtar -DskipTests clean package
Mac: mvn -Pdist -Dtar -DskipTests clean package
Ubuntu: mvn -Pnative -Pdist -Dtar -DskipTests clean package
Ubuntu: mvn -Pnative -Pdist -Dtar -Drequire.snappy -Dbundle.snappy -Dsnappy.lib=/usr/local/lib -DskipTests clean package
This works on Windows. Additionally, on Mac and Ubuntu, I compared the built tarballs from before and after my changes. I confirmed that the resulting tarballs have exactly the same contents, including permissions and symlinks.
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
> Assignee: Chris Nauroth
> Attachments: HADOOP-9008-branch-trunk-win.patch
>
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Raja Aluri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495633#comment-13495633 ]
Raja Aluri commented on HADOOP-9008:
------------------------------------
+1. LGTM.
Chris,
Thanks for incorporating the feedback.
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
> Assignee: Chris Nauroth
> Attachments: HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch
>
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth updated HADOOP-9008:
----------------------------------
Attachment: HADOOP-9008-branch-trunk-win.patch
Raja, I am attaching an updated patch that incorporates your advice to use abspath and / where applicable to improve readability. I verified that this works cross-platform by retesting all build variations.
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
> Assignee: Chris Nauroth
> Attachments: HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch
>
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Raja Aluri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494270#comment-13494270 ]
Raja Aluri commented on HADOOP-9008:
------------------------------------
+1 for the change. A minor nit, if you may choose to change,
I was wondering if the following code can eliminate some of the 'else' loops.
{code}
def filter_func(tar_info):
if tar_info.name == root:
return tar_info
elif tar_info.isfile() or tar_info.issym():
if file_name_filter(basename(tar_info.name)):
return tar_info
else:
return None
else:
return None
return filter_func
{code}
I just thought the code will be more concise
{code}
def filter_func(tar_info):
if tar_info.name == root:
return tar_info
if tar_info.isfile() or tar_info.issym():
if file_name_filter(basename(tar_info.name)):
return tar_info
return None
return filter_func
{code}
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
> Assignee: Chris Nauroth
> Attachments: HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch
>
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas resolved HADOOP-9008.
-------------------------------------
Resolution: Fixed
Fix Version/s: trunk-win
+1. I committed the patch to branch-trunk-win.
Thank you Chris. Thank you Raja for the review.
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
> Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch
>
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9008) Building hadoop tarball fails on
Windows
Posted by "Raja Aluri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494351#comment-13494351 ]
Raja Aluri commented on HADOOP-9008:
------------------------------------
Chris.
It is just for code readability sake. I will leave it to you.
Sorry for not pointing this out earlier.
Please consider using os.path.join() or os.path.abspath(os.path.join()) instead of normpath for cross platform sake.
Also for building directory structure like the one below without platform specific '/'s
{code}
arc_name = base_name + "/lib/native"
{code}
can be
{code}
os.path.join(base_name, 'lib' , 'native')
{code}
{code}
>>> os.path.abspath(".")
'/Users/raja/work/repos/'
>>> os.path.abspath("../")
'/Users/raja/work'
>>> os.path.normpath("../../")
'../..'
>>> os.path.normpath("..")
'..'
>>> os.path.normpath("..")
{code}
{code}
> Building hadoop tarball fails on Windows
> ----------------------------------------
>
> Key: HADOOP-9008
> URL: https://issues.apache.org/jira/browse/HADOOP-9008
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: trunk-win
> Reporter: Ivan Mitic
> Assignee: Chris Nauroth
> Attachments: HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch
>
>
> Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests -Dtar}} fails on Windows.
> Build system generates sh scripts that execute build tasks what does not work on Windows without Cygwin. It might make sense to apply the same pattern as in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira