You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Volodymyr Orlov (JIRA)" <ji...@apache.org> on 2010/05/17 00:35:43 UTC
[jira] Created: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Patch for running Hadoop on Windows without Cygwin
--------------------------------------------------
Key: HADOOP-6767
URL: https://issues.apache.org/jira/browse/HADOOP-6767
Project: Hadoop Common
Issue Type: Improvement
Components: build, conf, filecache, fs, io, scripts, security, test, util
Affects Versions: 0.20.2
Environment: Windows XP, 2003, 7, 2008
Reporter: Volodymyr Orlov
Fix For: 0.20.2
Proposed patch adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Posted by "Volodymyr Orlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Volodymyr Orlov updated HADOOP-6767:
------------------------------------
Description: Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin. (was: Proposed patch adds a possibility to run Hadoop on Windows without Cygwin. )
> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
> Key: HADOOP-6767
> URL: https://issues.apache.org/jira/browse/HADOOP-6767
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, conf, filecache, fs, io, scripts, security, test, util
> Affects Versions: 0.20.2
> Environment: Windows XP, 2003, 7, 2008
> Reporter: Volodymyr Orlov
> Fix For: 0.20.2
>
> Attachments: HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928501#action_12928501 ]
Hadoop QA commented on HADOOP-6767:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12456682/hadoop-6767-r1.tar.gz
against trunk revision 1031422.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/33//console
This message is automatically generated.
> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
> Key: HADOOP-6767
> URL: https://issues.apache.org/jira/browse/HADOOP-6767
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, conf, scripts
> Affects Versions: 0.22.0
> Environment: Windows XP, 2003, 7, 2008
> Reporter: Volodymyr Orlov
> Fix For: 0.22.0
>
> Attachments: Hadoop-0.20.2-patched.zip, hadoop-6767-r1.tar.gz, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Posted by "Volodymyr Orlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Volodymyr Orlov updated HADOOP-6767:
------------------------------------
Component/s: (was: security)
(was: filecache)
(was: test)
(was: util)
(was: fs)
(was: io)
Affects Version/s: (was: 0.20.2)
0.22.0
Fix Version/s: (was: 0.20.3)
0.22.0
> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
> Key: HADOOP-6767
> URL: https://issues.apache.org/jira/browse/HADOOP-6767
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, conf, scripts
> Affects Versions: 0.22.0
> Environment: Windows XP, 2003, 7, 2008
> Reporter: Volodymyr Orlov
> Fix For: 0.22.0
>
> Attachments: Hadoop-0.20.2-patched.zip, hadoop-6767-r1.tar.gz, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Posted by "Volodymyr Orlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Volodymyr Orlov updated HADOOP-6767:
------------------------------------
Attachment: HADOOP-6767.patch
Patch for Hadoop-0.20.2
> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
> Key: HADOOP-6767
> URL: https://issues.apache.org/jira/browse/HADOOP-6767
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, conf, filecache, fs, io, scripts, security, test, util
> Affects Versions: 0.20.2
> Environment: Windows XP, 2003, 7, 2008
> Reporter: Volodymyr Orlov
> Fix For: 0.20.2
>
> Attachments: HADOOP-6767.patch
>
>
> Proposed patch adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868591#action_12868591 ]
Steve Loughran commented on HADOOP-6767:
----------------------------------------
I actually think the ASF's interpretation of the LPGL is the issue here, there's been some coverage of the Hibernate Clause [http://www.mail-archive.com/legal-discuss@apache.org/msg00009.html] that could get things to co-exist -someone needs to talk to the JNA people about licensing here.
If we could get the licenses compatible then we could drop all cygwin requirements and say "use JNA for windows". That would be a step towards making windows a supported platform, rather than something for people to play with Hadoop on before going into production. A step: you still need to test at real or virtual production scale with the target OS, which means some variant of Windows Server. I don't want to get involved in that, dealing with inconsistencies between Linuxes is painful enough. MS might be able to offer cluster time.
First step: someone has to approach JNA and say "what do you mean by LGPL"?
> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
> Key: HADOOP-6767
> URL: https://issues.apache.org/jira/browse/HADOOP-6767
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, conf, filecache, fs, io, scripts, security, test, util
> Affects Versions: 0.20.2
> Environment: Windows XP, 2003, 7, 2008
> Reporter: Volodymyr Orlov
> Fix For: 0.20.2
>
> Attachments: Hadoop-0.20.2-patched.zip, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Posted by "Volodymyr Orlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868602#action_12868602 ]
Volodymyr Orlov commented on HADOOP-6767:
-----------------------------------------
Thank you Steve for your fast criticism on the merits. I have made some investigations and I would like to suggest changes that will make the patch more appropriate:
1) I will replace JNA with JNI. Thus, I'll make a code faster and there will be no need in getting round ASF + LGPL problem. Windows native code will be build only if releasing is done on Windows or with Wine. Hadoop will try to locate dll in the same way as it does it now for libhadoop.so and will fall back to Cygwin in case native library is not found
2) I will use Apache Commons Daemon instead of Java Service Wrapper
Also, all changes will be made against SVN_TRUNK, patch will not make changes to current javadocs and will use spaces instead of tab.
Nevertheless I would like to leave service wrapper configuration files/code and code that enables Hadoop to run on Windows without Cygwin in the same patch because there is currently no way to start Hadoop daemons on Windows without Cygwin. In case I will separate it into two patches, Hadoop users will still need to apply both.
Meanwhile I would like to add a description of a use case which prompted us to write the patch and a guide on how we are running Hadoop on Windows without Cygwin:
http://vorlsblog.blogspot.com/2010/05/running-hadoop-on-windows-without.html
> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
> Key: HADOOP-6767
> URL: https://issues.apache.org/jira/browse/HADOOP-6767
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, conf, filecache, fs, io, scripts, security, test, util
> Affects Versions: 0.20.2
> Environment: Windows XP, 2003, 7, 2008
> Reporter: Volodymyr Orlov
> Fix For: 0.20.2
>
> Attachments: Hadoop-0.20.2-patched.zip, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Posted by "Volodymyr Orlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Volodymyr Orlov updated HADOOP-6767:
------------------------------------
Attachment: hadoop-6767-r1.tar.gz
Hi,
After a pause I am continuing my work on porting Hadoop for Windows. In the attached archive I've added batch scripts for running Hadoop on Windows, made small changes to build.xml file to be able to build on Windows and added batch files and Apache Commons Daemon configuration files to run Hadoop as Windows services. All changes are against trunk
Best Regards,
Vladimir
> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
> Key: HADOOP-6767
> URL: https://issues.apache.org/jira/browse/HADOOP-6767
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, conf, filecache, fs, io, scripts, security, test, util
> Affects Versions: 0.20.2
> Environment: Windows XP, 2003, 7, 2008
> Reporter: Volodymyr Orlov
> Fix For: 0.20.3
>
> Attachments: Hadoop-0.20.2-patched.zip, hadoop-6767-r1.tar.gz, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Posted by "Volodymyr Orlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Volodymyr Orlov updated HADOOP-6767:
------------------------------------
Attachment: Hadoop-0.20.2-patched.zip
Added patched Hadoop 0.20.2 with configuration files, needed to run Hadoop as a Windows service
> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
> Key: HADOOP-6767
> URL: https://issues.apache.org/jira/browse/HADOOP-6767
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, conf, filecache, fs, io, scripts, security, test, util
> Affects Versions: 0.20.2
> Environment: Windows XP, 2003, 7, 2008
> Reporter: Volodymyr Orlov
> Fix For: 0.20.2
>
> Attachments: Hadoop-0.20.2-patched.zip, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868272#action_12868272 ]
Owen O'Malley commented on HADOOP-6767:
---------------------------------------
I agree with Steve. Apache doesn't allow LGPL. That said, I wish it would work for other platforms too...
> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
> Key: HADOOP-6767
> URL: https://issues.apache.org/jira/browse/HADOOP-6767
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, conf, filecache, fs, io, scripts, security, test, util
> Affects Versions: 0.20.2
> Environment: Windows XP, 2003, 7, 2008
> Reporter: Volodymyr Orlov
> Fix For: 0.20.2
>
> Attachments: Hadoop-0.20.2-patched.zip, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868187#action_12868187 ]
Steve Loughran commented on HADOOP-6767:
----------------------------------------
# JNA is LGPL, ASF is -1 to anything with this license that you must have to work, so it will take some care to get this to co-exist. Having an ivy dependency that says JNA must be on the classpath is not going to work.
# I don't like the .conf files which seem to build up classpaths from hard coded paths, assume you are running from the build directories, and could not be used in production systems for this reason.The conf files are there because you've chosen one way to run Hadoop as service, but as there are other options, that's a different patch, independent of everything else.
# Moving from UnixUserGroupInformation to UserGroupInformation is going to break a lot of downstream code
# The usual space vs tab rules and other changes to javadocs complicate the patch.
# A patch to 0.20.x will only break everything that uses cygwin
# I expect for performance reasons not using the commons-compress libs on Unix makes sense, but that codepath should still be tested on unix, so that hudson catches problems.
# I don't see any explicit tests of the new stuff, so the only way this stuff would get tested now is if someone sets up Hudson on one or more windows versions and runs this as a service.
what you have done is very impressive, but as it stands, I'm -1 to this. I may be more amenable if
# There is a strategy which the hadoop dev teams and the ASF are happy with for using JNA. Building against it when present but not redistributing it is one option, in which case Hadoop on windows has to fall back to cygwin where JNA isn't on the path
# changes are against SVN_TRUNK, no attempt to support existing releases.
# the wrapper stuff is separated completely off from the JNA stuff. How you run Hadoop as a service on windows is independent from making Hadoop work without cygwin; merging the two just makes it harder to use.
# {{TaskRunner.normalizeClassPath()}} needs a complete rewrite following apache coding best practices, some test cases that are designed to work cross platform (hint: split the windows and unix options, test both)
# Minor: the patch shouldn't make changes to javadocs, line endings that aren't needed.
The big concern I have is the split between windows and everything else. I know you've eliminated cygwin, but you've added in requirements to understand the Win32 APIs, win32 vs win64 issues, and a separate windows codebase that will need to be tested on 32-bit and 64 bit versions of Windows Vista, Win7 and windows server; 6 different OS combinations we currently avoid.
That scares me.
> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
> Key: HADOOP-6767
> URL: https://issues.apache.org/jira/browse/HADOOP-6767
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, conf, filecache, fs, io, scripts, security, test, util
> Affects Versions: 0.20.2
> Environment: Windows XP, 2003, 7, 2008
> Reporter: Volodymyr Orlov
> Fix For: 0.20.2
>
> Attachments: Hadoop-0.20.2-patched.zip, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6767) Patch for running Hadoop on Windows
without Cygwin
Posted by "Volodymyr Orlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Volodymyr Orlov updated HADOOP-6767:
------------------------------------
Release Note: Batch scripts for running Hadoop on windows, scripts for setting Hadoop as windows service using Apache Commons Daemon, and fixes in build (was: Patch uses JNA (https://jna.dev.java.net/) to perform operations such as getting folder size, getting disk space statistics, getting current user, current user groups, setting and getting FS permissions, setting and getting file or folder owner and group. Patch has been tested on Windows XP, Windows 2008, Windows 7. Later on I will add to the issue link to the description of how to set up Hadoop cluster on Windows without Cygwin)
Status: Patch Available (was: Open)
please see README.txt file in the archive I've attached to the issue
> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
> Key: HADOOP-6767
> URL: https://issues.apache.org/jira/browse/HADOOP-6767
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, conf, scripts
> Affects Versions: 0.22.0
> Environment: Windows XP, 2003, 7, 2008
> Reporter: Volodymyr Orlov
> Fix For: 0.22.0
>
> Attachments: Hadoop-0.20.2-patched.zip, hadoop-6767-r1.tar.gz, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.