You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Chris Nauroth (JIRA)" <ji...@apache.org> on 2012/10/12 23:23:03 UTC
[jira] [Created] (HADOOP-8924) Hadoop Common creating version
annotation must not depend on sh
Chris Nauroth created HADOOP-8924:
-------------------------------------
Summary: Hadoop Common creating version annotation must not depend on sh
Key: HADOOP-8924
URL: https://issues.apache.org/jira/browse/HADOOP-8924
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley reopened HADOOP-8924:
--------------------------------
After discussion in common-dev@, the community consensus rejected use of Python. (see references in HADOOP-9082) Also, it was encouraged to take functionality out of scripts and move it into core code.
I suggest that we convert saveVersion.sh to Java, and change the ant script to a recursive call that compiles the java class and then invokes it. Same thing should work in trunk, but be aware there are two instances of saveVersion.sh in trunk: hadoop-common-project/hadoop-common/dev-support/saveVersion.sh and hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/scripts/saveVersion.sh. They are slightly different, but I suspect that can and should be subsumed.
> Hadoop Common creating package-info.java must not depend on sh
> --------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510606#comment-13510606 ]
Matt Foley commented on HADOOP-8924:
------------------------------------
How about the following:
1. convert saveVersion.sh into saveVersion.java. Pass the parameter values (including the root of the source tree for checksum calculation) in to the class constructor at invocation time.
2. add a dependency rule that causes saveVersion.class to be compiled before any other java is compiled, and separately from it.
3. add a simple execution outcall from ant or maven that calls the class, and passes the parameter values in to the call. If maven requires a completely vanilla "plug-in" to do this well, I'm okay with that. I just don't want the significant functionality of saveVersion.java to be subsumed into the plug-in and become opaque to the hadoop source.
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Alejandro Abdelnur
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch, HADOOP-8924.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated HADOOP-8924:
---------------------------------------
Attachment: HADOOP-8924.patch
Thanks Chris.
(not setting as patch avail because the plugin is not avail in a Maven repo, details follow).
Attached is patch that uses a maven plugin to generate the version information.
The patch changes the VersionInfo class to read all the info from a properties file instead of the package-info.java and the HadoopVersionInfoAnnotation.
The HadoopVersionInfoAnnotation.java and the saveVersion.sh files are deleted.
A new properties file 'common-version-info.properties' with ${} variables is available in src/main/resources:
{code}
version=${pom.version}
revision=${version-info.scm.commit}
branch=${version-info.scm.branch}
user=${user.name}
date=${version-info.build.time}
url=${version-info.scm.uri}
srcChecksum=${version-info.source.md5}
{code}
This file is copied into target using filtering and the variables are resolved at copy time.
All the variables that start with version-info are resolved and injected into the project by the new plugin hadoop-maven-plugins using the goal version-info:
{code}
...
<plugin>
<groupId>org.apache.hadoop.maven.plugin</groupId>
<artifactId>hadoop-maven-plugins</artifactId>
<executions>
<execution>
<id>version-info</id>
<goals>
<goal>version-info</goal>
</goals>
<configuration>
<source>
<directory>${basedir}/src/main/java</directory>
<includes>
<include>**/*.java</include>
</includes>
</source>
</configuration>
</execution>
</executions>
</plugin>
...
{code}
Because the plugin is not yet available, to test it you'll have to build and install it locally. The plugin source is avail at:
https://github.com/tucu00/hadoop-maven-plugins
After cloning it, running 'mvn install' will install it locally.
You'll need to apply HADOOP-9116 before this patch in order to get things to work otherwise the sources JAR ends up in the classpath with the unresolved properties files and it may be first in the classpath.
I've tested in OSX and Windows and seems to be working as expected.
I still need to verify the MD5 generated is correct.
I think the current way VersionInfo works (only getting the md5 of common is not correct), but this is out of scope from this JIRA. Also, the YARN variant seems not be used from the command line when doing 'yarn version'.
Regarding the maven plugins, we have to figure how we'll build/publish them to a Maven repo in order to use them. I personally think they should be a sub-project of Hadoop to allow us to release new ones independent of Hadoop. Why a Hadoop sub-project and not a general project? IMO, the focus of these plugins (including HADOOP-8887 for cmake and the protoc that is already avail in the GH repo, another JIRA coming soon) is to solve Hadoop build problems.
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Alejandro Abdelnur
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch, HADOOP-8924.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477100#comment-13477100 ]
Suresh Srinivas commented on HADOOP-8924:
-----------------------------------------
+1 for the patch. I will commit it to branch-trunk-win.
> Hadoop Common creating package-info.java must not depend on sh
> --------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth updated HADOOP-8924:
----------------------------------
Summary: Hadoop Common creating package-info.java must not depend on sh (was: Hadoop Common creating version annotation must not depend on sh)
> Hadoop Common creating package-info.java must not depend on sh
> --------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501428#comment-13501428 ]
Radim Kolar commented on HADOOP-8924:
-------------------------------------
Its agreed to merge this into trunk? I can prepare and test version of this patch.
> Hadoop Common creating package-info.java must not depend on sh
> --------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477519#comment-13477519 ]
Matt Foley commented on HADOOP-8924:
------------------------------------
Hi Aaron, I advised Chris to use Python. As a general matter I think we should stay cross-platform wherever it makes sense to do so. In other words, if something is obviously totally platform-dependent, then go ahead and do two versions, one in shell for Linux, and one in powershell or cmd for Windows. However, where it can be platform-independent, I think we should use a platform-independent scripting language to write a single script (which may include some conditional code for platform-dependent cases if necessary). Obviously it works fine in this case.
Python is only one possibility, of course. However it seemed a reasonable choice. It's object-oriented, with reasonable IDE support. It's free (like beer and speech), available on essentially all platforms, and in good odour with the OSS community. The only other evident candidate would be Ruby, but I think more people know Python than Ruby, altho I can't substantiate that. (Altho there is the [Tiobe Index|http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html], which I don't claim is authoritative.)
This is worth discussing, because as we integrate the Windows port, we'll have lots of opportunity to follow whatever model we agree on. Shall we continue here, or pop it to a common-dev thread?
And I agree with you that whatever answer we agree on should be documented as a build dependency.
> Hadoop Common creating package-info.java must not depend on sh
> --------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510300#comment-13510300 ]
Matt Foley commented on HADOOP-8924:
------------------------------------
-1. Alejandro, I've said in several threads that this level of complexity is not acceptable for this task.
We have no need to use maven plugins everywhere they could possibly be applied, when a simpler approach will do. Very few people in the community are expert with these plug-ins, and this approach is increasing the opacity, not the simplicity. HADOOP-8887 is a reasonable attempt to do something already complex (native builds) in a way that fits better into the build process. But HADOOP-8887 has been in process for two months, and apparently is still not ready for prime time. Capturing the version information should be simple, and this approach isn't simple at all. Furthermore, the underlying plugin (referenced from github) is 3 files with 374 source loc, not simple and transparent.
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Alejandro Abdelnur
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch, HADOOP-8924.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477200#comment-13477200 ]
Aaron T. Myers commented on HADOOP-8924:
----------------------------------------
Hey guys, does it really make sense to trade a dependency on sh for a dependency on python? Maybe it does, but at least on Unix systems I feel like sh is more likely to be available than python. (Honest question here - not trying to be a pain.)
At the very least, if we stick with this being in python, we should update BUILDING.txt to say that we now have a dependency on python (and perhaps some specific version of python?) in order to build Hadoop.
> Hadoop Common creating package-info.java must not depend on sh
> --------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477557#comment-13477557 ]
Aaron T. Myers commented on HADOOP-8924:
----------------------------------------
Hi Matt, a discussion on common-dev@ makes sense to me. I don't feel strongly about what the right answer is, it just wasn't obvious to me that it's python. If it's determined to be python, then that's fine by me.
And yes, whenever this is resolved, we should definitely file a JIRA to update BUILDING.txt accordingly.
> Hadoop Common creating package-info.java must not depend on sh
> --------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8924:
-------------------------------
Summary: Hadoop Common creating package-info.java must not depend on sh, at least for Windows (was: Hadoop Common creating package-info.java must not depend on sh)
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510587#comment-13510587 ]
Radim Kolar commented on HADOOP-8924:
-------------------------------------
I voted for Groovy or some other language in pom.xml, but Matt is -1 on groovy in maven.
My understanding of 2nd vote option was to use an existing maven plugin and combine it with some language which JVM provides. There is couple of options available.
I offered to rework python code into Groovy in maven, because its high priority for me to get builds working on windows. I can rework it into pretty much anything just you guys need to decide what direction you want to go.
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Alejandro Abdelnur
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch, HADOOP-8924.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth updated HADOOP-8924:
----------------------------------
Assignee: Alejandro Abdelnur (was: Chris Nauroth)
Thanks, Alejandro. I've reassigned it to you. I can volunteer to code review it when the patch is available.
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Alejandro Abdelnur
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas resolved HADOOP-8924.
-------------------------------------
Resolution: Fixed
Fix Version/s: trunk-win
Hadoop Flags: Reviewed
I committed the patch to branch-trunk-win. Thank you Chris.
> Hadoop Common creating package-info.java must not depend on sh
> --------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510292#comment-13510292 ]
Alejandro Abdelnur commented on HADOOP-8924:
--------------------------------------------
Along similar lines, I've created and put a patch for protoc using a maven plugin, HADOOP-9117.
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Alejandro Abdelnur
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch, HADOOP-8924.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510667#comment-13510667 ]
Radim Kolar commented on HADOOP-8924:
-------------------------------------
it might be workable. Store saveVersion.java into separate directory but same module as source files, then bind maven-compiler and maven-exec plugins to generate-sources or process-sources phase.
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Alejandro Abdelnur
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch, HADOOP-8924.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth updated HADOOP-8924:
----------------------------------
Attachment: HADOOP-8924-branch-trunk-win.patch
This patch converts saveVersion.sh to an equivalent saveVersion.py. I also expanded the template and the arguments to include additional fields, so that the same script can be used from both the Hadoop Common and Yarn modules.
I have tested "mvn generate-sources" in the following combinations:
1. Mac/git
2. Windows/git
3. Windows + Cygwin/git
4. Mac/svn
5. Windows/svn
6. Windows + Cygwin/svn
All tests produced the correct package-info.java, with the same value for srcChecksum.
I'm starting this change in branch-trunk-win, because the driver for this change is to get a working Windows build for a branch off of trunk.
> Hadoop Common creating package-info.java must not depend on sh
> --------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510565#comment-13510565 ]
Alejandro Abdelnur commented on HADOOP-8924:
--------------------------------------------
If I'm not mistaken the votes for using Maven plugins didn't have a single -1s, including a +1 from you Randim.
Writing Maven plugins is more complex that writing scripts, I don't dispute this. The main motivations for using Maven plugins is to keep things in the POM declarative and to hide (if necessary) different handlings for different platforms.
IMO, having a bunch of stand alone java programs that do this will have the same complexity than a Maven plugin, plus we'll lose all the benefits from Maven, version management, distribution/download via maven repo, declarative builds.
Regarding HADOOP-8887, it has been sitting around for a while, not because it is incomplete or complex but because we are not solving how the plugin gets into a Maven repo so it is available for the Hadoop build just to download it. I thinking we should have a discussion in common-dev@ on how to address this.
Going back to this particular JIRA, the other approach is to do it in the way Oozie does it:
For building a release tarball, there is a script 'mkdistro' which computes the SCM (SVN or GIT) info (uri, branch, commit), the build timestamp and passes it the Maven invocation as -D options, then they are avail as properties in the POM and the version-info.properties file gets filtered with those.
Essentially is this patch minus the plugin plus a script to call Maven.
The drawback of this approach is that the script has to do very much what saveVersion.sh does, meaning we are at square one.
We never tackled this in Oozie as a plugin because, so far, nobody as attempted to build/run Oozie server in Windows. I assume in Oozie we'd go the plugin way when such need arises.
I'm OK in updating this patch to do things how Oozie does it today, using bash. For how to do that in Windows command without using cygwin, I don't know. Again, I think we are back to square one with this approach, we are just the script logic of saveVersion.sh around.
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Alejandro Abdelnur
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch, HADOOP-8924.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510022#comment-13510022 ]
Chris Nauroth commented on HADOOP-8924:
---------------------------------------
On branch-trunk-win, when we converted to Python, I consolidated to a single copy of saveVersion.py in /dev-support, which is used by both sub-modules. I needed to introduce some additional arguments to the script so that it could be parameterized differently for hadoop-common vs. hadoop-yarn-common. It didn't add much complexity, and it eliminated the need to maintain 2 different copies of the same code. We could take the same approach to parameterize the equivalent Java code (or whatever else).
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510543#comment-13510543 ]
Radim Kolar commented on HADOOP-8924:
-------------------------------------
I also do not like custom maven plugin. Can't it be reworked into separate java program?
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Alejandro Abdelnur
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch, HADOOP-8924.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8924) Hadoop Common creating
package-info.java must not depend on sh, at least for Windows
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510187#comment-13510187 ]
Alejandro Abdelnur commented on HADOOP-8924:
--------------------------------------------
Chris, As I've mentioned in the aliases earlier today, I've got a Maven plugin taking care of saveVersion.sh, I've tested in Windows, OSX, Linux. Would you mind reassigning this JIRA to me and I'll put a patch?
Thx
> Hadoop Common creating package-info.java must not depend on sh, at least for Windows
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-8924
> URL: https://issues.apache.org/jira/browse/HADOOP-8924
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: HADOOP-8924-branch-trunk-win.patch
>
>
> Currently, the build process relies on saveVersion.sh to generate package-info.java with a version annotation. The sh binary may not be available on all developers' machines (e.g. Windows without Cygwin). This issue tracks removal of that dependency in Hadoop Common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira