You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (Created) (JIRA)" <ji...@apache.org> on 2012/03/16 06:15:43 UTC
[jira] [Created] (PIG-2599) Mavenize Pig
Mavenize Pig
------------
Key: PIG-2599
URL: https://issues.apache.org/jira/browse/PIG-2599
Project: Pig
Issue Type: New Feature
Components: build
Reporter: Daniel Dai
Switch Pig build system from ant to maven.
This is a candidate project for Google summer of code 2012. More information about the program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2012
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2599) Mavenize Pig
Posted by "major seitan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
major seitan updated PIG-2599:
------------------------------
Attachment: maven-pig.1.zip
> Mavenize Pig
> ------------
>
> Key: PIG-2599
> URL: https://issues.apache.org/jira/browse/PIG-2599
> Project: Pig
> Issue Type: New Feature
> Components: build
> Reporter: Daniel Dai
> Labels: gsoc2012
> Attachments: maven-pig.1.zip
>
>
> Switch Pig build system from ant to maven.
> This is a candidate project for Google summer of code 2012. More information about the program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2012
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2599) Mavenize Pig
Posted by "major seitan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503317#comment-13503317 ]
major seitan commented on PIG-2599:
-----------------------------------
As maven is biased towards having one src directory
the shims used to support hadoop23 and hadoop20
most likely have to be changed. I would like to
recommend the refactoring the shims to be maven sub
projects corresponding to implementations needed for
the hadoop versions. (e.g.).
pig-hadoop23
pig-hadoop20
The following context with be placed in the main
pig project
{code:title=ShimContext.java|borderStyle=solid}
ShimContext {
// handle the reflection errors
private static final IShimFactory SHIM_FACTORY = Class.forName("org.apache.pig.backend.hadoop.executionengine.shims.ShimFactoryImpl").newInstance();
static IShimFactory getInstance(){ return SHIM_FACTORY; }
}
{code}
And each project will have its own ShimFactoryImpl implementating
IShimFactory. Where IShimFactory gives access to the the implementations
of the classes (and inner classes) :
PigJobControl
PigMapReduce
PigMapBase
HadoopShims
This is a substantial change so I would need some consensus/direction on how
best to proceed.
> Mavenize Pig
> ------------
>
> Key: PIG-2599
> URL: https://issues.apache.org/jira/browse/PIG-2599
> Project: Pig
> Issue Type: New Feature
> Components: build
> Reporter: Daniel Dai
> Labels: gsoc2012
>
> Switch Pig build system from ant to maven.
> This is a candidate project for Google summer of code 2012. More information about the program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2012
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2599) Mavenize Pig
Posted by "major seitan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503325#comment-13503325 ]
major seitan commented on PIG-2599:
-----------------------------------
Did some digging looks like the shims issue is PIG-2388
The source level is 1.5
build.xml: <property name="javac.version" value="1.5" />
but
org.apache.hadoop:hadoop-annotations:jar:2.0.0-alpha
requires
jdk.tools:jdk.tools:jar:1.6
what is the recommended source to set the compiler to in maven?
> Mavenize Pig
> ------------
>
> Key: PIG-2599
> URL: https://issues.apache.org/jira/browse/PIG-2599
> Project: Pig
> Issue Type: New Feature
> Components: build
> Reporter: Daniel Dai
> Labels: gsoc2012
>
> Switch Pig build system from ant to maven.
> This is a candidate project for Google summer of code 2012. More information about the program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2012
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2599) Mavenize Pig
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503940#comment-13503940 ]
Julien Le Dem commented on PIG-2599:
------------------------------------
Another way to incrementally change the build is to switch to the Maven Ant Tasks for dependency resolution in the current ant based build.
> Mavenize Pig
> ------------
>
> Key: PIG-2599
> URL: https://issues.apache.org/jira/browse/PIG-2599
> Project: Pig
> Issue Type: New Feature
> Components: build
> Reporter: Daniel Dai
> Labels: gsoc2012
>
> Switch Pig build system from ant to maven.
> This is a candidate project for Google summer of code 2012. More information about the program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2012
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2599) Mavenize Pig
Posted by "Dimitris Bousis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244778#comment-13244778 ]
Dimitris Bousis commented on PIG-2599:
--------------------------------------
Hi all,
My name is Dimitris Bousis, currently doing my Master in Computer Engineering & Informatics in University of Patras, Greece. My research interests include cloud & distributed computing with related technologies such as Hadoop, HBase, Cassandra , Pig & Hive. Though i have not started any research activity with the technologies (I plan to do so after the summer), i have taken an elective course in Hadoop,HDFS, HBase & Cassandra during my undergraduate studies.
I am interested in applying for this GsoC 2012 project. I have used Maven in several Jave EE projects i have been involved and i have some experience with Ant as well. I downloaded Pig and explored it's source. Switching to maven would roughly consist of these main tasks :
1) Provide projects dependencies in the main pom.xml file taken from the Ivy's configuration.
2) Process any required resources.
3) Build pig and any required components with the compile scope.
4) Build and run pig tests
5) Provide a packaged version of pig for component reuse through the maven repository distribution.
Please comment this post for anything you consider necessary. Looking forward working with you this summer.
Dimitris Bousis
> Mavenize Pig
> ------------
>
> Key: PIG-2599
> URL: https://issues.apache.org/jira/browse/PIG-2599
> Project: Pig
> Issue Type: New Feature
> Components: build
> Reporter: Daniel Dai
> Labels: gsoc2012
>
> Switch Pig build system from ant to maven.
> This is a candidate project for Google summer of code 2012. More information about the program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2012
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2599) Mavenize Pig
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247476#comment-13247476 ]
Daniel Dai commented on PIG-2599:
---------------------------------
Looks good, need to make sure current targets works:
1. build and package for both hadoop 20 and 23
2. run unit tests, e2e tests
3. build tarball, rpm/deb
4. publish maven artifacts
Looking forward to see your proposal.
> Mavenize Pig
> ------------
>
> Key: PIG-2599
> URL: https://issues.apache.org/jira/browse/PIG-2599
> Project: Pig
> Issue Type: New Feature
> Components: build
> Reporter: Daniel Dai
> Labels: gsoc2012
>
> Switch Pig build system from ant to maven.
> This is a candidate project for Google summer of code 2012. More information about the program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2012
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2599) Mavenize Pig
Posted by "major seitan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506578#comment-13506578 ]
major seitan commented on PIG-2599:
-----------------------------------
[coming soon to a theater near you]
[maven-pig.1.zip|https://issues.apache.org/jira/secure/attachment/12555367/maven-pig.1.zip]
Attached is a preview of the effort
to port pig from ant to maven in hopes
of getting feedback. So if folks can
kick the tires, it would be great.
q. Is there a recommended way to do such review?
q. Is there someone that I can correspond with if I need help?
[no code was harmed]
The approach taken was to migrate the
build.xml to a set of pom.xml.
A shell script is used to move around
directories and fields, and no code was
altered in the migration. So far the
pom's are 100% maven and no ant is used.
q. Should it be ant instead of a shell script?
[let's get the party started]
Instructions:
tested : Ubuntu, MacOS
1. Set JAVA_HOME (tools.jar)
2. Download [jdo2-api-2.3-ec.jar|http://www.datanucleus.org/downloads/maven2/javax/jdo/jdo2-api/2.3-ec/jdo2-api-2.3-ec.jar]
3. mvn install:install-file -DgroupId=javax.jdo -DartifactId=jdo2-api -Dversion=2.3-ec -Dpackaging=jar -Dfile=jdo2-api-2.3-ec.jar
4. Create a work directory
e.g. mkdir pig-2599 && cd pig-2599
5. Download and unzip : maven-pig.1.zip
6. Run
./mvn-pig.sh 2>&1
[remains of the day]
If the script ran sucessfuly there will be two
directories:
pig-ant : containing the ant version
pig-mvn : containing the mvn version
The project has the following layout
pig (pig-mvn)
- pig-core
- pig-bzip2
- pig-shock
- pig-piggybank
- pig-zebra
- pig-tutorial
q. Are the names reasonable?
q. Is the layout reasonable?
[issues]
There is work to be done and feedback is appreciated.
Here are some of the known issues:
- pig-core needs the "mvn clean test" target to run the tests
- jdo2-api2.3-jar has to be manually installed
- e2e, documentation are not migrated yet
- zebra tests are not completed (also zebra fails to compile)
- checkstyle, findbugs not supported
q. Did I miss any thing?
> Mavenize Pig
> ------------
>
> Key: PIG-2599
> URL: https://issues.apache.org/jira/browse/PIG-2599
> Project: Pig
> Issue Type: New Feature
> Components: build
> Reporter: Daniel Dai
> Labels: gsoc2012
> Attachments: maven-pig.1.zip
>
>
> Switch Pig build system from ant to maven.
> This is a candidate project for Google summer of code 2012. More information about the program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2012
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2599) Mavenize Pig
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503938#comment-13503938 ]
Julien Le Dem commented on PIG-2599:
------------------------------------
for the shim layer refactoring you can use this jira: PIG-2686
If you want to change the source level compatibility you should open a JIRA for this.
> Mavenize Pig
> ------------
>
> Key: PIG-2599
> URL: https://issues.apache.org/jira/browse/PIG-2599
> Project: Pig
> Issue Type: New Feature
> Components: build
> Reporter: Daniel Dai
> Labels: gsoc2012
>
> Switch Pig build system from ant to maven.
> This is a candidate project for Google summer of code 2012. More information about the program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2012
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira