[jira] Updated: (AVRO-647) Break avro.jar into avro.jar, avro-dev.jar and avro-hadoop.jar

["Scott Carey (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AVRO-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Carey updated AVRO-647:
-----------------------------

    Attachment: AVRO-647.patch
                migrateAvro.sh

h3.  Avro maven build patch and jar split-up.
This is a mostly complete patch for splitting the Java portion of the avro project up into 6 sub-projects.
This requires a maven plugin, and so solves AVRO-159 and part of AVRO-572 as well.

The project structure is as follows:

|| path from lang/java | artifactId | name | artifact type | notes |
| / | avro-parent | Apache Avro Parent | pom | parent, inherits from Apache master pom, sets common build properties and versions |
| /avro/ | avro |Apache Avro | jar | discussed as "avro-core" previously |
| /compiler/ | avro-compiler | Apache Avro Compiler | jar | Avro IDL compiler and Specific compiler, including ant tasks | 
| /maven-plugin/ | avro-maven-plugin | Apache Avro Maven Plugin | maven-plugin | Maven mojos for avpr > java; avsc > java; avdl > java; |
| /ipc/ | avro-ipc | Apache Avro IPC | jar | Avro IPC components, protocols, trancievers, etc |
| /mapred/ | avro-mapred | Apache Avro Mapred API | jar | An org.apache.hadoop.mapred API using Avro serialization |
| /tools/ | avro-tools | Apache Avro Tools | jar (with dependencies) | A single jar containing all of Avro and dependencies, with command line tools |

h4. Status
* Compiles, all tests run other than some IPC ones I need help with.  Those don't work for me (and have not for 6+ months on my machine).
* This is not integrated with the other language builds yet.  There is a little work left there to tie the master buld to this.
* This does not yet delete the old directory structure, so side-by-side comparrison is possible.
* There are other changes / enhancements to the build and test process that can leverage this.  I'm trying to get a commit done with the basics soon, we can open other tickets up for cleanup and enhancements.  This is a big checkin with guaranteed merge issues.  If we can get most of it in, that will solve the merge difficulties.
* I have not gotten the 'with dependencies' part of avro-tools copmlete,  that should not block reviewers from having a look.

h3. Patching Instructions
Example instructions.  Change to the lang/java directory, run the shell script, then the patch, then add the new items.  The patch and script is based off of lang/java.
{noformat}
$ cd lang/java
$ ./migrateAvro.sh
$ patch -p0 < ../../AVR0-647.patch
$ svn add pom.xml avro/pom.xml compiler/pom.xml maven-plugin/pom.xml ipc/pom.xml mapred/pom.xml tools/pom.xml
$ svn add maven-plugin/src/main/java/org/apache/avro/mojo/*
{noformat}

h3. Building Instructions: command-line

To clean build all components without testing and install them in your local repository:
{noformat}
$ mvn clean install -Dtest=false -DfailIfNoTests=false
{noformat}

To compile only:
{noformat}
$ mvn compile
{noformat}

To run tests:
{noformat}
$ mvn test
{noformat}

To install to local repo, including running tests:
{noformat}
$ mvn install
{noformat}

Other useful mvn commands:
{noformat}
$ mvn clean
$ mvn validate
$ mvn help:effective-pom
$ mvn site
$ mvn generate-resources
{noformat}

To download all available javadoc and source of dependent projects into your local repo:
{noformat}
$ mvn dependency:resolve -Dclassifier=javadoc
$ mvn dependency:resolve -Dclassifier=sources
{noformat}

h3. Building Instructions: Eclipse
Use Eclipse 3.6 Helios: http://www.eclipse.org/downloads/
Use the m2Eclipse plugin, latest version.
* Load the projects into the workspace using the "Import ..." dialog, and select "Existing Maven Projects"
* Select the lang/java directory, and it should show all 7 projects including the parent.  Import all of these.
* After the load and first build, it will not completely compile.  To fix it up to compile, select all of the projects and right-click.  Select *Maven > Update Project Configuration*.

h4. More maven information:

These are a good start:
http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
http://maven.apache.org/guides/introduction/introduction-to-the-pom.html

For those new and experienced, "Maven By Example" is a very good intro -- especially chapters 3+
http://www.sonatype.com/books/mvnex-book/reference/public-book.html

Apache's maven policies, tips, etc:
http://www.apache.org/dev/publishing-maven-artifacts.html#inherit-parent

Plugins used include:
http://mojo.codehaus.org/javacc-maven-plugin/
http://maven.apache.org/plugins/maven-surefire-plugin/
http://maven.apache.org/plugins/maven-checkstyle-plugin/
http://paranamer.codehaus.org/

Other useful plugins:
http://mojo.codehaus.org/build-helper-maven-plugin/usage.html
http://mojo.codehaus.org/cobertura-maven-plugin/
http://maven.apache.org/plugins/maven-shade-plugin/

h4. Documentation
Much of this message is preliminary documentation.  Please comment on it as well. 

> Break avro.jar into avro.jar, avro-dev.jar and avro-hadoop.jar
> --------------------------------------------------------------
>
>                 Key: AVRO-647
>                 URL: https://issues.apache.org/jira/browse/AVRO-647
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>            Reporter: Scott Carey
>            Assignee: Scott Carey
>         Attachments: AVRO-647.patch, migrateAvro.sh
>
>
> Our dependencies are starting to get a little complicated on the Java side.
> I propose we build two (possibly more) jars related to our major dependencies and functions.
> 1. avro.jar  (or perhaps avro-core.jar)
> This contains all of the core avro functionality for _using_ avro as a library.  This excludes the specific compiler, avro idl, and other build-time or development tools, as well as avro packages for third party integration such as hadoop.  This jar should then have a minimal set of dependencies (jackson, jetty, SLF4J ?).
> 2. avro-dev.jar
> This would contain compilers, idl, development tools, etc.  Most applications will not need this, but build systems and developers will.
> 3. avro-hadoop.jar
> This would contain the hadoop API and possibly pig/hive/whatever related to that.  This makes it easier for pig/hive/hadoop to consume avro-core without circular dependencies. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.