You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Aaron Kimball (JIRA)" <ji...@apache.org> on 2010/01/13 04:13:54 UTC
[jira] Created: (AVRO-314) Allow Avro to publish to a local m2
repository
Allow Avro to publish to a local m2 repository
----------------------------------------------
Key: AVRO-314
URL: https://issues.apache.org/jira/browse/AVRO-314
Project: Avro
Issue Type: Improvement
Components: build, java
Reporter: Aaron Kimball
To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-314:
------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Aaron.
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 1.3.0
>
> Attachments: AVRO-314.2.patch, AVRO-314.3.patch, AVRO-314.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated AVRO-314:
-------------------------------
Attachment: (was: AVRO-314.2.patch)
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.2.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated AVRO-314:
-------------------------------
Status: Open (was: Patch Available)
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.2.patch, AVRO-314.3.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated AVRO-314:
-------------------------------
Status: Open (was: Patch Available)
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.2.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated AVRO-314:
-------------------------------
Status: Patch Available (was: Open)
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.2.patch, AVRO-314.3.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated AVRO-314:
-------------------------------
Attachment: AVRO-314.3.patch
In general, I'm a fan of writing one patch rather than three if at all possible. This now uses ivy to grab mvn-ant-tasks.
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.2.patch, AVRO-314.3.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated AVRO-314:
-------------------------------
Attachment: AVRO-314.patch
Here's a patch that handles this, using code very similar to that in Hadoop (core, hdfs).
To use this:
{code}
$ cd $AVRO_HOME/lang/java
$ ant mvn-install
$ cd $MAPREDUCE_HOME # or another project
$ ant veryclean jar -Dresolvers=internal
{code}
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated AVRO-314:
-------------------------------
Status: Patch Available (was: Open)
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.2.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799993#action_12799993 ]
Aaron Kimball commented on AVRO-314:
------------------------------------
+1 to that patch
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 1.3.0
>
> Attachments: AVRO-314.2.patch, AVRO-314.3.patch, AVRO-314.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-314:
------------------------------
Status: Patch Available (was: Open)
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 1.3.0
>
> Attachments: AVRO-314.2.patch, AVRO-314.3.patch, AVRO-314.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated AVRO-314:
-------------------------------
Attachment: AVRO-314.2.patch
Doug,
This mirrors the framework already in use via HADOOP-5107. Hadoop already has an established resolver set which will read from local a mvn repository; no further modifications to those three projects are required if we integrate with their toolchain on this end. This is, for example, the accepted way to test how changes made to hadoop-core affect the mapreduce project. In any case, I think this is superior to manually copying build artifacts to specific directories.
As for your other comments -- all good suggestions. I had missed the auto-generated pom. I've removed the static pom file, and cleaned up the other things you mentioned.
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.2.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799942#action_12799942 ]
Doug Cutting commented on AVRO-314:
-----------------------------------
We don't have to rigidly emulate Hadoop's Ivy/Maven toolchain. Hadoop already uses Ivy to retrieve its dependencies, so using the local Ivy repo imposes little on Hadoop. Is there another reason why you prefer to publish to Maven's local repo rather than Ivy's?
> In any case, I think this is superior to manually copying build artifacts to specific directories.
I agree. That's why I suggested we try Ivy's publish task before we resort to using Maven.
If you're stuck on not changing Hadoop and conforming to that build, can we at least retrieve maven-ant-tasks using Ivy as a part of our build configuration? This should be a one-line change to Avro's ivy.xml, no?
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.2.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated AVRO-314:
-------------------------------
Status: Patch Available (was: Open)
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-314:
------------------------------
Attachment: AVRO-314.patch
Here's a version that addresses these issues.
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 1.3.0
>
> Attachments: AVRO-314.2.patch, AVRO-314.3.patch, AVRO-314.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated AVRO-314:
-------------------------------
Attachment: AVRO-314.2.patch
Re-uploading patch with {{--no-prefix}}
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.2.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-314:
------------------------------
Fix Version/s: 1.3.0
Assignee: Aaron Kimball
Status: Open (was: Patch Available)
This is really close:
- the ant tasks don't need to explicitly depend on tasks that their dependencies already depend on
- we shouldn't have the maven-ant-tasks version number both in ivy.xml and build.xml -- that's two places to update when we upgrade.
- we shouldn't cut-and-paste the pom-creation around, but rather make it a task.
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 1.3.0
>
> Attachments: AVRO-314.2.patch, AVRO-314.3.patch, AVRO-314.patch, AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-314) Allow Avro to publish to a local m2
repository
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799833#action_12799833 ]
Doug Cutting commented on AVRO-314:
-----------------------------------
It seems circuitous to me to use Maven to publish things locally so that Ivy can read them. Ivy should be able to publish locally to Ivy, no? I got this to work by manually copying jars to my local ivy repo (~/.ivy2/local/org.apache.hadoop/avro/1.2.0/jars/avro.jar) then modifying Hadoop's "internal" resolvers in its ivysettings.xml files to include "local", as in HADOOP-6486 and HDFS-892.
Have you tried using ivy's publish task to write Avro's jar to ivy's local resolver and configuring Hadoop to use Ivy's local resolver to read it?
Other comments on the patch:
- you intermix two .gitignore styles: one at top-level or per-directory. pick one. i prefer per-directory.
- downloaded jars should go in the lib/ directory, not in the ivy/ directory, no?
- we use Ivy to generate Avro's pom file. if we must use Maven to publish, can you please generate the pom this way? we should not maintain our dependencies two places, ivy.xml and avro.pom.
> Allow Avro to publish to a local m2 repository
> ----------------------------------------------
>
> Key: AVRO-314
> URL: https://issues.apache.org/jira/browse/AVRO-314
> Project: Avro
> Issue Type: Improvement
> Components: build, java
> Reporter: Aaron Kimball
> Attachments: AVRO-314.patch
>
>
> To test Hadoop or other systems with a locally-built avro jar, it is considerably easier if you can use a local maven repository to resolve dependencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.