You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Szehon Ho <sz...@cloudera.com> on 2013/12/06 00:04:43 UTC
Review Request 16053: HIVE-5966: Fix eclipse:eclipse post shim aggregation
changes
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16053/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-5966
https://issues.apache.org/jira/browse/HIVE-5966
Repository: hive-git
Description
-------
After shims aggregator changes, mvn install would work because all projects had dependency to the hadoop-shims uberjar, but eclipse plugin fails because it did not understand that dependency.
This patch eliminates the shims uberjar and replacing it with an shims-aggregator project, so that both mvn eclipse:eclipse and mvn install will work.
Chose only shims-common and shims-common-secure as compile time dependency in new shims-aggregator project, the shims-version are runtime dependency. The hive projects are already declaring explicit compile-time dependencies to shims-version dependencies. (In fact, maybe we dont even need dependency from shims-aggregator to shims-version?)
Diffs
-----
shims/aggregator/pom.xml PRE-CREATION
shims/assembly/pom.xml a03ebff
shims/assembly/src/assemble/uberjar.xml 3eba66d
shims/pom.xml 8c14d9b
Diff: https://reviews.apache.org/r/16053/diff/
Testing
-------
Basic sanity tests with hadoop-2 profile:
Compiled and imported projects to eclipse, verified no error.
Ran a qfile test.
Built the assembly and ran a basic MR query.
Thanks,
Szehon Ho
Re: Review Request 16053: HIVE-5966: Fix eclipse:eclipse post shim
aggregation changes
Posted by Szehon Ho <sz...@cloudera.com>.
> On Dec. 5, 2013, 11:36 p.m., Brock Noland wrote:
> > "The hive projects are already declaring explicit compile-time dependencies to shims-version dependencies. (In fact, maybe we don't even need dependency from shims-aggregator to shims-version?)"
> >
> > Which projects? I see this in the ql module?
> >
> > Also in the ql/pom.xml you'll see us creating an uber jar. Are all the shims still included in the output jar?
Sorry , ignore that first comment.
Looks like after changing the shims from uber-jar to aggregator project, the ql uber-jar no longer contains the right shims as you mentioned. Any ideas?
- Szehon
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16053/#review29849
-----------------------------------------------------------
On Dec. 5, 2013, 11:04 p.m., Szehon Ho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16053/
> -----------------------------------------------------------
>
> (Updated Dec. 5, 2013, 11:04 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-5966
> https://issues.apache.org/jira/browse/HIVE-5966
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> After shims aggregator changes, mvn install would work because all projects had dependency to the hadoop-shims uberjar, but eclipse plugin fails because it did not understand that dependency.
>
> This patch eliminates the shims uberjar and replacing it with an shims-aggregator project, so that both mvn eclipse:eclipse and mvn install will work.
>
> Chose only shims-common and shims-common-secure as compile time dependency in new shims-aggregator project, the shims-version are runtime dependency. The hive projects are already declaring explicit compile-time dependencies to shims-version dependencies. (In fact, maybe we dont even need dependency from shims-aggregator to shims-version?)
>
>
> Diffs
> -----
>
> shims/aggregator/pom.xml PRE-CREATION
> shims/assembly/pom.xml a03ebff
> shims/assembly/src/assemble/uberjar.xml 3eba66d
> shims/pom.xml 8c14d9b
>
> Diff: https://reviews.apache.org/r/16053/diff/
>
>
> Testing
> -------
>
> Basic sanity tests with hadoop-2 profile:
>
> Compiled and imported projects to eclipse, verified no error.
> Ran a qfile test.
> Built the assembly and ran a basic MR query.
>
>
> Thanks,
>
> Szehon Ho
>
>
Re: Review Request 16053: HIVE-5966: Fix eclipse:eclipse post shim
aggregation changes
Posted by Brock Noland <br...@cloudera.com>.
> On Dec. 5, 2013, 11:36 p.m., Brock Noland wrote:
> > "The hive projects are already declaring explicit compile-time dependencies to shims-version dependencies. (In fact, maybe we don't even need dependency from shims-aggregator to shims-version?)"
> >
> > Which projects? I see this in the ql module?
> >
> > Also in the ql/pom.xml you'll see us creating an uber jar. Are all the shims still included in the output jar?
>
> Szehon Ho wrote:
> Sorry , ignore that first comment.
>
> Looks like after changing the shims from uber-jar to aggregator project, the ql uber-jar no longer contains the right shims as you mentioned. Any ideas?
>
> Szehon Ho wrote:
> I manually included the 5 shims projects (0.20, 0.23, 0.23S, commons, and commons-secure) in the ql's project shade plugin where it constructs the uber-jar. Is that the best approach? Thanks.
That works! I think it takes wildcards hive-shims-* but I am totally fine with this.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16053/#review29849
-----------------------------------------------------------
On Dec. 6, 2013, 2:13 a.m., Szehon Ho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16053/
> -----------------------------------------------------------
>
> (Updated Dec. 6, 2013, 2:13 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-5966
> https://issues.apache.org/jira/browse/HIVE-5966
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> After shims aggregator changes, mvn install would work because all projects had dependency to the hadoop-shims uberjar, but eclipse plugin fails because it did not understand that dependency.
>
> This patch eliminates the shims uberjar and replacing it with an shims-aggregator project, so that both mvn eclipse:eclipse and mvn install will work.
>
> Chose only shims-common and shims-common-secure as compile time dependency in new shims-aggregator project, the shims-version are runtime dependency. The hive projects are already declaring explicit compile-time dependencies to shims-version dependencies. (In fact, maybe we dont even need dependency from shims-aggregator to shims-version?)
>
>
> Diffs
> -----
>
> ql/pom.xml fc65608
> shims/aggregator/pom.xml PRE-CREATION
> shims/assembly/pom.xml a03ebff
> shims/assembly/src/assemble/uberjar.xml 3eba66d
> shims/pom.xml 8c14d9b
>
> Diff: https://reviews.apache.org/r/16053/diff/
>
>
> Testing
> -------
>
> Basic sanity tests with hadoop-2 profile:
>
> Compiled and imported projects to eclipse, verified no error.
> Ran a qfile test.
> Built the assembly and ran a basic MR query.
>
>
> Thanks,
>
> Szehon Ho
>
>
Re: Review Request 16053: HIVE-5966: Fix eclipse:eclipse post shim
aggregation changes
Posted by Szehon Ho <sz...@cloudera.com>.
> On Dec. 5, 2013, 11:36 p.m., Brock Noland wrote:
> > "The hive projects are already declaring explicit compile-time dependencies to shims-version dependencies. (In fact, maybe we don't even need dependency from shims-aggregator to shims-version?)"
> >
> > Which projects? I see this in the ql module?
> >
> > Also in the ql/pom.xml you'll see us creating an uber jar. Are all the shims still included in the output jar?
>
> Szehon Ho wrote:
> Sorry , ignore that first comment.
>
> Looks like after changing the shims from uber-jar to aggregator project, the ql uber-jar no longer contains the right shims as you mentioned. Any ideas?
I manually included the 5 shims projects (0.20, 0.23, 0.23S, commons, and commons-secure) in the ql's project shade plugin where it constructs the uber-jar. Is that the best approach? Thanks.
- Szehon
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16053/#review29849
-----------------------------------------------------------
On Dec. 5, 2013, 11:04 p.m., Szehon Ho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16053/
> -----------------------------------------------------------
>
> (Updated Dec. 5, 2013, 11:04 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-5966
> https://issues.apache.org/jira/browse/HIVE-5966
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> After shims aggregator changes, mvn install would work because all projects had dependency to the hadoop-shims uberjar, but eclipse plugin fails because it did not understand that dependency.
>
> This patch eliminates the shims uberjar and replacing it with an shims-aggregator project, so that both mvn eclipse:eclipse and mvn install will work.
>
> Chose only shims-common and shims-common-secure as compile time dependency in new shims-aggregator project, the shims-version are runtime dependency. The hive projects are already declaring explicit compile-time dependencies to shims-version dependencies. (In fact, maybe we dont even need dependency from shims-aggregator to shims-version?)
>
>
> Diffs
> -----
>
> shims/aggregator/pom.xml PRE-CREATION
> shims/assembly/pom.xml a03ebff
> shims/assembly/src/assemble/uberjar.xml 3eba66d
> shims/pom.xml 8c14d9b
>
> Diff: https://reviews.apache.org/r/16053/diff/
>
>
> Testing
> -------
>
> Basic sanity tests with hadoop-2 profile:
>
> Compiled and imported projects to eclipse, verified no error.
> Ran a qfile test.
> Built the assembly and ran a basic MR query.
>
>
> Thanks,
>
> Szehon Ho
>
>
Re: Review Request 16053: HIVE-5966: Fix eclipse:eclipse post shim
aggregation changes
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16053/#review29849
-----------------------------------------------------------
"The hive projects are already declaring explicit compile-time dependencies to shims-version dependencies. (In fact, maybe we don't even need dependency from shims-aggregator to shims-version?)"
Which projects? I see this in the ql module?
Also in the ql/pom.xml you'll see us creating an uber jar. Are all the shims still included in the output jar?
- Brock Noland
On Dec. 5, 2013, 11:04 p.m., Szehon Ho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16053/
> -----------------------------------------------------------
>
> (Updated Dec. 5, 2013, 11:04 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-5966
> https://issues.apache.org/jira/browse/HIVE-5966
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> After shims aggregator changes, mvn install would work because all projects had dependency to the hadoop-shims uberjar, but eclipse plugin fails because it did not understand that dependency.
>
> This patch eliminates the shims uberjar and replacing it with an shims-aggregator project, so that both mvn eclipse:eclipse and mvn install will work.
>
> Chose only shims-common and shims-common-secure as compile time dependency in new shims-aggregator project, the shims-version are runtime dependency. The hive projects are already declaring explicit compile-time dependencies to shims-version dependencies. (In fact, maybe we dont even need dependency from shims-aggregator to shims-version?)
>
>
> Diffs
> -----
>
> shims/aggregator/pom.xml PRE-CREATION
> shims/assembly/pom.xml a03ebff
> shims/assembly/src/assemble/uberjar.xml 3eba66d
> shims/pom.xml 8c14d9b
>
> Diff: https://reviews.apache.org/r/16053/diff/
>
>
> Testing
> -------
>
> Basic sanity tests with hadoop-2 profile:
>
> Compiled and imported projects to eclipse, verified no error.
> Ran a qfile test.
> Built the assembly and ran a basic MR query.
>
>
> Thanks,
>
> Szehon Ho
>
>
Re: Review Request 16053: HIVE-5966: Fix eclipse:eclipse post shim
aggregation changes
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16053/#review29868
-----------------------------------------------------------
Ship it!
Ship It!
- Brock Noland
On Dec. 6, 2013, 2:13 a.m., Szehon Ho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16053/
> -----------------------------------------------------------
>
> (Updated Dec. 6, 2013, 2:13 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-5966
> https://issues.apache.org/jira/browse/HIVE-5966
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> After shims aggregator changes, mvn install would work because all projects had dependency to the hadoop-shims uberjar, but eclipse plugin fails because it did not understand that dependency.
>
> This patch eliminates the shims uberjar and replacing it with an shims-aggregator project, so that both mvn eclipse:eclipse and mvn install will work.
>
> Chose only shims-common and shims-common-secure as compile time dependency in new shims-aggregator project, the shims-version are runtime dependency. The hive projects are already declaring explicit compile-time dependencies to shims-version dependencies. (In fact, maybe we dont even need dependency from shims-aggregator to shims-version?)
>
>
> Diffs
> -----
>
> ql/pom.xml fc65608
> shims/aggregator/pom.xml PRE-CREATION
> shims/assembly/pom.xml a03ebff
> shims/assembly/src/assemble/uberjar.xml 3eba66d
> shims/pom.xml 8c14d9b
>
> Diff: https://reviews.apache.org/r/16053/diff/
>
>
> Testing
> -------
>
> Basic sanity tests with hadoop-2 profile:
>
> Compiled and imported projects to eclipse, verified no error.
> Ran a qfile test.
> Built the assembly and ran a basic MR query.
>
>
> Thanks,
>
> Szehon Ho
>
>
Re: Review Request 16053: HIVE-5966: Fix eclipse:eclipse post shim
aggregation changes
Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16053/
-----------------------------------------------------------
(Updated Dec. 6, 2013, 2:13 a.m.)
Review request for hive.
Changes
-------
Manually adding the 5 shims projects to the ql uber-jar. Let me know if there is a better way to do this.
Bugs: HIVE-5966
https://issues.apache.org/jira/browse/HIVE-5966
Repository: hive-git
Description
-------
After shims aggregator changes, mvn install would work because all projects had dependency to the hadoop-shims uberjar, but eclipse plugin fails because it did not understand that dependency.
This patch eliminates the shims uberjar and replacing it with an shims-aggregator project, so that both mvn eclipse:eclipse and mvn install will work.
Chose only shims-common and shims-common-secure as compile time dependency in new shims-aggregator project, the shims-version are runtime dependency. The hive projects are already declaring explicit compile-time dependencies to shims-version dependencies. (In fact, maybe we dont even need dependency from shims-aggregator to shims-version?)
Diffs (updated)
-----
ql/pom.xml fc65608
shims/aggregator/pom.xml PRE-CREATION
shims/assembly/pom.xml a03ebff
shims/assembly/src/assemble/uberjar.xml 3eba66d
shims/pom.xml 8c14d9b
Diff: https://reviews.apache.org/r/16053/diff/
Testing
-------
Basic sanity tests with hadoop-2 profile:
Compiled and imported projects to eclipse, verified no error.
Ran a qfile test.
Built the assembly and ran a basic MR query.
Thanks,
Szehon Ho