You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Gopal V <go...@hortonworks.com> on 2014/02/01 04:05:13 UTC
Review Request 17632: HDFS ZeroCopy Shims for Hive
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17632/
-----------------------------------------------------------
Review request for hive, Gunther Hagleitner and Owen O'Malley.
Summary (updated)
-----------------
HDFS ZeroCopy Shims for Hive
Bugs: HIVE-6346
https://issues.apache.org/jira/browse/HIVE-6346
Repository: hive-git
Description (updated)
-------
Hive Shims for ZeroCopy FS read and Direct ByteBuffer decompression (hadoop/branch-2 changes)
Diffs (updated)
-----
pom.xml 41f5337
ql/pom.xml 7087a4c
shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java ec1f18e
shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java d0ff7d4
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 54c38ee
shims/0.23C/pom.xml PRE-CREATION
shims/0.23C/src/main/java/org/apache/hadoop/hive/shims/Hadoop23CShims.java PRE-CREATION
shims/aggregator/pom.xml 7aa8c4c
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 2b3c6c1
shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java bf9c84f
shims/pom.xml 9843836
Diff: https://reviews.apache.org/r/17632/diff/
Testing (updated)
-------
TPC-DS queries.
Thanks,
Gopal V
Re: Review Request 17632: HDFS ZeroCopy Shims for Hive
Posted by Gopal V <go...@hortonworks.com>.
> On Feb. 3, 2014, 10:27 p.m., Gunther Hagleitner wrote:
> > pom.xml, line 102
> > <https://reviews.apache.org/r/17632/diff/2/?file=462507#file462507line102>
> >
> > I don't think we need another version, do we? for the branch we can just temporarily make the 23 version 2.4.0 until that one is released. then we switch everything over.
> >
> > Is there another reason to keep both?
Redoing the shims to add a zcr shim within this shim so that we can shim better.
> On Feb. 3, 2014, 10:27 p.m., Gunther Hagleitner wrote:
> > shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java, line 526
> > <https://reviews.apache.org/r/17632/diff/2/?file=462515#file462515line526>
> >
> > nit: lots of trailing ws.
fixing in rebase.
- Gopal
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17632/#review33519
-----------------------------------------------------------
On Feb. 1, 2014, 3:05 a.m., Gopal V wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17632/
> -----------------------------------------------------------
>
> (Updated Feb. 1, 2014, 3:05 a.m.)
>
>
> Review request for hive, Gunther Hagleitner and Owen O'Malley.
>
>
> Bugs: HIVE-6346
> https://issues.apache.org/jira/browse/HIVE-6346
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Hive Shims for ZeroCopy FS read and Direct ByteBuffer decompression (hadoop/branch-2 changes)
>
>
> Diffs
> -----
>
> pom.xml 41f5337
> ql/pom.xml 7087a4c
> shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java ec1f18e
> shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java d0ff7d4
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 54c38ee
> shims/0.23C/pom.xml PRE-CREATION
> shims/0.23C/src/main/java/org/apache/hadoop/hive/shims/Hadoop23CShims.java PRE-CREATION
> shims/aggregator/pom.xml 7aa8c4c
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 2b3c6c1
> shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java bf9c84f
> shims/pom.xml 9843836
>
> Diff: https://reviews.apache.org/r/17632/diff/
>
>
> Testing
> -------
>
> TPC-DS queries.
>
>
> Thanks,
>
> Gopal V
>
>
Re: Review Request 17632: HDFS ZeroCopy Shims for Hive
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17632/#review33519
-----------------------------------------------------------
pom.xml
<https://reviews.apache.org/r/17632/#comment63001>
I don't think we need another version, do we? for the branch we can just temporarily make the 23 version 2.4.0 until that one is released. then we switch everything over.
Is there another reason to keep both?
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java
<https://reviews.apache.org/r/17632/#comment63004>
nit: lots of trailing ws.
- Gunther Hagleitner
On Feb. 1, 2014, 3:05 a.m., Gopal V wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17632/
> -----------------------------------------------------------
>
> (Updated Feb. 1, 2014, 3:05 a.m.)
>
>
> Review request for hive, Gunther Hagleitner and Owen O'Malley.
>
>
> Bugs: HIVE-6346
> https://issues.apache.org/jira/browse/HIVE-6346
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Hive Shims for ZeroCopy FS read and Direct ByteBuffer decompression (hadoop/branch-2 changes)
>
>
> Diffs
> -----
>
> pom.xml 41f5337
> ql/pom.xml 7087a4c
> shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java ec1f18e
> shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java d0ff7d4
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 54c38ee
> shims/0.23C/pom.xml PRE-CREATION
> shims/0.23C/src/main/java/org/apache/hadoop/hive/shims/Hadoop23CShims.java PRE-CREATION
> shims/aggregator/pom.xml 7aa8c4c
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 2b3c6c1
> shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java bf9c84f
> shims/pom.xml 9843836
>
> Diff: https://reviews.apache.org/r/17632/diff/
>
>
> Testing
> -------
>
> TPC-DS queries.
>
>
> Thanks,
>
> Gopal V
>
>
Re: Review Request 17632: HDFS ZeroCopy Shims for Hive
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17632/#review33571
-----------------------------------------------------------
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
<https://reviews.apache.org/r/17632/#comment63104>
this comment should probably move 3 lines down.
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/ZeroCopyShims.java
<https://reviews.apache.org/r/17632/#comment63105>
nit: as a const this would look better as CHECK_SUM, NO_CHECK_SUM.
- Gunther Hagleitner
On Feb. 4, 2014, 6:11 a.m., Gopal V wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17632/
> -----------------------------------------------------------
>
> (Updated Feb. 4, 2014, 6:11 a.m.)
>
>
> Review request for hive, Gunther Hagleitner and Owen O'Malley.
>
>
> Bugs: HIVE-6346
> https://issues.apache.org/jira/browse/HIVE-6346
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Hive Shims for ZeroCopy FS read and Direct ByteBuffer decompression (hadoop/branch-2 changes)
>
>
> Diffs
> -----
>
> pom.xml 0af4b08
> shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java ec1f18e
> shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java d0ff7d4
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 54c38ee
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/ZeroCopyShims.java PRE-CREATION
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 2b3c6c1
>
> Diff: https://reviews.apache.org/r/17632/diff/
>
>
> Testing
> -------
>
> TPC-DS queries.
>
>
> Thanks,
>
> Gopal V
>
>
Re: Review Request 17632: HDFS ZeroCopy Shims for Hive
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17632/
-----------------------------------------------------------
(Updated Feb. 4, 2014, 6:11 a.m.)
Review request for hive, Gunther Hagleitner and Owen O'Malley.
Changes
-------
Remove new shim and add the feature-set into a package protected class.
Bugs: HIVE-6346
https://issues.apache.org/jira/browse/HIVE-6346
Repository: hive-git
Description
-------
Hive Shims for ZeroCopy FS read and Direct ByteBuffer decompression (hadoop/branch-2 changes)
Diffs (updated)
-----
pom.xml 0af4b08
shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java ec1f18e
shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java d0ff7d4
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 54c38ee
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/ZeroCopyShims.java PRE-CREATION
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 2b3c6c1
Diff: https://reviews.apache.org/r/17632/diff/
Testing
-------
TPC-DS queries.
Thanks,
Gopal V