You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Ning Zhang <nz...@fb.com> on 2011/04/06 17:05:25 UTC
Review Request: HIVE-2082. Reduce memory consumption in preparing MapReduce
job
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/556/
-----------------------------------------------------------
Review request for hive.
Summary
-------
The major change is to construct PartitionDesc from TableDesc and reuse the column info from the TableDesc.
Diffs
-----
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1087411
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1087411
trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 1087411
trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 1087411
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1087411
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java 1087411
trunk/ql/src/test/results/clientpositive/combine2.q.out 1087411
trunk/ql/src/test/results/clientpositive/merge3.q.out 1087411
trunk/ql/src/test/results/clientpositive/pcr.q.out 1087411
trunk/ql/src/test/results/clientpositive/sample10.q.out 1087411
trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1087411
Diff: https://reviews.apache.org/r/556/diff
Testing
-------
passed all unit tests.
Thanks,
Ning
Re: Review Request: HIVE-2082. Reduce memory consumption in preparing
MapReduce job
Posted by Ning Zhang <nz...@fb.com>.
> On 2011-04-06 20:36:23, namit jain wrote:
> > Also, why should the test outputs be different.
> >
The previous test outputs are wrong due to a bug: previously partition properties are set by the table level property. Now we use partition level property. If you look at one of the plan diffs (e.g., combine2.q.out), the partition level stats previously are the same as the table level stats. Now they are different for different partitions.
- Ning
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/556/#review396
-----------------------------------------------------------
On 2011-04-06 15:05:25, Ning Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/556/
> -----------------------------------------------------------
>
> (Updated 2011-04-06 15:05:25)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> The major change is to construct PartitionDesc from TableDesc and reuse the column info from the TableDesc.
>
>
> Diffs
> -----
>
> trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1087411
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1087411
> trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 1087411
> trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 1087411
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1087411
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java 1087411
> trunk/ql/src/test/results/clientpositive/combine2.q.out 1087411
> trunk/ql/src/test/results/clientpositive/merge3.q.out 1087411
> trunk/ql/src/test/results/clientpositive/pcr.q.out 1087411
> trunk/ql/src/test/results/clientpositive/sample10.q.out 1087411
> trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1087411
>
> Diff: https://reviews.apache.org/r/556/diff
>
>
> Testing
> -------
>
> passed all unit tests.
>
>
> Thanks,
>
> Ning
>
>
Re: Review Request: HIVE-2082. Reduce memory consumption in preparing
MapReduce job
Posted by namit jain <nj...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/556/#review396
-----------------------------------------------------------
Also, why should the test outputs be different.
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
<https://reviews.apache.org/r/556/#comment746>
- namit
On 2011-04-06 15:05:25, Ning Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/556/
> -----------------------------------------------------------
>
> (Updated 2011-04-06 15:05:25)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> The major change is to construct PartitionDesc from TableDesc and reuse the column info from the TableDesc.
>
>
> Diffs
> -----
>
> trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1087411
> trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1087411
> trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 1087411
> trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 1087411
> trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1087411
> trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java 1087411
> trunk/ql/src/test/results/clientpositive/combine2.q.out 1087411
> trunk/ql/src/test/results/clientpositive/merge3.q.out 1087411
> trunk/ql/src/test/results/clientpositive/pcr.q.out 1087411
> trunk/ql/src/test/results/clientpositive/sample10.q.out 1087411
> trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1087411
>
> Diff: https://reviews.apache.org/r/556/diff
>
>
> Testing
> -------
>
> passed all unit tests.
>
>
> Thanks,
>
> Ning
>
>