You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Deepak Jaiswal <dj...@hortonworks.com> on 2018/07/03 00:23:49 UTC
Review Request 67800: HIVE-20039
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67800/
-----------------------------------------------------------
Review request for hive and Gopal V.
Bugs: HIVE-20039
https://issues.apache.org/jira/browse/HIVE-20039
Repository: hive-git
Description
-------
Bucket pruning: Left Outer Join on bucketed table gives wrong result.
The context was reused by all the predicates. Instead use TS op directly.
Diffs
-----
data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_datajoin_1_s2_2018022300104_1/000000_0 PRE-CREATION
data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_dw_stg_2018022300104_1/000000_0 PRE-CREATION
data/files/bucket_pruning/l3_clarity__l3_snap_number_2018022300104/000000_0 PRE-CREATION
data/files/bucket_pruning/l3_monthly_dw_dimplan/000056_0 PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java 2debacacb5
ql/src/test/queries/clientpositive/tez_fixed_bucket_pruning.q PRE-CREATION
ql/src/test/results/clientpositive/llap/tez_fixed_bucket_pruning.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/67800/diff/1/
Testing
-------
Thanks,
Deepak Jaiswal
Re: Review Request 67800: HIVE-20039
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67800/#review205652
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
Lines 90 (patched)
<https://reviews.apache.org/r/67800/#comment288542>
Is it even possible to change bucket count for a partition of in a table?
As far as I can see bucket number is a table wide property.
- Deepak Jaiswal
On July 3, 2018, 12:23 a.m., Deepak Jaiswal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67800/
> -----------------------------------------------------------
>
> (Updated July 3, 2018, 12:23 a.m.)
>
>
> Review request for hive and Gopal V.
>
>
> Bugs: HIVE-20039
> https://issues.apache.org/jira/browse/HIVE-20039
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Bucket pruning: Left Outer Join on bucketed table gives wrong result.
> The context was reused by all the predicates. Instead use TS op directly.
>
>
> Diffs
> -----
>
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_datajoin_1_s2_2018022300104_1/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_dw_stg_2018022300104_1/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_clarity__l3_snap_number_2018022300104/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_monthly_dw_dimplan/000056_0 PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java 2debacacb5
> ql/src/test/queries/clientpositive/tez_fixed_bucket_pruning.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/tez_fixed_bucket_pruning.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/67800/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Deepak Jaiswal
>
>
Re: Review Request 67800: HIVE-20039
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67800/#review205686
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
Lines 46 (patched)
<https://reviews.apache.org/r/67800/#comment288585>
minor nit
- Gopal V
On July 3, 2018, 8:20 p.m., Deepak Jaiswal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67800/
> -----------------------------------------------------------
>
> (Updated July 3, 2018, 8:20 p.m.)
>
>
> Review request for hive and Gopal V.
>
>
> Bugs: HIVE-20039
> https://issues.apache.org/jira/browse/HIVE-20039
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Bucket pruning: Left Outer Join on bucketed table gives wrong result.
> The context was reused by all the predicates. Instead use TS op directly.
>
>
> Diffs
> -----
>
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_datajoin_1_s2_2018022300104_1/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_dw_stg_2018022300104_1/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_clarity__l3_snap_number_2018022300104/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_monthly_dw_dimplan/000056_0 PRE-CREATION
> itests/src/test/resources/testconfiguration.properties d02c0fe8ba
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java 2debacacb5
> ql/src/test/queries/clientpositive/tez_fixed_bucket_pruning.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/tez_fixed_bucket_pruning.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/67800/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Deepak Jaiswal
>
>
Re: Review Request 67800: HIVE-20039
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67800/#review205687
-----------------------------------------------------------
Ship it!
Ship It!
- Gopal V
On July 3, 2018, 8:20 p.m., Deepak Jaiswal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67800/
> -----------------------------------------------------------
>
> (Updated July 3, 2018, 8:20 p.m.)
>
>
> Review request for hive and Gopal V.
>
>
> Bugs: HIVE-20039
> https://issues.apache.org/jira/browse/HIVE-20039
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Bucket pruning: Left Outer Join on bucketed table gives wrong result.
> The context was reused by all the predicates. Instead use TS op directly.
>
>
> Diffs
> -----
>
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_datajoin_1_s2_2018022300104_1/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_dw_stg_2018022300104_1/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_clarity__l3_snap_number_2018022300104/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_monthly_dw_dimplan/000056_0 PRE-CREATION
> itests/src/test/resources/testconfiguration.properties d02c0fe8ba
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java 2debacacb5
> ql/src/test/queries/clientpositive/tez_fixed_bucket_pruning.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/tez_fixed_bucket_pruning.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/67800/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Deepak Jaiswal
>
>
Re: Review Request 67800: HIVE-20039
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67800/
-----------------------------------------------------------
(Updated July 3, 2018, 8:20 p.m.)
Review request for hive and Gopal V.
Changes
-------
Added test to llap only runs.
Bugs: HIVE-20039
https://issues.apache.org/jira/browse/HIVE-20039
Repository: hive-git
Description
-------
Bucket pruning: Left Outer Join on bucketed table gives wrong result.
The context was reused by all the predicates. Instead use TS op directly.
Diffs (updated)
-----
data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_datajoin_1_s2_2018022300104_1/000000_0 PRE-CREATION
data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_dw_stg_2018022300104_1/000000_0 PRE-CREATION
data/files/bucket_pruning/l3_clarity__l3_snap_number_2018022300104/000000_0 PRE-CREATION
data/files/bucket_pruning/l3_monthly_dw_dimplan/000056_0 PRE-CREATION
itests/src/test/resources/testconfiguration.properties d02c0fe8ba
ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java 2debacacb5
ql/src/test/queries/clientpositive/tez_fixed_bucket_pruning.q PRE-CREATION
ql/src/test/results/clientpositive/llap/tez_fixed_bucket_pruning.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/67800/diff/3/
Changes: https://reviews.apache.org/r/67800/diff/2-3/
Testing
-------
Thanks,
Deepak Jaiswal
Re: Review Request 67800: HIVE-20039
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67800/
-----------------------------------------------------------
(Updated July 3, 2018, 6:49 a.m.)
Review request for hive and Gopal V.
Changes
-------
Implemented recommended changes.
Added order by in queries for predictable results.
Bugs: HIVE-20039
https://issues.apache.org/jira/browse/HIVE-20039
Repository: hive-git
Description
-------
Bucket pruning: Left Outer Join on bucketed table gives wrong result.
The context was reused by all the predicates. Instead use TS op directly.
Diffs (updated)
-----
data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_datajoin_1_s2_2018022300104_1/000000_0 PRE-CREATION
data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_dw_stg_2018022300104_1/000000_0 PRE-CREATION
data/files/bucket_pruning/l3_clarity__l3_snap_number_2018022300104/000000_0 PRE-CREATION
data/files/bucket_pruning/l3_monthly_dw_dimplan/000056_0 PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java 2debacacb5
ql/src/test/queries/clientpositive/tez_fixed_bucket_pruning.q PRE-CREATION
ql/src/test/results/clientpositive/llap/tez_fixed_bucket_pruning.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/67800/diff/2/
Changes: https://reviews.apache.org/r/67800/diff/1-2/
Testing
-------
Thanks,
Deepak Jaiswal
Re: Review Request 67800: HIVE-20039
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67800/#review205649
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
Lines 90 (patched)
<https://reviews.apache.org/r/67800/#comment288536>
This needs the pruned partition list loop (to make sure all partitions have the same bucket-count).
This is probably something we can prevent from happening in Bucketing version = 2 so that tables with new bucketing version will never allow an
ALTER TABLE clustered by (id) buckets
from changing a partition's bucket count.
- Gopal V
On July 3, 2018, 12:23 a.m., Deepak Jaiswal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67800/
> -----------------------------------------------------------
>
> (Updated July 3, 2018, 12:23 a.m.)
>
>
> Review request for hive and Gopal V.
>
>
> Bugs: HIVE-20039
> https://issues.apache.org/jira/browse/HIVE-20039
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Bucket pruning: Left Outer Join on bucketed table gives wrong result.
> The context was reused by all the predicates. Instead use TS op directly.
>
>
> Diffs
> -----
>
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_datajoin_1_s2_2018022300104_1/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_dw_stg_2018022300104_1/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_clarity__l3_snap_number_2018022300104/000000_0 PRE-CREATION
> data/files/bucket_pruning/l3_monthly_dw_dimplan/000056_0 PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java 2debacacb5
> ql/src/test/queries/clientpositive/tez_fixed_bucket_pruning.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/tez_fixed_bucket_pruning.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/67800/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Deepak Jaiswal
>
>