You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Karen Coppage via Review Board <no...@reviews.apache.org> on 2020/01/30 15:30:16 UTC
Re: Review Request 72063: HIVE-10362: Support Type check/conversion in
dynamic partition column
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/
-----------------------------------------------------------
(Updated Jan. 30, 2020, 3:30 p.m.)
Review request for hive and Peter Vary.
Bugs: HIVE-10362
https://issues.apache.org/jira/browse/HIVE-10362
Repository: hive-git
Description
-------
Example:
create table dynparttypechecknum (key int, value string) partitioned by (part int);
insert into dynparttypechecknum partition (part) select key, value, '00001' from src limit 1;
show partitions dynparttypechecknum;
Partition created will be named:
part=00001
even though the type of `part` is int.
Solution is to cast the inserted DP columns in the SelectOperator before FileSinkOperator which creates the partition dir, not after.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 5fcc367cc9
ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java c1aeb8f136
ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 2c4b69b2fe
ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION
ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb
ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION
ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 43bb789840
ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out f745b46899
ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out fc9050b2c3
ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 95aae7286f
ql/src/test/results/clientpositive/llap/llap_smb.q.out 24026d0bab
ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac
ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f
ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234
ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a
ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e
ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582
ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340
ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f
ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18
ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f
ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out bc6c3add54
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 67b62c1265
ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 56d5ed945b
ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1
ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3
ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d
Diff: https://reviews.apache.org/r/72063/diff/1/
Testing
-------
There were changes in query output in two spark auto_sortmerge_join_16.q.out files. They now match the query output of llap/auto_sortmerge_join_16.q.out.
Thanks,
Karen Coppage
Re: Review Request 72063: HIVE-10362: Support Type check/conversion in
dynamic partition column
Posted by Vineet Garg <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219519
-----------------------------------------------------------
ql/src/test/queries/clientpositive/dynpart_cast.q
Lines 2 (patched)
<https://reviews.apache.org/r/72063/#comment307717>
Thanks for addressing the comments. Just another minor comment.
Can we add this test under 'minillap.query.files'? Right now this is MR test but since we use Tez/Llap it will be better under that.
You'll need to run "TestTezMiniLlapCliDriver' to genrate q file
- Vineet Garg
On Feb. 6, 2020, 2:22 p.m., Karen Coppage wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> -----------------------------------------------------------
>
> (Updated Feb. 6, 2020, 2:22 p.m.)
>
>
> Review request for hive and Peter Vary.
>
>
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part int);
> insert into dynparttypechecknum partition (part) select key, value, '00001' from src limit 1;
> show partitions dynparttypechecknum;
>
> Partition created will be named:
> part=00001
> even though the type of `part` is int.
>
> Solution is to cast the inserted DP columns in the SelectOperator before FileSinkOperator which creates the partition dir, not after.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 12a022c590
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2514eedb1
> ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION
> ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb
> ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION
> ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 43bb789840
> ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out f745b46899
> ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out 453d2451df
> ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac
> ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f
> ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234
> ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a
> ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e
> ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582
> ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340
> ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f
> ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18
> ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f
> ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 56d5ed945b
> ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1
> ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3
> ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d
>
>
> Diff: https://reviews.apache.org/r/72063/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Karen Coppage
>
>
Re: Review Request 72063: HIVE-10362: Support Type check/conversion in
dynamic partition column
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219540
-----------------------------------------------------------
I looked at the diff between 1:4
ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out
Line 98 (original)
<https://reviews.apache.org/r/72063/#comment307730>
it seems we are missing the stats update task here - is this expected
note: these compute_stats() things started appearing in the 1st patch uploaded to the reviewboard
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark
Line 454 (original), 454 (patched)
<https://reviews.apache.org/r/72063/#comment307731>
with the latest patch(#12)
we got back this null; and also the disagreement between llap/spark...
```
grep val_169 `find . -name auto_sortmerge_join_16.q.out*`|cut -d: -f1|uniq -c
24 ./ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out
17 ./ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark
17 ./ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out
```
- Zoltan Haindrich
On Feb. 7, 2020, 9:40 a.m., Karen Coppage wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> -----------------------------------------------------------
>
> (Updated Feb. 7, 2020, 9:40 a.m.)
>
>
> Review request for hive and Peter Vary.
>
>
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part int);
> insert into dynparttypechecknum partition (part) select key, value, '00001' from src limit 1;
> show partitions dynparttypechecknum;
>
> Partition created will be named:
> part=00001
> even though the type of `part` is int.
>
> Solution is to cast the inserted DP columns in the SelectOperator before FileSinkOperator which creates the partition dir, not after.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 12a022c590
> itests/src/test/resources/testconfiguration.properties 99ca9867b1
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2514eedb1
> ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION
> ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb
> ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION
> ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 43bb789840
> ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out f745b46899
> ql/src/test/results/clientpositive/llap/dynpart_cast.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out 453d2451df
> ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac
> ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f
> ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234
> ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a
> ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e
> ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582
> ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340
> ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f
> ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18
> ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f
> ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 56d5ed945b
> ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1
> ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3
> ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d
>
>
> Diff: https://reviews.apache.org/r/72063/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Karen Coppage
>
>
Re: Review Request 72063: HIVE-10362: Support Type check/conversion in
dynamic partition column
Posted by Karen Coppage via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/
-----------------------------------------------------------
(Updated Feb. 7, 2020, 8:40 a.m.)
Review request for hive and Peter Vary.
Changes
-------
Patch 12
Bugs: HIVE-10362
https://issues.apache.org/jira/browse/HIVE-10362
Repository: hive-git
Description
-------
Example:
create table dynparttypechecknum (key int, value string) partitioned by (part int);
insert into dynparttypechecknum partition (part) select key, value, '00001' from src limit 1;
show partitions dynparttypechecknum;
Partition created will be named:
part=00001
even though the type of `part` is int.
Solution is to cast the inserted DP columns in the SelectOperator before FileSinkOperator which creates the partition dir, not after.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 12a022c590
itests/src/test/resources/testconfiguration.properties 99ca9867b1
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2514eedb1
ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION
ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb
ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION
ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 43bb789840
ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out f745b46899
ql/src/test/results/clientpositive/llap/dynpart_cast.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out 453d2451df
ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac
ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f
ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234
ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a
ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e
ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582
ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340
ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f
ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18
ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f
ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a
ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 56d5ed945b
ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1
ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3
ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d
Diff: https://reviews.apache.org/r/72063/diff/4/
Changes: https://reviews.apache.org/r/72063/diff/3-4/
Testing
-------
Thanks,
Karen Coppage
Re: Review Request 72063: HIVE-10362: Support Type check/conversion in
dynamic partition column
Posted by Karen Coppage via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/
-----------------------------------------------------------
(Updated Feb. 6, 2020, 2:22 p.m.)
Review request for hive and Peter Vary.
Changes
-------
Patch 11 - qtest changes, dropped use of DynamicPartitionCtx
Bugs: HIVE-10362
https://issues.apache.org/jira/browse/HIVE-10362
Repository: hive-git
Description
-------
Example:
create table dynparttypechecknum (key int, value string) partitioned by (part int);
insert into dynparttypechecknum partition (part) select key, value, '00001' from src limit 1;
show partitions dynparttypechecknum;
Partition created will be named:
part=00001
even though the type of `part` is int.
Solution is to cast the inserted DP columns in the SelectOperator before FileSinkOperator which creates the partition dir, not after.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 12a022c590
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2514eedb1
ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION
ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb
ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION
ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 43bb789840
ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out f745b46899
ql/src/test/results/clientpositive/llap/dynpart_sort_opt_bucketing.q.out 453d2451df
ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac
ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f
ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234
ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a
ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e
ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582
ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340
ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f
ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18
ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f
ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a
ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 56d5ed945b
ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1
ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3
ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d
Diff: https://reviews.apache.org/r/72063/diff/3/
Changes: https://reviews.apache.org/r/72063/diff/2-3/
Testing
-------
There were changes in query output in two spark auto_sortmerge_join_16.q.out files. They now match the query output of llap/auto_sortmerge_join_16.q.out.
Thanks,
Karen Coppage
Re: Review Request 72063: HIVE-10362: Support Type check/conversion in
dynamic partition column
Posted by Karen Coppage via Review Board <no...@reviews.apache.org>.
> On Feb. 4, 2020, 2:03 p.m., Zoltan Haindrich wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 8475 (patched)
> > <https://reviews.apache.org/r/72063/diff/1/?file=2210048#file2210048line8476>
> >
> > do we really need this much if-s ?
> >
> >
> > what do you think about:
> > * throw real exceptions - if appropriate
> > * introduce feature toggle for this case/check and make it enabled by default - so that there will be a way back to the old behaviour
Good idea, will do.
> On Feb. 4, 2020, 2:03 p.m., Zoltan Haindrich wrote:
> > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark
> > Lines 458 (patched)
> > <https://reviews.apache.org/r/72063/diff/1/?file=2210071#file2210071line458>
> >
> > is this change expected?
> > is the new resultset the correct one?
> > do we have an agreement between spark/llap ?
With this change we have an agreement with llap. I also checked the results and llap was correct.
- Karen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219486
-----------------------------------------------------------
On Jan. 30, 2020, 3:30 p.m., Karen Coppage wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> -----------------------------------------------------------
>
> (Updated Jan. 30, 2020, 3:30 p.m.)
>
>
> Review request for hive and Peter Vary.
>
>
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part int);
> insert into dynparttypechecknum partition (part) select key, value, '00001' from src limit 1;
> show partitions dynparttypechecknum;
>
> Partition created will be named:
> part=00001
> even though the type of `part` is int.
>
> Solution is to cast the inserted DP columns in the SelectOperator before FileSinkOperator which creates the partition dir, not after.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 5fcc367cc9
> ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java c1aeb8f136
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 2c4b69b2fe
> ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION
> ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb
> ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION
> ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 43bb789840
> ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out f745b46899
> ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out fc9050b2c3
> ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 95aae7286f
> ql/src/test/results/clientpositive/llap/llap_smb.q.out 24026d0bab
> ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac
> ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f
> ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234
> ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a
> ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e
> ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582
> ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340
> ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f
> ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18
> ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f
> ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a
> ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out bc6c3add54
> ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 67b62c1265
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 56d5ed945b
> ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1
> ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3
> ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d
>
>
> Diff: https://reviews.apache.org/r/72063/diff/1/
>
>
> Testing
> -------
>
> There were changes in query output in two spark auto_sortmerge_join_16.q.out files. They now match the query output of llap/auto_sortmerge_join_16.q.out.
>
>
> Thanks,
>
> Karen Coppage
>
>
Re: Review Request 72063: HIVE-10362: Support Type check/conversion in
dynamic partition column
Posted by Zoltan Haindrich <ki...@rxd.hu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219486
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 8475 (patched)
<https://reviews.apache.org/r/72063/#comment307668>
do we really need this much if-s ?
what do you think about:
* throw real exceptions - if appropriate
* introduce feature toggle for this case/check and make it enabled by default - so that there will be a way back to the old behaviour
ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out
Line 231 (original), 231 (patched)
<https://reviews.apache.org/r/72063/#comment307670>
it's great that this will start fixing some case in which column stats was not there;
I guess it was not calculated correctly earlier
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark
Lines 458 (patched)
<https://reviews.apache.org/r/72063/#comment307669>
is this change expected?
is the new resultset the correct one?
do we have an agreement between spark/llap ?
- Zoltan Haindrich
On Jan. 30, 2020, 4:30 p.m., Karen Coppage wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> -----------------------------------------------------------
>
> (Updated Jan. 30, 2020, 4:30 p.m.)
>
>
> Review request for hive and Peter Vary.
>
>
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part int);
> insert into dynparttypechecknum partition (part) select key, value, '00001' from src limit 1;
> show partitions dynparttypechecknum;
>
> Partition created will be named:
> part=00001
> even though the type of `part` is int.
>
> Solution is to cast the inserted DP columns in the SelectOperator before FileSinkOperator which creates the partition dir, not after.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 5fcc367cc9
> ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java c1aeb8f136
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 2c4b69b2fe
> ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION
> ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb
> ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION
> ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 43bb789840
> ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out f745b46899
> ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out fc9050b2c3
> ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 95aae7286f
> ql/src/test/results/clientpositive/llap/llap_smb.q.out 24026d0bab
> ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac
> ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f
> ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234
> ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a
> ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e
> ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582
> ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340
> ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f
> ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18
> ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f
> ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a
> ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out bc6c3add54
> ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 67b62c1265
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 56d5ed945b
> ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1
> ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3
> ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d
>
>
> Diff: https://reviews.apache.org/r/72063/diff/1/
>
>
> Testing
> -------
>
> There were changes in query output in two spark auto_sortmerge_join_16.q.out files. They now match the query output of llap/auto_sortmerge_join_16.q.out.
>
>
> Thanks,
>
> Karen Coppage
>
>
Re: Review Request 72063: HIVE-10362: Support Type check/conversion in
dynamic partition column
Posted by Vineet Garg <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72063/#review219513
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java
Lines 88 (patched)
<https://reviews.apache.org/r/72063/#comment307708>
This will only be used if DYNAMICPARTITIONCONVERT is set to true but right now the map is always populated. It will be good to explicitly populate the map (may be by a call from genConversionSelectOperator before adding the cast)
ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java
Lines 222 (patched)
<https://reviews.apache.org/r/72063/#comment307709>
A comment would be helpful here to explain why we need to keep it
ql/src/test/queries/clientpositive/dynpart_cast.q
Lines 7 (patched)
<https://reviews.apache.org/r/72063/#comment307710>
Can we add explain plan?
ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out
Lines 36 (patched)
<https://reviews.apache.org/r/72063/#comment307711>
This test is to test Sort dynamic bucket partitioning, with the change this optimization is not kicking in anymore (PARTITION_BUCKET_SORTED keyword is missing in File sink). We should understand why is it happening and may be either fix it or log a jira.
- Vineet Garg
On Jan. 30, 2020, 3:30 p.m., Karen Coppage wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72063/
> -----------------------------------------------------------
>
> (Updated Jan. 30, 2020, 3:30 p.m.)
>
>
> Review request for hive and Peter Vary.
>
>
> Bugs: HIVE-10362
> https://issues.apache.org/jira/browse/HIVE-10362
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Example:
> create table dynparttypechecknum (key int, value string) partitioned by (part int);
> insert into dynparttypechecknum partition (part) select key, value, '00001' from src limit 1;
> show partitions dynparttypechecknum;
>
> Partition created will be named:
> part=00001
> even though the type of `part` is int.
>
> Solution is to cast the inserted DP columns in the SelectOperator before FileSinkOperator which creates the partition dir, not after.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 12a022c590
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c2514eedb1
> ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java c1aeb8f136
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 2c4b69b2fe
> ql/src/test/queries/clientpositive/dynpart_cast.q PRE-CREATION
> ql/src/test/results/clientpositive/autoColumnStats_6.q.out da3be3e5bb
> ql/src/test/results/clientpositive/dynpart_cast.q.out PRE-CREATION
> ql/src/test/results/clientpositive/dynpart_sort_optimization_acid2.q.out 43bb789840
> ql/src/test/results/clientpositive/infer_bucket_sort_num_buckets.q.out f745b46899
> ql/src/test/results/clientpositive/llap/auto_sortmerge_join_16.q.out fc9050b2c3
> ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 95aae7286f
> ql/src/test/results/clientpositive/llap/llap_smb.q.out 24026d0bab
> ql/src/test/results/clientpositive/llap/orc_merge1.q.out 9da73e65ac
> ql/src/test/results/clientpositive/llap/orc_merge10.q.out a6ea33493f
> ql/src/test/results/clientpositive/llap/orc_merge2.q.out 9b0d3b4234
> ql/src/test/results/clientpositive/llap/orc_merge_diff_fs.q.out d35f44b10a
> ql/src/test/results/clientpositive/llap/rcfile_merge2.q.out fcff20a68e
> ql/src/test/results/clientpositive/llap/tez_dml.q.out 4ad78d8582
> ql/src/test/results/clientpositive/orc_merge1.q.out 9c07816340
> ql/src/test/results/clientpositive/orc_merge10.q.out 4a5f03c82f
> ql/src/test/results/clientpositive/orc_merge2.q.out d132d62b18
> ql/src/test/results/clientpositive/orc_merge_diff_fs.q.out 7f9a04b09f
> ql/src/test/results/clientpositive/smb_join_partition_key.q.out c18d01d26a
> ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out bc6c3add54
> ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out_spark 67b62c1265
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out 56d5ed945b
> ql/src/test/results/clientpositive/spark/orc_merge1.q.out 977c4cbfc1
> ql/src/test/results/clientpositive/spark/orc_merge2.q.out 4647b86ea3
> ql/src/test/results/clientpositive/spark/orc_merge_diff_fs.q.out b7d3dd725d
>
>
> Diff: https://reviews.apache.org/r/72063/diff/2/
>
>
> Testing
> -------
>
> There were changes in query output in two spark auto_sortmerge_join_16.q.out files. They now match the query output of llap/auto_sortmerge_join_16.q.out.
>
>
> Thanks,
>
> Karen Coppage
>
>