You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Jesús Camacho Rodríguez <jc...@hortonworks.com> on 2018/07/27 18:12:12 UTC
Review Request 68086: HIVE-20241
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68086/
-----------------------------------------------------------
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-20241
https://issues.apache.org/jira/browse/HIVE-20241
Repository: hive-git
Description
-------
HIVE-20241
Diffs
-----
itests/src/test/resources/testconfiguration.properties 50bfe6a06a77814decb6d48a19c56e8a41e34432
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 397cee2a5f5db8874b402377f36d9a33551327c3
ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java a1f5133c8626cb5e2fb64ea29236bea48aacacea
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fa923853785b078dd30f000bd803e1bf4b86bcde
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 6be48ca23beeadd03d822f211130176d41ca5164
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b5adf1bd046bece1affce6bb0f962cdf5c4af03b
ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 7a2a2c7a2841640adc6414087efa8ef5e9ab360d
ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 871844b30d17ac351da90c8a7ba03bcc6341b085
ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 6af783302220de51c9c3933039a4122b28400d12
ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java af2ece44dc9853607a8737f31284f9442f6c1d69
ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 71127c20b7e7d99394c5251ecf7a04a89e0abfc4
ql/src/test/queries/clientnegative/partition_ctas_wrong_order.q PRE-CREATION
ql/src/test/queries/clientpositive/partition_ctas.q PRE-CREATION
ql/src/test/results/clientnegative/partition_ctas_wrong_order.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/partition_ctas.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/68086/diff/1/
Testing
-------
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 68086: HIVE-20241
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
> On July 30, 2018, 10:06 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
> > Lines 2002 (patched)
> > <https://reviews.apache.org/r/68086/diff/1/?file=2064678#file2064678line2002>
> >
> > I don't think we want to allow type in partitioned by clause. Only _createTablePartitionColumnSpec_ should be allowed.
The problem is that this rule matches normal CREATE TABLE statement as well as CTAS. In SemanticAnalyzer, I check whether the right type of partitionSpec has been specified and fail otherwise.
> On July 30, 2018, 10:06 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
> > Lines 7620 (patched)
> > <https://reviews.apache.org/r/68086/diff/1/?file=2064679#file2064679line7630>
> >
> > Will this guanatee atomicity for Acid and MM tables? Specifically case where table gets created but some or all of partitions didnt get added in metastore. Add TODO comment if this will be handled later.
The commitTxn is triggered by the Driver when tasks have been completed, thus I think current model guarantees atomiticy for data for any kind of transactional table.
However, if tasks fail, we may end up with an empty table and dangling partitions in the metastore. I believe this problem probably exists nowadays even for INSERT statements?
> On July 30, 2018, 10:06 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientnegative/partition_ctas_wrong_order.q
> > Lines 3 (patched)
> > <https://reviews.apache.org/r/68086/diff/1/?file=2064685#file2064685line3>
> >
> > Is there a reason to disallow partition column other than last one, since columns are referred explictly by name, I imagine user would expect this to work. Especially since we have recently allowed this to work for insert statements as well.
Yes, it would be good to support this, I will work on it. Difficulty was that for CTAS we infer the table schema from the query schema, so I will need to change that logic.
- Jesús
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68086/#review206615
-----------------------------------------------------------
On July 27, 2018, 6:12 p.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68086/
> -----------------------------------------------------------
>
> (Updated July 27, 2018, 6:12 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-20241
> https://issues.apache.org/jira/browse/HIVE-20241
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-20241
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties 50bfe6a06a77814decb6d48a19c56e8a41e34432
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 397cee2a5f5db8874b402377f36d9a33551327c3
> ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java a1f5133c8626cb5e2fb64ea29236bea48aacacea
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fa923853785b078dd30f000bd803e1bf4b86bcde
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 6be48ca23beeadd03d822f211130176d41ca5164
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b5adf1bd046bece1affce6bb0f962cdf5c4af03b
> ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 7a2a2c7a2841640adc6414087efa8ef5e9ab360d
> ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 871844b30d17ac351da90c8a7ba03bcc6341b085
> ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 6af783302220de51c9c3933039a4122b28400d12
> ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java af2ece44dc9853607a8737f31284f9442f6c1d69
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 71127c20b7e7d99394c5251ecf7a04a89e0abfc4
> ql/src/test/queries/clientnegative/partition_ctas_wrong_order.q PRE-CREATION
> ql/src/test/queries/clientpositive/partition_ctas.q PRE-CREATION
> ql/src/test/results/clientnegative/partition_ctas_wrong_order.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/partition_ctas.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/68086/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 68086: HIVE-20241
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68086/#review206615
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
Lines 2002 (patched)
<https://reviews.apache.org/r/68086/#comment289644>
I don't think we want to allow type in partitioned by clause. Only _createTablePartitionColumnSpec_ should be allowed.
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Lines 7620 (patched)
<https://reviews.apache.org/r/68086/#comment289645>
Will this guanatee atomicity for Acid and MM tables? Specifically case where table gets created but some or all of partitions didnt get added in metastore. Add TODO comment if this will be handled later.
ql/src/test/queries/clientnegative/partition_ctas_wrong_order.q
Lines 3 (patched)
<https://reviews.apache.org/r/68086/#comment289650>
Is there a reason to disallow partition column other than last one, since columns are referred explictly by name, I imagine user would expect this to work. Especially since we have recently allowed this to work for insert statements as well.
- Ashutosh Chauhan
On July 27, 2018, 6:12 p.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68086/
> -----------------------------------------------------------
>
> (Updated July 27, 2018, 6:12 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-20241
> https://issues.apache.org/jira/browse/HIVE-20241
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-20241
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties 50bfe6a06a77814decb6d48a19c56e8a41e34432
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 397cee2a5f5db8874b402377f36d9a33551327c3
> ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java a1f5133c8626cb5e2fb64ea29236bea48aacacea
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fa923853785b078dd30f000bd803e1bf4b86bcde
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 6be48ca23beeadd03d822f211130176d41ca5164
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b5adf1bd046bece1affce6bb0f962cdf5c4af03b
> ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 7a2a2c7a2841640adc6414087efa8ef5e9ab360d
> ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 871844b30d17ac351da90c8a7ba03bcc6341b085
> ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 6af783302220de51c9c3933039a4122b28400d12
> ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java af2ece44dc9853607a8737f31284f9442f6c1d69
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 71127c20b7e7d99394c5251ecf7a04a89e0abfc4
> ql/src/test/queries/clientnegative/partition_ctas_wrong_order.q PRE-CREATION
> ql/src/test/queries/clientpositive/partition_ctas.q PRE-CREATION
> ql/src/test/results/clientnegative/partition_ctas_wrong_order.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/partition_ctas.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/68086/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 68086: HIVE-20241
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68086/
-----------------------------------------------------------
(Updated July 31, 2018, 2:17 a.m.)
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-20241
https://issues.apache.org/jira/browse/HIVE-20241
Repository: hive-git
Description
-------
HIVE-20241
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties 125ad19278b30a8b3a018e20805524c76f864e9a
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 397cee2a5f5db8874b402377f36d9a33551327c3
ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java a1f5133c8626cb5e2fb64ea29236bea48aacacea
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fa923853785b078dd30f000bd803e1bf4b86bcde
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 6be48ca23beeadd03d822f211130176d41ca5164
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b5adf1bd046bece1affce6bb0f962cdf5c4af03b
ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 7a2a2c7a2841640adc6414087efa8ef5e9ab360d
ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 871844b30d17ac351da90c8a7ba03bcc6341b085
ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 6af783302220de51c9c3933039a4122b28400d12
ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java af2ece44dc9853607a8737f31284f9442f6c1d69
ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 71127c20b7e7d99394c5251ecf7a04a89e0abfc4
ql/src/test/queries/clientpositive/partition_ctas.q PRE-CREATION
ql/src/test/results/clientpositive/llap/partition_ctas.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/68086/diff/3/
Changes: https://reviews.apache.org/r/68086/diff/2-3/
Testing
-------
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 68086: HIVE-20241
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
> On July 31, 2018, 1:27 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
> > Lines 219 (patched)
> > <https://reviews.apache.org/r/68086/diff/1/?file=2064678#file2064678line219>
> >
> > This token is unused. Can be deleted.
Thanks!
> On July 31, 2018, 1:27 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
> > Line 1990 (original), 1992 (patched)
> > <https://reviews.apache.org/r/68086/diff/1/?file=2064678#file2064678line1992>
> >
> > Looks this rule is unused and can be deleted.
Thanks!
- Jesús
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68086/#review206632
-----------------------------------------------------------
On July 31, 2018, 12:57 a.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68086/
> -----------------------------------------------------------
>
> (Updated July 31, 2018, 12:57 a.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-20241
> https://issues.apache.org/jira/browse/HIVE-20241
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-20241
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties 125ad19278b30a8b3a018e20805524c76f864e9a
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 397cee2a5f5db8874b402377f36d9a33551327c3
> ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java a1f5133c8626cb5e2fb64ea29236bea48aacacea
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fa923853785b078dd30f000bd803e1bf4b86bcde
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 6be48ca23beeadd03d822f211130176d41ca5164
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b5adf1bd046bece1affce6bb0f962cdf5c4af03b
> ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 7a2a2c7a2841640adc6414087efa8ef5e9ab360d
> ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 871844b30d17ac351da90c8a7ba03bcc6341b085
> ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 6af783302220de51c9c3933039a4122b28400d12
> ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java af2ece44dc9853607a8737f31284f9442f6c1d69
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 71127c20b7e7d99394c5251ecf7a04a89e0abfc4
> ql/src/test/queries/clientpositive/partition_ctas.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/partition_ctas.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/68086/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 68086: HIVE-20241
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68086/#review206632
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
Lines 219 (patched)
<https://reviews.apache.org/r/68086/#comment289689>
This token is unused. Can be deleted.
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
Line 1990 (original), 1992 (patched)
<https://reviews.apache.org/r/68086/#comment289688>
Looks this rule is unused and can be deleted.
- Ashutosh Chauhan
On July 31, 2018, 12:57 a.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68086/
> -----------------------------------------------------------
>
> (Updated July 31, 2018, 12:57 a.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-20241
> https://issues.apache.org/jira/browse/HIVE-20241
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-20241
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties 125ad19278b30a8b3a018e20805524c76f864e9a
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 397cee2a5f5db8874b402377f36d9a33551327c3
> ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java a1f5133c8626cb5e2fb64ea29236bea48aacacea
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fa923853785b078dd30f000bd803e1bf4b86bcde
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 6be48ca23beeadd03d822f211130176d41ca5164
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b5adf1bd046bece1affce6bb0f962cdf5c4af03b
> ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 7a2a2c7a2841640adc6414087efa8ef5e9ab360d
> ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 871844b30d17ac351da90c8a7ba03bcc6341b085
> ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 6af783302220de51c9c3933039a4122b28400d12
> ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java af2ece44dc9853607a8737f31284f9442f6c1d69
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 71127c20b7e7d99394c5251ecf7a04a89e0abfc4
> ql/src/test/queries/clientpositive/partition_ctas.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/partition_ctas.q.out PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/68086/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 68086: HIVE-20241
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68086/
-----------------------------------------------------------
(Updated July 31, 2018, 12:57 a.m.)
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-20241
https://issues.apache.org/jira/browse/HIVE-20241
Repository: hive-git
Description
-------
HIVE-20241
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties 125ad19278b30a8b3a018e20805524c76f864e9a
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 397cee2a5f5db8874b402377f36d9a33551327c3
ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java a1f5133c8626cb5e2fb64ea29236bea48aacacea
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fa923853785b078dd30f000bd803e1bf4b86bcde
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 6be48ca23beeadd03d822f211130176d41ca5164
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b5adf1bd046bece1affce6bb0f962cdf5c4af03b
ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 7a2a2c7a2841640adc6414087efa8ef5e9ab360d
ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 871844b30d17ac351da90c8a7ba03bcc6341b085
ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 6af783302220de51c9c3933039a4122b28400d12
ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java af2ece44dc9853607a8737f31284f9442f6c1d69
ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 71127c20b7e7d99394c5251ecf7a04a89e0abfc4
ql/src/test/queries/clientpositive/partition_ctas.q PRE-CREATION
ql/src/test/results/clientpositive/llap/partition_ctas.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/68086/diff/2/
Changes: https://reviews.apache.org/r/68086/diff/1-2/
Testing
-------
Thanks,
Jesús Camacho Rodríguez