You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Quanlong Huang <hu...@gmail.com> on 2018/08/24 02:44:15 UTC
Breaking change for query hints?
Hi all,
After we upgrade Impala from 2.5 to 2.12, we found some queries on views
failed. The views contain a query hint which I think is the cause. I can't
find anything about such kind of incompatibility was mentioned in
https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
or
https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
The error can be reproduced by
[impala-shell:21000] > use test;
[impala-shell:21000] > create view view_2_12 as select /*
+`queryId=abc` */ * from video_parq;
Query: create view view_2_12 as select /* +`queryId=abc` */ * from video_parq
+------------------------+
| summary |
+------------------------+
| View has been created. |
+------------------------+
WARNINGS: PLAN hint not recognized: queryId=abc
Fetched 1 row(s) in 24.00s
[impala-shell:21000] > desc formatted view_2_12;
Query: describe formatted view_2_12
ERROR: AnalysisException: Failed to parse view-definition statement of
view: test.view_2_12
CAUSED BY: TableLoadingException: Failed to parse view-definition
statement of view: test.view_2_12
Since this's an emergency for us, I post an email here for help. I'm also
looking into the history of the cup file and looking for a hotfix for this.
Thanks if anyone can give some pointers!
Thanks,
Quanlong
Re: Breaking change for query hints?
Posted by Lars Volker <lv...@cloudera.com>.
Great, glad you found a fix. Looking through the code I noticed that we
will interpret any unknown select hint as 'straigh_join'. IMPALA-7484
<https://issues.apache.org/jira/browse/IMPALA-7484> has the details, you
might want to keep an eye on this when using this annotation technique with
joins.
Cheers, Lars
On Thu, Aug 23, 2018 at 9:46 PM Quanlong Huang <hu...@gmail.com>
wrote:
> Thank you, Todd and Lars! We plan to hotfix this by quotes the vars in
> PlanHint:
>
> *diff --git a/fe/src/main/java/org/apache/impala/analysis/PlanHint.java
> b/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
>
> *index d16919f09..4da0e91c2 100644*
>
> *--- a/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
>
> *+++ b/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
>
> @@ -61,8 +61,10 @@ public class PlanHint {
>
> @Override
>
> public String toString() {
>
> StringBuilder sb = new StringBuilder();
>
> - sb.append(name_);
>
> - if (!args_.isEmpty()) {
>
> + if (args_.isEmpty()) {
>
> + sb.append('`').append(name_).append('`');
>
> + } else {
>
> + sb.append(name_);
>
> sb.append("(");
>
> sb.append(Joiner.on(",").join(args_));
>
> sb.append(")");
>
>
>
> Quanlong Huang <hu...@gmail.com> 于2018年8月24日周五 下午12:09写道：
>
>> Unfortunately, the queries come from a system that we don't have control.
>> I think we should change the behavior of the CreateView statement to save
>> everything in the comments.
>>
>> Lars Volker <lv...@cloudera.com> 于2018年8月24日周五 上午11:37写道：
>>
>>> We changed query hints from being interpreted as strings to being parsed
>>> properly in commit ce9b332e
>>> <https://github.com/apache/impala/commit/ce9b332ee9e640d79c8ae35e7abb8c7d787ddf78#diff-27ff375b57f7fc982be98c4f503cb04dR2385>
>>> for IMPALA-4163 <https://issues.apache.org/jira/browse/IMPALA-4163>.
>>> Before that change we called String.trim() on the hint which I suspect
>>> preserved the ``. After this change we parse it as an IDENT, which likely
>>> strips the ``.
>>>
>>> I feel that hints are not the right solution to embed a query id into
>>> the view definition, but I cannot think of a better one, either. With the
>>> current syntax you can write the hint as queryId_abc or queryId(abc) and
>>> while you'll get a bunch of warning it seems to work on my end.
>>>
>>> I also think there's a (unrelated) bug in SelectList.java:87
>>> <https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87>:
>>> It should have a continue statement inside the check, or wrap the call to
>>> setIsStraightJoin() in an else clause.
>>>
>>> On Thu, Aug 23, 2018 at 8:19 PM Quanlong Huang <hu...@gmail.com>
>>> wrote:
>>>
>>>> Yeah, the results of CreateView statement is different. We created the
>>>> same view by impala-2.5. The difference is that impala-2.12 delete the
>>>> quotes in the comment.
>>>>
>>>> Found the difference in beeline:
>>>>
>>>> > show create table view_2_12;
>>>>
>>>> createtab_stmt
>>>>
>>>> CREATE VIEW `view_2_12` AS SELECT
>>>>
>>>> -- +queryId=abc
>>>>
>>>> * FROM test.video_parq
>>>>
>>>> 3 rows selected (4.518 seconds)
>>>>
>>>>
>>>> > show create table view_2_5;
>>>>
>>>> createtab_stmt
>>>>
>>>> CREATE VIEW `view_2_5` AS SELECT
>>>>
>>>> -- +`queryId=abc`
>>>>
>>>> * FROM test.video_parq
>>>>
>>>> 3 rows selected (1.202 seconds)
>>>>
>>>> view_2_12 is created by impala-2.12, while view_2_5 is created by
>>>> impala-2.5.
>>>>
>>>> Todd Lipcon <to...@cloudera.com> 于2018年8月24日周五 上午10:53写道：
>>>>
>>>>> I'm curious: can you describe the view using Hive to see what the
>>>>> stored query consists of?
>>>>>
>>>>> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> After we upgrade Impala from 2.5 to 2.12, we found some queries on
>>>>>> views
>>>>>> failed. The views contain a query hint which I think is the cause. I
>>>>>> can't
>>>>>> find anything about such kind of incompatibility was mentioned in
>>>>>>
>>>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>>>>>> or
>>>>>>
>>>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>>>>>
>>>>>> The error can be reproduced by
>>>>>>
>>>>>> [impala-shell:21000] > use test;
>>>>>> [impala-shell:21000] > create view view_2_12 as select /*
>>>>>> +`queryId=abc` */ * from video_parq;
>>>>>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>>>>>> video_parq
>>>>>> +------------------------+
>>>>>> | summary |
>>>>>> +------------------------+
>>>>>> | View has been created. |
>>>>>> +------------------------+
>>>>>> WARNINGS: PLAN hint not recognized: queryId=abc
>>>>>>
>>>>>> Fetched 1 row(s) in 24.00s
>>>>>> [impala-shell:21000] > desc formatted view_2_12;
>>>>>> Query: describe formatted view_2_12
>>>>>> ERROR: AnalysisException: Failed to parse view-definition statement of
>>>>>> view: test.view_2_12
>>>>>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>>>>>> statement of view: test.view_2_12
>>>>>>
>>>>>>
>>>>>> Since this's an emergency for us, I post an email here for help. I'm
>>>>>> also
>>>>>> looking into the history of the cup file and looking for a hotfix for
>>>>>> this.
>>>>>> Thanks if anyone can give some pointers!
>>>>>>
>>>>>> Thanks,
>>>>>> Quanlong
>>>>>>
>>>>>
Re: Breaking change for query hints?
Posted by Lars Volker <lv...@cloudera.com>.
Great, glad you found a fix. Looking through the code I noticed that we
will interpret any unknown select hint as 'straigh_join'. IMPALA-7484
<https://issues.apache.org/jira/browse/IMPALA-7484> has the details, you
might want to keep an eye on this when using this annotation technique with
joins.
Cheers, Lars
On Thu, Aug 23, 2018 at 9:46 PM Quanlong Huang <hu...@gmail.com>
wrote:
> Thank you, Todd and Lars! We plan to hotfix this by quotes the vars in
> PlanHint:
>
> *diff --git a/fe/src/main/java/org/apache/impala/analysis/PlanHint.java
> b/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
>
> *index d16919f09..4da0e91c2 100644*
>
> *--- a/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
>
> *+++ b/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
>
> @@ -61,8 +61,10 @@ public class PlanHint {
>
> @Override
>
> public String toString() {
>
> StringBuilder sb = new StringBuilder();
>
> - sb.append(name_);
>
> - if (!args_.isEmpty()) {
>
> + if (args_.isEmpty()) {
>
> + sb.append('`').append(name_).append('`');
>
> + } else {
>
> + sb.append(name_);
>
> sb.append("(");
>
> sb.append(Joiner.on(",").join(args_));
>
> sb.append(")");
>
>
>
> Quanlong Huang <hu...@gmail.com> 于2018年8月24日周五 下午12:09写道：
>
>> Unfortunately, the queries come from a system that we don't have control.
>> I think we should change the behavior of the CreateView statement to save
>> everything in the comments.
>>
>> Lars Volker <lv...@cloudera.com> 于2018年8月24日周五 上午11:37写道：
>>
>>> We changed query hints from being interpreted as strings to being parsed
>>> properly in commit ce9b332e
>>> <https://github.com/apache/impala/commit/ce9b332ee9e640d79c8ae35e7abb8c7d787ddf78#diff-27ff375b57f7fc982be98c4f503cb04dR2385>
>>> for IMPALA-4163 <https://issues.apache.org/jira/browse/IMPALA-4163>.
>>> Before that change we called String.trim() on the hint which I suspect
>>> preserved the ``. After this change we parse it as an IDENT, which likely
>>> strips the ``.
>>>
>>> I feel that hints are not the right solution to embed a query id into
>>> the view definition, but I cannot think of a better one, either. With the
>>> current syntax you can write the hint as queryId_abc or queryId(abc) and
>>> while you'll get a bunch of warning it seems to work on my end.
>>>
>>> I also think there's a (unrelated) bug in SelectList.java:87
>>> <https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87>:
>>> It should have a continue statement inside the check, or wrap the call to
>>> setIsStraightJoin() in an else clause.
>>>
>>> On Thu, Aug 23, 2018 at 8:19 PM Quanlong Huang <hu...@gmail.com>
>>> wrote:
>>>
>>>> Yeah, the results of CreateView statement is different. We created the
>>>> same view by impala-2.5. The difference is that impala-2.12 delete the
>>>> quotes in the comment.
>>>>
>>>> Found the difference in beeline:
>>>>
>>>> > show create table view_2_12;
>>>>
>>>> createtab_stmt
>>>>
>>>> CREATE VIEW `view_2_12` AS SELECT
>>>>
>>>> -- +queryId=abc
>>>>
>>>> * FROM test.video_parq
>>>>
>>>> 3 rows selected (4.518 seconds)
>>>>
>>>>
>>>> > show create table view_2_5;
>>>>
>>>> createtab_stmt
>>>>
>>>> CREATE VIEW `view_2_5` AS SELECT
>>>>
>>>> -- +`queryId=abc`
>>>>
>>>> * FROM test.video_parq
>>>>
>>>> 3 rows selected (1.202 seconds)
>>>>
>>>> view_2_12 is created by impala-2.12, while view_2_5 is created by
>>>> impala-2.5.
>>>>
>>>> Todd Lipcon <to...@cloudera.com> 于2018年8月24日周五 上午10:53写道：
>>>>
>>>>> I'm curious: can you describe the view using Hive to see what the
>>>>> stored query consists of?
>>>>>
>>>>> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> After we upgrade Impala from 2.5 to 2.12, we found some queries on
>>>>>> views
>>>>>> failed. The views contain a query hint which I think is the cause. I
>>>>>> can't
>>>>>> find anything about such kind of incompatibility was mentioned in
>>>>>>
>>>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>>>>>> or
>>>>>>
>>>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>>>>>
>>>>>> The error can be reproduced by
>>>>>>
>>>>>> [impala-shell:21000] > use test;
>>>>>> [impala-shell:21000] > create view view_2_12 as select /*
>>>>>> +`queryId=abc` */ * from video_parq;
>>>>>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>>>>>> video_parq
>>>>>> +------------------------+
>>>>>> | summary |
>>>>>> +------------------------+
>>>>>> | View has been created. |
>>>>>> +------------------------+
>>>>>> WARNINGS: PLAN hint not recognized: queryId=abc
>>>>>>
>>>>>> Fetched 1 row(s) in 24.00s
>>>>>> [impala-shell:21000] > desc formatted view_2_12;
>>>>>> Query: describe formatted view_2_12
>>>>>> ERROR: AnalysisException: Failed to parse view-definition statement of
>>>>>> view: test.view_2_12
>>>>>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>>>>>> statement of view: test.view_2_12
>>>>>>
>>>>>>
>>>>>> Since this's an emergency for us, I post an email here for help. I'm
>>>>>> also
>>>>>> looking into the history of the cup file and looking for a hotfix for
>>>>>> this.
>>>>>> Thanks if anyone can give some pointers!
>>>>>>
>>>>>> Thanks,
>>>>>> Quanlong
>>>>>>
>>>>>
Re: Breaking change for query hints?
Posted by Quanlong Huang <hu...@gmail.com>.
Thank you, Todd and Lars! We plan to hotfix this by quotes the vars in
PlanHint:
*diff --git a/fe/src/main/java/org/apache/impala/analysis/PlanHint.java
b/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
*index d16919f09..4da0e91c2 100644*
*--- a/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
*+++ b/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
@@ -61,8 +61,10 @@ public class PlanHint {
@Override
public String toString() {
StringBuilder sb = new StringBuilder();
- sb.append(name_);
- if (!args_.isEmpty()) {
+ if (args_.isEmpty()) {
+ sb.append('`').append(name_).append('`');
+ } else {
+ sb.append(name_);
sb.append("(");
sb.append(Joiner.on(",").join(args_));
sb.append(")");
Quanlong Huang <hu...@gmail.com> 于2018年8月24日周五 下午12:09写道：
> Unfortunately, the queries come from a system that we don't have control.
> I think we should change the behavior of the CreateView statement to save
> everything in the comments.
>
> Lars Volker <lv...@cloudera.com> 于2018年8月24日周五 上午11:37写道：
>
>> We changed query hints from being interpreted as strings to being parsed
>> properly in commit ce9b332e
>> <https://github.com/apache/impala/commit/ce9b332ee9e640d79c8ae35e7abb8c7d787ddf78#diff-27ff375b57f7fc982be98c4f503cb04dR2385>
>> for IMPALA-4163 <https://issues.apache.org/jira/browse/IMPALA-4163>.
>> Before that change we called String.trim() on the hint which I suspect
>> preserved the ``. After this change we parse it as an IDENT, which likely
>> strips the ``.
>>
>> I feel that hints are not the right solution to embed a query id into the
>> view definition, but I cannot think of a better one, either. With the
>> current syntax you can write the hint as queryId_abc or queryId(abc) and
>> while you'll get a bunch of warning it seems to work on my end.
>>
>> I also think there's a (unrelated) bug in SelectList.java:87
>> <https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87>:
>> It should have a continue statement inside the check, or wrap the call to
>> setIsStraightJoin() in an else clause.
>>
>> On Thu, Aug 23, 2018 at 8:19 PM Quanlong Huang <hu...@gmail.com>
>> wrote:
>>
>>> Yeah, the results of CreateView statement is different. We created the
>>> same view by impala-2.5. The difference is that impala-2.12 delete the
>>> quotes in the comment.
>>>
>>> Found the difference in beeline:
>>>
>>> > show create table view_2_12;
>>>
>>> createtab_stmt
>>>
>>> CREATE VIEW `view_2_12` AS SELECT
>>>
>>> -- +queryId=abc
>>>
>>> * FROM test.video_parq
>>>
>>> 3 rows selected (4.518 seconds)
>>>
>>>
>>> > show create table view_2_5;
>>>
>>> createtab_stmt
>>>
>>> CREATE VIEW `view_2_5` AS SELECT
>>>
>>> -- +`queryId=abc`
>>>
>>> * FROM test.video_parq
>>>
>>> 3 rows selected (1.202 seconds)
>>>
>>> view_2_12 is created by impala-2.12, while view_2_5 is created by
>>> impala-2.5.
>>>
>>> Todd Lipcon <to...@cloudera.com> 于2018年8月24日周五 上午10:53写道：
>>>
>>>> I'm curious: can you describe the view using Hive to see what the
>>>> stored query consists of?
>>>>
>>>> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> After we upgrade Impala from 2.5 to 2.12, we found some queries on
>>>>> views
>>>>> failed. The views contain a query hint which I think is the cause. I
>>>>> can't
>>>>> find anything about such kind of incompatibility was mentioned in
>>>>>
>>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>>>>> or
>>>>>
>>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>>>>
>>>>> The error can be reproduced by
>>>>>
>>>>> [impala-shell:21000] > use test;
>>>>> [impala-shell:21000] > create view view_2_12 as select /*
>>>>> +`queryId=abc` */ * from video_parq;
>>>>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>>>>> video_parq
>>>>> +------------------------+
>>>>> | summary |
>>>>> +------------------------+
>>>>> | View has been created. |
>>>>> +------------------------+
>>>>> WARNINGS: PLAN hint not recognized: queryId=abc
>>>>>
>>>>> Fetched 1 row(s) in 24.00s
>>>>> [impala-shell:21000] > desc formatted view_2_12;
>>>>> Query: describe formatted view_2_12
>>>>> ERROR: AnalysisException: Failed to parse view-definition statement of
>>>>> view: test.view_2_12
>>>>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>>>>> statement of view: test.view_2_12
>>>>>
>>>>>
>>>>> Since this's an emergency for us, I post an email here for help. I'm
>>>>> also
>>>>> looking into the history of the cup file and looking for a hotfix for
>>>>> this.
>>>>> Thanks if anyone can give some pointers!
>>>>>
>>>>> Thanks,
>>>>> Quanlong
>>>>>
>>>>
Re: Breaking change for query hints?
Posted by Quanlong Huang <hu...@gmail.com>.
Thank you, Todd and Lars! We plan to hotfix this by quotes the vars in
PlanHint:
*diff --git a/fe/src/main/java/org/apache/impala/analysis/PlanHint.java
b/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
*index d16919f09..4da0e91c2 100644*
*--- a/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
*+++ b/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
@@ -61,8 +61,10 @@ public class PlanHint {
@Override
public String toString() {
StringBuilder sb = new StringBuilder();
- sb.append(name_);
- if (!args_.isEmpty()) {
+ if (args_.isEmpty()) {
+ sb.append('`').append(name_).append('`');
+ } else {
+ sb.append(name_);
sb.append("(");
sb.append(Joiner.on(",").join(args_));
sb.append(")");
Quanlong Huang <hu...@gmail.com> 于2018年8月24日周五 下午12:09写道：
> Unfortunately, the queries come from a system that we don't have control.
> I think we should change the behavior of the CreateView statement to save
> everything in the comments.
>
> Lars Volker <lv...@cloudera.com> 于2018年8月24日周五 上午11:37写道：
>
>> We changed query hints from being interpreted as strings to being parsed
>> properly in commit ce9b332e
>> <https://github.com/apache/impala/commit/ce9b332ee9e640d79c8ae35e7abb8c7d787ddf78#diff-27ff375b57f7fc982be98c4f503cb04dR2385>
>> for IMPALA-4163 <https://issues.apache.org/jira/browse/IMPALA-4163>.
>> Before that change we called String.trim() on the hint which I suspect
>> preserved the ``. After this change we parse it as an IDENT, which likely
>> strips the ``.
>>
>> I feel that hints are not the right solution to embed a query id into the
>> view definition, but I cannot think of a better one, either. With the
>> current syntax you can write the hint as queryId_abc or queryId(abc) and
>> while you'll get a bunch of warning it seems to work on my end.
>>
>> I also think there's a (unrelated) bug in SelectList.java:87
>> <https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87>:
>> It should have a continue statement inside the check, or wrap the call to
>> setIsStraightJoin() in an else clause.
>>
>> On Thu, Aug 23, 2018 at 8:19 PM Quanlong Huang <hu...@gmail.com>
>> wrote:
>>
>>> Yeah, the results of CreateView statement is different. We created the
>>> same view by impala-2.5. The difference is that impala-2.12 delete the
>>> quotes in the comment.
>>>
>>> Found the difference in beeline:
>>>
>>> > show create table view_2_12;
>>>
>>> createtab_stmt
>>>
>>> CREATE VIEW `view_2_12` AS SELECT
>>>
>>> -- +queryId=abc
>>>
>>> * FROM test.video_parq
>>>
>>> 3 rows selected (4.518 seconds)
>>>
>>>
>>> > show create table view_2_5;
>>>
>>> createtab_stmt
>>>
>>> CREATE VIEW `view_2_5` AS SELECT
>>>
>>> -- +`queryId=abc`
>>>
>>> * FROM test.video_parq
>>>
>>> 3 rows selected (1.202 seconds)
>>>
>>> view_2_12 is created by impala-2.12, while view_2_5 is created by
>>> impala-2.5.
>>>
>>> Todd Lipcon <to...@cloudera.com> 于2018年8月24日周五 上午10:53写道：
>>>
>>>> I'm curious: can you describe the view using Hive to see what the
>>>> stored query consists of?
>>>>
>>>> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> After we upgrade Impala from 2.5 to 2.12, we found some queries on
>>>>> views
>>>>> failed. The views contain a query hint which I think is the cause. I
>>>>> can't
>>>>> find anything about such kind of incompatibility was mentioned in
>>>>>
>>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>>>>> or
>>>>>
>>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>>>>
>>>>> The error can be reproduced by
>>>>>
>>>>> [impala-shell:21000] > use test;
>>>>> [impala-shell:21000] > create view view_2_12 as select /*
>>>>> +`queryId=abc` */ * from video_parq;
>>>>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>>>>> video_parq
>>>>> +------------------------+
>>>>> | summary |
>>>>> +------------------------+
>>>>> | View has been created. |
>>>>> +------------------------+
>>>>> WARNINGS: PLAN hint not recognized: queryId=abc
>>>>>
>>>>> Fetched 1 row(s) in 24.00s
>>>>> [impala-shell:21000] > desc formatted view_2_12;
>>>>> Query: describe formatted view_2_12
>>>>> ERROR: AnalysisException: Failed to parse view-definition statement of
>>>>> view: test.view_2_12
>>>>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>>>>> statement of view: test.view_2_12
>>>>>
>>>>>
>>>>> Since this's an emergency for us, I post an email here for help. I'm
>>>>> also
>>>>> looking into the history of the cup file and looking for a hotfix for
>>>>> this.
>>>>> Thanks if anyone can give some pointers!
>>>>>
>>>>> Thanks,
>>>>> Quanlong
>>>>>
>>>>
Re: Breaking change for query hints?
Posted by Quanlong Huang <hu...@gmail.com>.
Unfortunately, the queries come from a system that we don't have control. I
think we should change the behavior of the CreateView statement to save
everything in the comments.
Lars Volker <lv...@cloudera.com> 于2018年8月24日周五 上午11:37写道：
> We changed query hints from being interpreted as strings to being parsed
> properly in commit ce9b332e
> <https://github.com/apache/impala/commit/ce9b332ee9e640d79c8ae35e7abb8c7d787ddf78#diff-27ff375b57f7fc982be98c4f503cb04dR2385>
> for IMPALA-4163 <https://issues.apache.org/jira/browse/IMPALA-4163>.
> Before that change we called String.trim() on the hint which I suspect
> preserved the ``. After this change we parse it as an IDENT, which likely
> strips the ``.
>
> I feel that hints are not the right solution to embed a query id into the
> view definition, but I cannot think of a better one, either. With the
> current syntax you can write the hint as queryId_abc or queryId(abc) and
> while you'll get a bunch of warning it seems to work on my end.
>
> I also think there's a (unrelated) bug in SelectList.java:87
> <https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87>:
> It should have a continue statement inside the check, or wrap the call to
> setIsStraightJoin() in an else clause.
>
> On Thu, Aug 23, 2018 at 8:19 PM Quanlong Huang <hu...@gmail.com>
> wrote:
>
>> Yeah, the results of CreateView statement is different. We created the
>> same view by impala-2.5. The difference is that impala-2.12 delete the
>> quotes in the comment.
>>
>> Found the difference in beeline:
>>
>> > show create table view_2_12;
>>
>> createtab_stmt
>>
>> CREATE VIEW `view_2_12` AS SELECT
>>
>> -- +queryId=abc
>>
>> * FROM test.video_parq
>>
>> 3 rows selected (4.518 seconds)
>>
>>
>> > show create table view_2_5;
>>
>> createtab_stmt
>>
>> CREATE VIEW `view_2_5` AS SELECT
>>
>> -- +`queryId=abc`
>>
>> * FROM test.video_parq
>>
>> 3 rows selected (1.202 seconds)
>>
>> view_2_12 is created by impala-2.12, while view_2_5 is created by
>> impala-2.5.
>>
>> Todd Lipcon <to...@cloudera.com> 于2018年8月24日周五 上午10:53写道：
>>
>>> I'm curious: can you describe the view using Hive to see what the stored
>>> query consists of?
>>>
>>> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
>>>> failed. The views contain a query hint which I think is the cause. I
>>>> can't
>>>> find anything about such kind of incompatibility was mentioned in
>>>>
>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>>>> or
>>>>
>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>>>
>>>> The error can be reproduced by
>>>>
>>>> [impala-shell:21000] > use test;
>>>> [impala-shell:21000] > create view view_2_12 as select /*
>>>> +`queryId=abc` */ * from video_parq;
>>>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>>>> video_parq
>>>> +------------------------+
>>>> | summary |
>>>> +------------------------+
>>>> | View has been created. |
>>>> +------------------------+
>>>> WARNINGS: PLAN hint not recognized: queryId=abc
>>>>
>>>> Fetched 1 row(s) in 24.00s
>>>> [impala-shell:21000] > desc formatted view_2_12;
>>>> Query: describe formatted view_2_12
>>>> ERROR: AnalysisException: Failed to parse view-definition statement of
>>>> view: test.view_2_12
>>>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>>>> statement of view: test.view_2_12
>>>>
>>>>
>>>> Since this's an emergency for us, I post an email here for help. I'm
>>>> also
>>>> looking into the history of the cup file and looking for a hotfix for
>>>> this.
>>>> Thanks if anyone can give some pointers!
>>>>
>>>> Thanks,
>>>> Quanlong
>>>>
>>>
Re: Breaking change for query hints?
Posted by Quanlong Huang <hu...@gmail.com>.
Unfortunately, the queries come from a system that we don't have control. I
think we should change the behavior of the CreateView statement to save
everything in the comments.
Lars Volker <lv...@cloudera.com> 于2018年8月24日周五 上午11:37写道：
> We changed query hints from being interpreted as strings to being parsed
> properly in commit ce9b332e
> <https://github.com/apache/impala/commit/ce9b332ee9e640d79c8ae35e7abb8c7d787ddf78#diff-27ff375b57f7fc982be98c4f503cb04dR2385>
> for IMPALA-4163 <https://issues.apache.org/jira/browse/IMPALA-4163>.
> Before that change we called String.trim() on the hint which I suspect
> preserved the ``. After this change we parse it as an IDENT, which likely
> strips the ``.
>
> I feel that hints are not the right solution to embed a query id into the
> view definition, but I cannot think of a better one, either. With the
> current syntax you can write the hint as queryId_abc or queryId(abc) and
> while you'll get a bunch of warning it seems to work on my end.
>
> I also think there's a (unrelated) bug in SelectList.java:87
> <https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87>:
> It should have a continue statement inside the check, or wrap the call to
> setIsStraightJoin() in an else clause.
>
> On Thu, Aug 23, 2018 at 8:19 PM Quanlong Huang <hu...@gmail.com>
> wrote:
>
>> Yeah, the results of CreateView statement is different. We created the
>> same view by impala-2.5. The difference is that impala-2.12 delete the
>> quotes in the comment.
>>
>> Found the difference in beeline:
>>
>> > show create table view_2_12;
>>
>> createtab_stmt
>>
>> CREATE VIEW `view_2_12` AS SELECT
>>
>> -- +queryId=abc
>>
>> * FROM test.video_parq
>>
>> 3 rows selected (4.518 seconds)
>>
>>
>> > show create table view_2_5;
>>
>> createtab_stmt
>>
>> CREATE VIEW `view_2_5` AS SELECT
>>
>> -- +`queryId=abc`
>>
>> * FROM test.video_parq
>>
>> 3 rows selected (1.202 seconds)
>>
>> view_2_12 is created by impala-2.12, while view_2_5 is created by
>> impala-2.5.
>>
>> Todd Lipcon <to...@cloudera.com> 于2018年8月24日周五 上午10:53写道：
>>
>>> I'm curious: can you describe the view using Hive to see what the stored
>>> query consists of?
>>>
>>> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
>>>> failed. The views contain a query hint which I think is the cause. I
>>>> can't
>>>> find anything about such kind of incompatibility was mentioned in
>>>>
>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>>>> or
>>>>
>>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>>>
>>>> The error can be reproduced by
>>>>
>>>> [impala-shell:21000] > use test;
>>>> [impala-shell:21000] > create view view_2_12 as select /*
>>>> +`queryId=abc` */ * from video_parq;
>>>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>>>> video_parq
>>>> +------------------------+
>>>> | summary |
>>>> +------------------------+
>>>> | View has been created. |
>>>> +------------------------+
>>>> WARNINGS: PLAN hint not recognized: queryId=abc
>>>>
>>>> Fetched 1 row(s) in 24.00s
>>>> [impala-shell:21000] > desc formatted view_2_12;
>>>> Query: describe formatted view_2_12
>>>> ERROR: AnalysisException: Failed to parse view-definition statement of
>>>> view: test.view_2_12
>>>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>>>> statement of view: test.view_2_12
>>>>
>>>>
>>>> Since this's an emergency for us, I post an email here for help. I'm
>>>> also
>>>> looking into the history of the cup file and looking for a hotfix for
>>>> this.
>>>> Thanks if anyone can give some pointers!
>>>>
>>>> Thanks,
>>>> Quanlong
>>>>
>>>
Re: Breaking change for query hints?
Posted by Lars Volker <lv...@cloudera.com>.
We changed query hints from being interpreted as strings to being parsed
properly in commit ce9b332e
<https://github.com/apache/impala/commit/ce9b332ee9e640d79c8ae35e7abb8c7d787ddf78#diff-27ff375b57f7fc982be98c4f503cb04dR2385>
for IMPALA-4163 <https://issues.apache.org/jira/browse/IMPALA-4163>.
Before that change we called String.trim() on the hint which I suspect
preserved the ``. After this change we parse it as an IDENT, which likely
strips the ``.
I feel that hints are not the right solution to embed a query id into the
view definition, but I cannot think of a better one, either. With the
current syntax you can write the hint as queryId_abc or queryId(abc) and
while you'll get a bunch of warning it seems to work on my end.
I also think there's a (unrelated) bug in SelectList.java:87
<https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87>:
It should have a continue statement inside the check, or wrap the call to
setIsStraightJoin() in an else clause.
On Thu, Aug 23, 2018 at 8:19 PM Quanlong Huang <hu...@gmail.com>
wrote:
> Yeah, the results of CreateView statement is different. We created the
> same view by impala-2.5. The difference is that impala-2.12 delete the
> quotes in the comment.
>
> Found the difference in beeline:
>
> > show create table view_2_12;
>
> createtab_stmt
>
> CREATE VIEW `view_2_12` AS SELECT
>
> -- +queryId=abc
>
> * FROM test.video_parq
>
> 3 rows selected (4.518 seconds)
>
>
> > show create table view_2_5;
>
> createtab_stmt
>
> CREATE VIEW `view_2_5` AS SELECT
>
> -- +`queryId=abc`
>
> * FROM test.video_parq
>
> 3 rows selected (1.202 seconds)
>
> view_2_12 is created by impala-2.12, while view_2_5 is created by
> impala-2.5.
>
> Todd Lipcon <to...@cloudera.com> 于2018年8月24日周五 上午10:53写道：
>
>> I'm curious: can you describe the view using Hive to see what the stored
>> query consists of?
>>
>> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
>>> failed. The views contain a query hint which I think is the cause. I
>>> can't
>>> find anything about such kind of incompatibility was mentioned in
>>>
>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>>> or
>>>
>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>>
>>> The error can be reproduced by
>>>
>>> [impala-shell:21000] > use test;
>>> [impala-shell:21000] > create view view_2_12 as select /*
>>> +`queryId=abc` */ * from video_parq;
>>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>>> video_parq
>>> +------------------------+
>>> | summary |
>>> +------------------------+
>>> | View has been created. |
>>> +------------------------+
>>> WARNINGS: PLAN hint not recognized: queryId=abc
>>>
>>> Fetched 1 row(s) in 24.00s
>>> [impala-shell:21000] > desc formatted view_2_12;
>>> Query: describe formatted view_2_12
>>> ERROR: AnalysisException: Failed to parse view-definition statement of
>>> view: test.view_2_12
>>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>>> statement of view: test.view_2_12
>>>
>>>
>>> Since this's an emergency for us, I post an email here for help. I'm also
>>> looking into the history of the cup file and looking for a hotfix for
>>> this.
>>> Thanks if anyone can give some pointers!
>>>
>>> Thanks,
>>> Quanlong
>>>
>>
Re: Breaking change for query hints?
Posted by Lars Volker <lv...@cloudera.com>.
We changed query hints from being interpreted as strings to being parsed
properly in commit ce9b332e
<https://github.com/apache/impala/commit/ce9b332ee9e640d79c8ae35e7abb8c7d787ddf78#diff-27ff375b57f7fc982be98c4f503cb04dR2385>
for IMPALA-4163 <https://issues.apache.org/jira/browse/IMPALA-4163>.
Before that change we called String.trim() on the hint which I suspect
preserved the ``. After this change we parse it as an IDENT, which likely
strips the ``.
I feel that hints are not the right solution to embed a query id into the
view definition, but I cannot think of a better one, either. With the
current syntax you can write the hint as queryId_abc or queryId(abc) and
while you'll get a bunch of warning it seems to work on my end.
I also think there's a (unrelated) bug in SelectList.java:87
<https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87>:
It should have a continue statement inside the check, or wrap the call to
setIsStraightJoin() in an else clause.
On Thu, Aug 23, 2018 at 8:19 PM Quanlong Huang <hu...@gmail.com>
wrote:
> Yeah, the results of CreateView statement is different. We created the
> same view by impala-2.5. The difference is that impala-2.12 delete the
> quotes in the comment.
>
> Found the difference in beeline:
>
> > show create table view_2_12;
>
> createtab_stmt
>
> CREATE VIEW `view_2_12` AS SELECT
>
> -- +queryId=abc
>
> * FROM test.video_parq
>
> 3 rows selected (4.518 seconds)
>
>
> > show create table view_2_5;
>
> createtab_stmt
>
> CREATE VIEW `view_2_5` AS SELECT
>
> -- +`queryId=abc`
>
> * FROM test.video_parq
>
> 3 rows selected (1.202 seconds)
>
> view_2_12 is created by impala-2.12, while view_2_5 is created by
> impala-2.5.
>
> Todd Lipcon <to...@cloudera.com> 于2018年8月24日周五 上午10:53写道：
>
>> I'm curious: can you describe the view using Hive to see what the stored
>> query consists of?
>>
>> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
>>> failed. The views contain a query hint which I think is the cause. I
>>> can't
>>> find anything about such kind of incompatibility was mentioned in
>>>
>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>>> or
>>>
>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>>
>>> The error can be reproduced by
>>>
>>> [impala-shell:21000] > use test;
>>> [impala-shell:21000] > create view view_2_12 as select /*
>>> +`queryId=abc` */ * from video_parq;
>>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>>> video_parq
>>> +------------------------+
>>> | summary |
>>> +------------------------+
>>> | View has been created. |
>>> +------------------------+
>>> WARNINGS: PLAN hint not recognized: queryId=abc
>>>
>>> Fetched 1 row(s) in 24.00s
>>> [impala-shell:21000] > desc formatted view_2_12;
>>> Query: describe formatted view_2_12
>>> ERROR: AnalysisException: Failed to parse view-definition statement of
>>> view: test.view_2_12
>>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>>> statement of view: test.view_2_12
>>>
>>>
>>> Since this's an emergency for us, I post an email here for help. I'm also
>>> looking into the history of the cup file and looking for a hotfix for
>>> this.
>>> Thanks if anyone can give some pointers!
>>>
>>> Thanks,
>>> Quanlong
>>>
>>
Re: Breaking change for query hints?
Posted by Quanlong Huang <hu...@gmail.com>.
Yeah, the results of CreateView statement is different. We created the same
view by impala-2.5. The difference is that impala-2.12 delete the quotes in
the comment.
Found the difference in beeline:
> show create table view_2_12;
createtab_stmt
CREATE VIEW `view_2_12` AS SELECT
-- +queryId=abc
* FROM test.video_parq
3 rows selected (4.518 seconds)
> show create table view_2_5;
createtab_stmt
CREATE VIEW `view_2_5` AS SELECT
-- +`queryId=abc`
* FROM test.video_parq
3 rows selected (1.202 seconds)
view_2_12 is created by impala-2.12, while view_2_5 is created by
impala-2.5.
Todd Lipcon <to...@cloudera.com> 于2018年8月24日周五 上午10:53写道：
> I'm curious: can you describe the view using Hive to see what the stored
> query consists of?
>
> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
>> failed. The views contain a query hint which I think is the cause. I can't
>> find anything about such kind of incompatibility was mentioned in
>>
>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>> or
>>
>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>
>> The error can be reproduced by
>>
>> [impala-shell:21000] > use test;
>> [impala-shell:21000] > create view view_2_12 as select /*
>> +`queryId=abc` */ * from video_parq;
>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>> video_parq
>> +------------------------+
>> | summary |
>> +------------------------+
>> | View has been created. |
>> +------------------------+
>> WARNINGS: PLAN hint not recognized: queryId=abc
>>
>> Fetched 1 row(s) in 24.00s
>> [impala-shell:21000] > desc formatted view_2_12;
>> Query: describe formatted view_2_12
>> ERROR: AnalysisException: Failed to parse view-definition statement of
>> view: test.view_2_12
>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>> statement of view: test.view_2_12
>>
>>
>> Since this's an emergency for us, I post an email here for help. I'm also
>> looking into the history of the cup file and looking for a hotfix for
>> this.
>> Thanks if anyone can give some pointers!
>>
>> Thanks,
>> Quanlong
>>
>
Re: Breaking change for query hints?
Posted by Lars Volker <lv...@cloudera.com>.
The warning seems to come from SelectList.java:87
<https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87>
which
also was in 2.5 (SelectList.java:85
<https://github.com/cloudera/Impala/blob/cdh5-2.5.0_5.7.0/fe/src/main/java/com/cloudera/impala/analysis/SelectList.java#L85>.
Do you see corresponding errors in the logfiles for the AnalysisException?
On Thu, Aug 23, 2018 at 7:53 PM Todd Lipcon <to...@cloudera.com> wrote:
> I'm curious: can you describe the view using Hive to see what the stored
> query consists of?
>
> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
>> failed. The views contain a query hint which I think is the cause. I can't
>> find anything about such kind of incompatibility was mentioned in
>>
>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>> or
>>
>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>
>> The error can be reproduced by
>>
>> [impala-shell:21000] > use test;
>> [impala-shell:21000] > create view view_2_12 as select /*
>> +`queryId=abc` */ * from video_parq;
>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>> video_parq
>> +------------------------+
>> | summary |
>> +------------------------+
>> | View has been created. |
>> +------------------------+
>> WARNINGS: PLAN hint not recognized: queryId=abc
>>
>> Fetched 1 row(s) in 24.00s
>> [impala-shell:21000] > desc formatted view_2_12;
>> Query: describe formatted view_2_12
>> ERROR: AnalysisException: Failed to parse view-definition statement of
>> view: test.view_2_12
>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>> statement of view: test.view_2_12
>>
>>
>> Since this's an emergency for us, I post an email here for help. I'm also
>> looking into the history of the cup file and looking for a hotfix for
>> this.
>> Thanks if anyone can give some pointers!
>>
>> Thanks,
>> Quanlong
>>
>
Re: Breaking change for query hints?
Posted by Quanlong Huang <hu...@gmail.com>.
Yeah, the results of CreateView statement is different. We created the same
view by impala-2.5. The difference is that impala-2.12 delete the quotes in
the comment.
Found the difference in beeline:
> show create table view_2_12;
createtab_stmt
CREATE VIEW `view_2_12` AS SELECT
-- +queryId=abc
* FROM test.video_parq
3 rows selected (4.518 seconds)
> show create table view_2_5;
createtab_stmt
CREATE VIEW `view_2_5` AS SELECT
-- +`queryId=abc`
* FROM test.video_parq
3 rows selected (1.202 seconds)
view_2_12 is created by impala-2.12, while view_2_5 is created by
impala-2.5.
Todd Lipcon <to...@cloudera.com> 于2018年8月24日周五 上午10:53写道：
> I'm curious: can you describe the view using Hive to see what the stored
> query consists of?
>
> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
>> failed. The views contain a query hint which I think is the cause. I can't
>> find anything about such kind of incompatibility was mentioned in
>>
>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>> or
>>
>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>
>> The error can be reproduced by
>>
>> [impala-shell:21000] > use test;
>> [impala-shell:21000] > create view view_2_12 as select /*
>> +`queryId=abc` */ * from video_parq;
>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>> video_parq
>> +------------------------+
>> | summary |
>> +------------------------+
>> | View has been created. |
>> +------------------------+
>> WARNINGS: PLAN hint not recognized: queryId=abc
>>
>> Fetched 1 row(s) in 24.00s
>> [impala-shell:21000] > desc formatted view_2_12;
>> Query: describe formatted view_2_12
>> ERROR: AnalysisException: Failed to parse view-definition statement of
>> view: test.view_2_12
>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>> statement of view: test.view_2_12
>>
>>
>> Since this's an emergency for us, I post an email here for help. I'm also
>> looking into the history of the cup file and looking for a hotfix for
>> this.
>> Thanks if anyone can give some pointers!
>>
>> Thanks,
>> Quanlong
>>
>
Re: Breaking change for query hints?
Posted by Lars Volker <lv...@cloudera.com>.
The warning seems to come from SelectList.java:87
<https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87>
which
also was in 2.5 (SelectList.java:85
<https://github.com/cloudera/Impala/blob/cdh5-2.5.0_5.7.0/fe/src/main/java/com/cloudera/impala/analysis/SelectList.java#L85>.
Do you see corresponding errors in the logfiles for the AnalysisException?
On Thu, Aug 23, 2018 at 7:53 PM Todd Lipcon <to...@cloudera.com> wrote:
> I'm curious: can you describe the view using Hive to see what the stored
> query consists of?
>
> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
>> failed. The views contain a query hint which I think is the cause. I can't
>> find anything about such kind of incompatibility was mentioned in
>>
>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>> or
>>
>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>
>> The error can be reproduced by
>>
>> [impala-shell:21000] > use test;
>> [impala-shell:21000] > create view view_2_12 as select /*
>> +`queryId=abc` */ * from video_parq;
>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>> video_parq
>> +------------------------+
>> | summary |
>> +------------------------+
>> | View has been created. |
>> +------------------------+
>> WARNINGS: PLAN hint not recognized: queryId=abc
>>
>> Fetched 1 row(s) in 24.00s
>> [impala-shell:21000] > desc formatted view_2_12;
>> Query: describe formatted view_2_12
>> ERROR: AnalysisException: Failed to parse view-definition statement of
>> view: test.view_2_12
>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>> statement of view: test.view_2_12
>>
>>
>> Since this's an emergency for us, I post an email here for help. I'm also
>> looking into the history of the cup file and looking for a hotfix for
>> this.
>> Thanks if anyone can give some pointers!
>>
>> Thanks,
>> Quanlong
>>
>
Re: Breaking change for query hints?
Posted by Todd Lipcon <to...@cloudera.com>.
I'm curious: can you describe the view using Hive to see what the stored
query consists of?
On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
wrote:
> Hi all,
>
> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
> failed. The views contain a query hint which I think is the cause. I can't
> find anything about such kind of incompatibility was mentioned in
>
> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
> or
>
> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>
> The error can be reproduced by
>
> [impala-shell:21000] > use test;
> [impala-shell:21000] > create view view_2_12 as select /*
> +`queryId=abc` */ * from video_parq;
> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
> video_parq
> +------------------------+
> | summary |
> +------------------------+
> | View has been created. |
> +------------------------+
> WARNINGS: PLAN hint not recognized: queryId=abc
>
> Fetched 1 row(s) in 24.00s
> [impala-shell:21000] > desc formatted view_2_12;
> Query: describe formatted view_2_12
> ERROR: AnalysisException: Failed to parse view-definition statement of
> view: test.view_2_12
> CAUSED BY: TableLoadingException: Failed to parse view-definition
> statement of view: test.view_2_12
>
>
> Since this's an emergency for us, I post an email here for help. I'm also
> looking into the history of the cup file and looking for a hotfix for this.
> Thanks if anyone can give some pointers!
>
> Thanks,
> Quanlong
>
Re: Breaking change for query hints?
Posted by Todd Lipcon <to...@cloudera.com>.
I'm curious: can you describe the view using Hive to see what the stored
query consists of?
On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang <hu...@gmail.com>
wrote:
> Hi all,
>
> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
> failed. The views contain a query hint which I think is the cause. I can't
> find anything about such kind of incompatibility was mentioned in
>
> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
> or
>
> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>
> The error can be reproduced by
>
> [impala-shell:21000] > use test;
> [impala-shell:21000] > create view view_2_12 as select /*
> +`queryId=abc` */ * from video_parq;
> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
> video_parq
> +------------------------+
> | summary |
> +------------------------+
> | View has been created. |
> +------------------------+
> WARNINGS: PLAN hint not recognized: queryId=abc
>
> Fetched 1 row(s) in 24.00s
> [impala-shell:21000] > desc formatted view_2_12;
> Query: describe formatted view_2_12
> ERROR: AnalysisException: Failed to parse view-definition statement of
> view: test.view_2_12
> CAUSED BY: TableLoadingException: Failed to parse view-definition
> statement of view: test.view_2_12
>
>
> Since this's an emergency for us, I post an email here for help. I'm also
> looking into the history of the cup file and looking for a hotfix for this.
> Thanks if anyone can give some pointers!
>
> Thanks,
> Quanlong
>