You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Kim Jin Chul (Code Review)" <ge...@cloudera.org> on 2017/12/12 07:37:41 UTC

[Impala-ASF-CR] IMPALA-3942: Fix unexpected conversion of string literal in front-end

Kim Jin Chul has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8818


Change subject: IMPALA-3942: Fix unexpected conversion of string literal in front-end
......................................................................

IMPALA-3942: Fix unexpected conversion of string literal in front-end

Unexpected conversion from double quoted string literal to single quoted
string literal. A given string literal should be kept.

Testing:
Add some test cases to
TestStringLiteralBetweenDoubleQuotes.test_string_literal_between_double_quotes

Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
---
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/main/jflex/sql-scanner.flex
M tests/query_test/test_queries.py
3 files changed, 53 insertions(+), 6 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8818/1
-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 1
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, Jim Apple, Tim Armstrong, Alex Behm, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8818

to look at the new patch set (#2).

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................

IMPALA-3942: Fix wronly escaped string literal in front-end

String literal can be wrapped by either single or double quotes.
There are some holes in escaping the string literal. The solution
is to normalize any string which comes from user's given string or
a generated string (e.g. constant fold by the rewritter rule).

Testing:
Add some test cases to TestEscapingStringLiteral

Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
---
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M tests/query_test/test_queries.py
2 files changed, 112 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8818/2
-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 2
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................


Patch Set 2:

(1 comment)

We should get this moving along again now that people are getting back from holidays.

http://gerrit.cloudera.org:8080/#/c/8818/2/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/8818/2/tests/query_test/test_queries.py@315
PS2, Line 315:     self.prepare()
We should use the unique_database test fixture instead of this approach. E.g. test_scanners.py has a bunch of tests that create tables in temporary databases.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 2
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Jan 2018 21:50:06 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................


Patch Set 10: Code-Review+2

Thanks for your patience and contribution! Nice work.


-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 10
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jan 2018 16:51:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................


Patch Set 7:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@87
PS7, Line 87:     return BaseSemanticAnalyzer.unescapeSQLString("'" + getNormalizedValue() 
> whitespace
Done


http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@91
PS7, Line 91:   // Assumes a string literal from single quotes and escapes it because:
> What does this function do and return? Please consider these snippets which
Done


http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@100
PS7, Line 100:   private String getNormalizedValue() {
> nit: we typically use /* */ style comment for class and function comments
Done


http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@101
PS7, Line 101:     final int lengthOfValue = value_.length();
> len?
Done


http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
File fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java:

http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java@468
PS7, Line 468:   public void escapeStringLiteralTest() {
> normalizeStringLiteralsTest
Done


http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java@471
PS7, Line 471:     testToSql("select \"\\\\'\"", "SELECT '\\\\\\''");
> add a few tests with single quotes that show they are not transformed
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 7
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Jan 2018 15:11:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@104
PS2, Line 104: getEscapedValue
Escaping and unescaping the value happen here. Do you have more intuitive name for this? (e.g. getNormalizedValue)


http://gerrit.cloudera.org:8080/#/c/8818/2/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/8818/2/tests/query_test/test_queries.py@276
PS2, Line 276:   def test_simple(self):
Please let me know if there is any missing tests.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 2
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2017 08:35:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................


Patch Set 9:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py@310
PS6, Line 310: 
> I'm ok with leaving some E2E tests.
I just realized this comment has not been addressed, sorry. Can you move these tests?

You might put them in views-ddl.test or test_ddl.py



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 9
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jan 2018 06:06:00 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix unexpected conversion of string literal in front-end

Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix unexpected conversion of string literal in front-end
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8818/1/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/1/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@49
PS1, Line 49:     boolean useSingleQuote = true;
Just considering what options we have here:
Can't we store the value as it comes in (regardless if it's quoted or not) and then enhance the compareTo(), equals() etc functions to handle whether the value_ is not quoted, single quoted or double quoted?



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 1
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Comment-Date: Tue, 12 Dec 2017 11:08:00 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix unexpected conversion of string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix unexpected conversion of string literal in front-end
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8818/1/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/1/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@49
PS1, Line 49:     boolean useSingleQuote = true;
> Just considering what options we have here:
Your suggestion is possible. Actually I tried what you suggested firstly which might be same as Jim's proposal, but I had to handle some corner cases because value_ contains can have non of/single/double quotes. I still have the ongoing code locally, but the change is bigger than this and I am not sure any hole still exists. If you and Jim still think "keeping a given string literal at  value_", please let me know pros and cons compared with this version.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 1
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Dec 2017 00:10:23 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix unexpected conversion of string literal in front-end

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix unexpected conversion of string literal in front-end
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8818/1/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/1/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@49
PS1, Line 49:     boolean useSingleQuote = true;
> Your suggestion is possible. Actually I tried what you suggested firstly wh
I think all proposed approaches do not consider that string literals can also be produced by constant folding.

For any query with a string literal in SQL we can write an equivalent query that uses concat() to produce the same string literal via constant folding. Both cases should behave the same.

It's definitely possible that we have more bugs in this area.

So for some string literals we may not know whether to use single or double quotes.

To me the most sane solution is to normalize any string before toSql(). I have not thought this through completely, but I think it might be safe to always unescape the string and then escape all single and double quotes. Then it should not matter what quote is used on the outside.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 1
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Dec 2017 18:13:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................


Patch Set 6: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 6
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2018 12:48:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Gabor Kaszab, Jim Apple, Tim Armstrong, Alex Behm, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8818

to look at the new patch set (#3).

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................

IMPALA-3942: Fix wronly escaped string literal in front-end

String literal can be wrapped by either single or double quotes.
There are some holes in escaping the string literal:
- toSql() always returns strings that are single quoted, resulting in
improper escaping in the output if the original string was actually
double quoted.
- It is not always possible to determine if a string "should" be single
or double quoted, eg concat('a', "b") and that's why we just choose one
to always normalize to.

The solution is to normalize any string which comes from user's given
string or a generated string (e.g. constant fold by the rewritter rule).

Testing:
Add some test cases to TestEscapingStringLiteral

Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
---
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M tests/query_test/test_queries.py
2 files changed, 115 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8818/3
-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 3
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Gabor Kaszab, Jim Apple, Tim Armstrong, Alex Behm, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8818

to look at the new patch set (#10).

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................

IMPALA-3942: Fix wrongly escaped string literal in front-end

String literal can be wrapped by either single or double quotes.
There are some holes in escaping the string literal:
- toSql() always returns strings that are single quoted, resulting in
improper escaping in the output if the original string was actually
double quoted.
- It is not always possible to determine if a string "should" be single
or double quoted. e.g. concat('a', "b")

The solution is to normalize any string which comes from user's given
string or a generated string (e.g. constant fold by the rewritter rule)
to single quotes.

Testing:
Added unit tests into expr-test, ToSqlTest

Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
---
M be/src/exprs/expr-test.cc
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
3 files changed, 93 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8818/10
-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 10
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................


Patch Set 11:

Thanks you all for the reviews!


-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 11
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jan 2018 20:53:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................


Patch Set 7:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@44
PS6, Line 44:     this(value, ScalarType.STRING, true);
> Done. By the way, would you tell me the reason why you don't want to keep t
That's a fair question, thanks for asking!

* In most cases there will be no redundant computation. The common case is that this literal is passed to the BE via toThrift() so getNormalizedValue() is only called once.

* I'm less concerned about redundant computations and more concerned about doubling the memory consumption. This point will probably only matter in extreme cases.

* Easier to reason about and maintain fewer class members. The normalized string is "redundant" information because it can be derived from 'value_'.


http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@87
PS7, Line 87:     return BaseSemanticAnalyzer.unescapeSQLString("'" + getNormalizedValue() 
whitespace


http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@91
PS7, Line 91:   // Assumes a string literal from single quotes and escapes it because:
What does this function do and return? Please consider these snippets which might improve the comment:

Returns a normalized representation of the string value suitable for embedding in SQL as a single-quoted string literal.
String literals can come directly from the SQL of a query or from rewrites like constant folding, so we generally do not know whether single or double quotes are appropriate for a given value.
The SQL standard prescribes single-quoted string literals so we normalize that way.


http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@100
PS7, Line 100:   private String getNormalizedValue() {
nit: we typically use /* */ style comment for class and function comments


http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@101
PS7, Line 101:     final int lengthOfValue = value_.length();
len?


http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
File fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java:

http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java@468
PS7, Line 468:   public void escapeStringLiteralTest() {
normalizeStringLiteralsTest


http://gerrit.cloudera.org:8080/#/c/8818/7/fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java@471
PS7, Line 471:     testToSql("select \"\\\\'\"", "SELECT '\\\\\\''");
add a few tests with single quotes that show they are not transformed


http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py@310
PS6, Line 310: 
> I agree with your suggestion. ToSqlTest mainly checks the problem. By the w
I'm ok with leaving some E2E tests.

Imo they belong in views-ddl.test since the user-visible bug is really around creating/using views, and the toSql() issue is already covered by ToSqlTest unit tests.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 7
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jan 2018 00:20:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Gabor Kaszab, Jim Apple, Tim Armstrong, Alex Behm, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8818

to look at the new patch set (#4).

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................

IMPALA-3942: Fix wronly escaped string literal in front-end

String literal can be wrapped by either single or double quotes.
There are some holes in escaping the string literal:
- toSql() always returns strings that are single quoted, resulting in
improper escaping in the output if the original string was actually
double quoted.
- It is not always possible to determine if a string "should" be single
or double quoted, eg concat('a', "b") and that's why we just choose one
to always normalize to.

The solution is to normalize any string which comes from user's given
string or a generated string (e.g. constant fold by the rewritter rule).

Testing:
Add some test cases to TestEscapingStringLiteral

Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
---
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M tests/query_test/test_queries.py
2 files changed, 113 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8818/4
-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 4
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Gabor Kaszab, Jim Apple, Tim Armstrong, Alex Behm, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8818

to look at the new patch set (#6).

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................

IMPALA-3942: Fix wronly escaped string literal in front-end

String literal can be wrapped by either single or double quotes.
There are some holes in escaping the string literal:
- toSql() always returns strings that are single quoted, resulting in
improper escaping in the output if the original string was actually
double quoted.
- It is not always possible to determine if a string "should" be single
or double quoted, eg concat('a', "b") and that's why we just choose one
to always normalize to.

The solution is to normalize any string which comes from user's given
string or a generated string (e.g. constant fold by the rewritter rule).

Testing:
Add some test cases to TestEscapingStringLiteral

Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
---
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M tests/query_test/test_queries.py
2 files changed, 110 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8818/6
-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 6
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Gabor Kaszab, Jim Apple, Tim Armstrong, Alex Behm, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8818

to look at the new patch set (#5).

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................

IMPALA-3942: Fix wronly escaped string literal in front-end

String literal can be wrapped by either single or double quotes.
There are some holes in escaping the string literal:
- toSql() always returns strings that are single quoted, resulting in
improper escaping in the output if the original string was actually
double quoted.
- It is not always possible to determine if a string "should" be single
or double quoted, eg concat('a', "b") and that's why we just choose one
to always normalize to.

The solution is to normalize any string which comes from user's given
string or a generated string (e.g. constant fold by the rewritter rule).

Testing:
Add some test cases to TestEscapingStringLiteral

Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
---
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M tests/query_test/test_queries.py
2 files changed, 115 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8818/5
-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 5
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................


Patch Set 8:

(1 comment)

Applied the update.

http://gerrit.cloudera.org:8080/#/c/8818/8/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/8/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@92
PS8, Line 92:    *  String literal can come directly from the SQL of a query or from rewrites like
> Some grammar issues, suggest this correction:
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 8
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jan 2018 04:19:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Gabor Kaszab, Jim Apple, Tim Armstrong, Alex Behm, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8818

to look at the new patch set (#8).

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................

IMPALA-3942: Fix wrongly escaped string literal in front-end

String literal can be wrapped by either single or double quotes.
There are some holes in escaping the string literal:
- toSql() always returns strings that are single quoted, resulting in
improper escaping in the output if the original string was actually
double quoted.
- It is not always possible to determine if a string "should" be single
or double quoted. e.g. concat('a', "b")

The solution is to normalize any string which comes from user's given
string or a generated string (e.g. constant fold by the rewritter rule)
to single quotes.

Testing:
Added unit tests into expr-test, ToSqlTest
Added E2E tests into test_queries

Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
---
M be/src/exprs/expr-test.cc
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M tests/query_test/test_queries.py
4 files changed, 113 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8818/8
-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 8
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................


Patch Set 10: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 10
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jan 2018 20:35:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1784/


-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 10
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jan 2018 16:51:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................


Patch Set 6:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@44
PS6, Line 44:   private final String escaped_single_quoted_value_;
Let's not store this value and instead compute it on-the-fly.


http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@110
PS6, Line 110:   private String getEscapedSingleQuotedValue() {
getNormalizedValue()


http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@211
PS6, Line 211:         + escaped_single_quoted_value_ + "' to number.");
let's keep value_ here because that was the original value the user passed


http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py@273
PS6, Line 273:     result = self.execute_query("select 'a\"b'")
My understanding is that these tests do not cover your new changes. These are just plain-old string literals and we return their StringLiteral.value_

Still it's nice to have these StringLiteral tests! They belong in expr-test.cc though


http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py@310
PS6, Line 310:   def test_escaping_string_literal(self, unique_database):
We should have a more focused test for this in ToSqlTest.java. This fix affects StringLiteral.toSql()



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 6
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2018 23:59:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/8818/5/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/5/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@120
PS5, Line 120:             sb.append("\"");
Just asking: Wouldn't this be more readable to say sb.append('"')
(double quote surrounded by single quotes) same as the row above?
Or can't you append nextChar here?

If the answer is yes for the last question then it might seem reasonable to rework this if-else:
if (nextchar is not doublequote) {
  append(currentChar)
}
append(nextchar)


http://gerrit.cloudera.org:8080/#/c/8818/5/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/8818/5/tests/query_test/test_queries.py@272
PS5, Line 272:   def prepare(self, unique_database):
nit: as the only usage of prepare() is in the last test of this file, I'd move this function closer to it.


http://gerrit.cloudera.org:8080/#/c/8818/5/tests/query_test/test_queries.py@297
PS5, Line 297:     result = self.execute_query("select '\\\''")
I'm a bit confused of all these backslashes and single-double quotes to be honest :), but shouldn't this query return:
"\'" instead of "'"?



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 5
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 09 Jan 2018 10:24:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Gabor Kaszab, Jim Apple, Tim Armstrong, Alex Behm, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8818

to look at the new patch set (#9).

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................

IMPALA-3942: Fix wrongly escaped string literal in front-end

String literal can be wrapped by either single or double quotes.
There are some holes in escaping the string literal:
- toSql() always returns strings that are single quoted, resulting in
improper escaping in the output if the original string was actually
double quoted.
- It is not always possible to determine if a string "should" be single
or double quoted. e.g. concat('a', "b")

The solution is to normalize any string which comes from user's given
string or a generated string (e.g. constant fold by the rewritter rule)
to single quotes.

Testing:
Added unit tests into expr-test, ToSqlTest
Added E2E tests into test_queries

Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
---
M be/src/exprs/expr-test.cc
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M tests/query_test/test_queries.py
4 files changed, 113 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8818/9
-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 9
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8818/1/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/1/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@49
PS1, Line 49:     boolean useSingleQuote = true;
> Another, umh, "pragmatic" solution might be to just try both quotes and see
This approach can be implemented easily, but I would prefer the above than this: 
1. To avoid unnecessary overhead from try and error method.
2. More elaborative and robust code.
3. Fix for the problem is not urgent.


http://gerrit.cloudera.org:8080/#/c/8818/1/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@49
PS1, Line 49:     boolean useSingleQuote = true;
> I think all proposed approaches do not consider that string literals can al
Thanks a lot for the information. I didn't aware of the string literal can come from the rewritter rule. The normalization stage should be required, so I introduced getEscapedValue() method in the recent patch set.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 1
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2017 08:32:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................


Patch Set 10:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py@310
PS6, Line 310: 
> I just realized this comment has not been addressed, sorry. Can you move th
As you mentioned in the previous comment, our unit test can fully cover the issue, so I just remove the E2E test.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 10
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jan 2018 15:28:33 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Gabor Kaszab, Jim Apple, Tim Armstrong, Alex Behm, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8818

to look at the new patch set (#7).

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................

IMPALA-3942: Fix wrongly escaped string literal in front-end

String literal can be wrapped by either single or double quotes.
There are some holes in escaping the string literal:
- toSql() always returns strings that are single quoted, resulting in
improper escaping in the output if the original string was actually
double quoted.
- It is not always possible to determine if a string "should" be single
or double quoted. e.g. concat('a', "b")

The solution is to normalize any string which comes from user's given
string or a generated string (e.g. constant fold by the rewritter rule)
to single quotes.

Testing:
Added unit tests into expr-test, ToSqlTest
Added E2E tests into test_queries

Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
---
M be/src/exprs/expr-test.cc
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M tests/query_test/test_queries.py
4 files changed, 110 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8818/7
-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 7
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3942: Fix unexpected conversion of string literal in front-end

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix unexpected conversion of string literal in front-end
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8818/1/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/1/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@49
PS1, Line 49:     boolean useSingleQuote = true;
> I think all proposed approaches do not consider that string literals can al
Another, umh, "pragmatic" solution might be to just try both quotes and see which one parses successfully. It's definitely a hack, but might be ok as a short-term solution since our string literals probably need major reworking in the future to be more standard SQL conforming.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 1
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Dec 2017 18:31:39 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................

IMPALA-3942: Fix wrongly escaped string literal in front-end

String literal can be wrapped by either single or double quotes.
There are some holes in escaping the string literal:
- toSql() always returns strings that are single quoted, resulting in
improper escaping in the output if the original string was actually
double quoted.
- It is not always possible to determine if a string "should" be single
or double quoted. e.g. concat('a', "b")

The solution is to normalize any string which comes from user's given
string or a generated string (e.g. constant fold by the rewritter rule)
to single quotes.

Testing:
Added unit tests into expr-test, ToSqlTest

Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Reviewed-on: http://gerrit.cloudera.org:8080/8818
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/exprs/expr-test.cc
M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
3 files changed, 93 insertions(+), 4 deletions(-)

Approvals:
  Alex Behm: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 11
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................


Patch Set 4:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/8818/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8818/2//COMMIT_MSG@7
PS2, Line 7: wronly
> wrongly
Done


http://gerrit.cloudera.org:8080/#/c/8818/2//COMMIT_MSG@10
PS2, Line 10: There are some holes in escaping the string literal
> Can you expand on this a little more, eg mention that:
Done


http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@39
PS2, Line 39: 
> Add a comment explaining what this is.
Done


http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@104
PS2, Line 104: ression rewritt
> You might name it something like "getEscapedSingleQuotedValue" to make it m
Done


http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@105
PS2, Line 105:  2. Single quotes are closer to the SQL standard.
> Since getEscapedValue() is called in the constructor, this will always be t
Done


http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@121
PS2, Line 121:   s
> nit: ++i
Done


http://gerrit.cloudera.org:8080/#/c/8818/2/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/8818/2/tests/query_test/test_queries.py@315
PS2, Line 315:     result = self.execute_query("select concat(\"a\\\\'b\", 'c\\\\\"d')")
> We should use the unique_database test fixture instead of this approach. E.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 4
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jan 2018 02:31:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/8818/5/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/5/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@120
PS5, Line 120:             sb.append("\"");
> Just asking: Wouldn't this be more readable to say sb.append('"')
You're right. It's more readable! Thanks.


http://gerrit.cloudera.org:8080/#/c/8818/5/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/8818/5/tests/query_test/test_queries.py@272
PS5, Line 272:   def prepare(self, unique_database):
> nit: as the only usage of prepare() is in the last test of this file, I'd m
Done


http://gerrit.cloudera.org:8080/#/c/8818/5/tests/query_test/test_queries.py@297
PS5, Line 297:     result = self.execute_query("select '\\\''")
> I'm a bit confused of all these backslashes and single-double quotes to be 
I still think the current result is valid. The single quote is already escaped, so there is no additional escaping.The result "a single quote character' should be a result of select '\''

Regarding the line 301, the single quote should be escaped explicitly. The result should be same as above. Currently Impala returns empty result for the query in the line 301.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 5
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2018 13:32:44 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end
......................................................................


Patch Set 8: Code-Review+2

(1 comment)

Thanks! I'll merge after you've addressed the final change.

http://gerrit.cloudera.org:8080/#/c/8818/8/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/8/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@92
PS8, Line 92:    *  String literal can come directly from the SQL of a query or from rewrites like
Some grammar issues, suggest this correction:

String literals can come directly from the SQL of a query or from rewrites like constant folding. So this value normalization to a single-quoted string is necessary because we do not know whether single or double quotes are appropriate.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 8
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jan 2018 02:15:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................


Patch Set 2:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/8818/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8818/2//COMMIT_MSG@7
PS2, Line 7: wronly
wrongly


http://gerrit.cloudera.org:8080/#/c/8818/2//COMMIT_MSG@10
PS2, Line 10: There are some holes in escaping the string literal
Can you expand on this a little more, eg mention that:

- toSql() always returns strings that are single quoted, resulting in improper escaping in the output if the original string was actually double quoted
- Its not always possible to determine if a string "should" be single or double quoted, eg concat('a', "b") and that's why we just choose one to always normalize to.


http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@39
PS2, Line 39:   private final String cached_escaped_value_;
Add a comment explaining what this is.

Also, per my comments below, let's get rid of the concept of this being a 'cached' value, and name it something like 'escaped_single_quoted_value_'.


http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@104
PS2, Line 104: getEscapedValue
> Escaping and unescaping the value happen here. Do you have more intuitive n
You might name it something like "getEscapedSingleQuotedValue" to make it more clear what its doing.


http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@105
PS2, Line 105: if (cached_escaped_value_ != null) return cached_escaped_value_;
Since getEscapedValue() is called in the constructor, this will always be true outside the constructor.

So, let's simplify the logic here by removing this and only calling getEscapedValue() in the constructor, then using the cached_escaped_value_ directly in the other places, eg. toSqlImpl().


http://gerrit.cloudera.org:8080/#/c/8818/2/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@121
PS2, Line 121: i++
nit: ++i



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 2
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 06 Jan 2018 00:44:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................


Patch Set 6:

(6 comments)

Code looks good to me, thanks for the contribution!

I have just one more round of comments, all about keeping your comments clear and concise. After that, we should be able to pull a +2-er in

http://gerrit.cloudera.org:8080/#/c/8818/6//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8818/6//COMMIT_MSG@7
PS6, Line 7: wronly
wrongly


http://gerrit.cloudera.org:8080/#/c/8818/6//COMMIT_MSG@15
PS6, Line 15:  and that's why we just choose one
            : to always normalize to
remove this, covered by the paragraph below.


http://gerrit.cloudera.org:8080/#/c/8818/6//COMMIT_MSG@19
PS6, Line 19: .
... to single quotes


http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@41
PS6, Line 41:  It is not always possible to determine if a string "should" be
            :   // single or double quoted(e.g. concat('a', "b")) and that's why we just choose one
            :   // to always normalize to.
This second sentence is not necessary, this is explained better in the larger comment below anyways.


http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@99
PS6, Line 99: .
            :   // Here is a background why we should deploy single quotes as a default
Its nice to keep comments short, so this can just be reduced to 'because:'


http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@103
PS6, Line 103: It is easy to know if a given string is wrapped by single/double quotes, but
Similarly, in the interest of being concise, this can be removed.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 6
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2018 21:14:02 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3942: Fix wronly escaped string literal in front-end

Posted by "Kim Jin Chul (Code Review)" <ge...@cloudera.org>.
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 )

Change subject: IMPALA-3942: Fix wronly escaped string literal in front-end
......................................................................


Patch Set 6:

(11 comments)

http://gerrit.cloudera.org:8080/#/c/8818/6//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8818/6//COMMIT_MSG@7
PS6, Line 7: wronly
> wrongly
Done


http://gerrit.cloudera.org:8080/#/c/8818/6//COMMIT_MSG@15
PS6, Line 15:  and that's why we just choose one
            : to always normalize to
> remove this, covered by the paragraph below.
Done


http://gerrit.cloudera.org:8080/#/c/8818/6//COMMIT_MSG@19
PS6, Line 19: .
> ... to single quotes
Done


http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java
File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java:

http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@41
PS6, Line 41:  It is not always possible to determine if a string "should" be
            :   // single or double quoted(e.g. concat('a', "b")) and that's why we just choose one
            :   // to always normalize to.
> This second sentence is not necessary, this is explained better in the larg
Done


http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@44
PS6, Line 44:   private final String escaped_single_quoted_value_;
> Let's not store this value and instead compute it on-the-fly.
Done. By the way, would you tell me the reason why you don't want to keep this? You may know I intended to avoid any change of the redundant computation. I think the escaped value is always valid since value_ is immutable.


http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@99
PS6, Line 99: .
            :   // Here is a background why we should deploy single quotes as a default
> Its nice to keep comments short, so this can just be reduced to 'because:'
Done


http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@103
PS6, Line 103: It is easy to know if a given string is wrapped by single/double quotes, but
> Similarly, in the interest of being concise, this can be removed.
Done


http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@110
PS6, Line 110:   private String getEscapedSingleQuotedValue() {
> getNormalizedValue()
Done


http://gerrit.cloudera.org:8080/#/c/8818/6/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@211
PS6, Line 211:         + escaped_single_quoted_value_ + "' to number.");
> let's keep value_ here because that was the original value the user passed
Done


http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py@273
PS6, Line 273:     result = self.execute_query("select 'a\"b'")
> My understanding is that these tests do not cover your new changes. These a
Done


http://gerrit.cloudera.org:8080/#/c/8818/6/tests/query_test/test_queries.py@310
PS6, Line 310:   def test_escaping_string_literal(self, unique_database):
> We should have a more focused test for this in ToSqlTest.java. This fix aff
I agree with your suggestion. ToSqlTest mainly checks the problem. By the way, I think the E2E tests are still meaningful to reproduce the reported issues. I would like we have the tests are in the both parts.



-- 
To view, visit http://gerrit.cloudera.org:8080/8818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec
Gerrit-Change-Number: 8818
Gerrit-PatchSet: 6
Gerrit-Owner: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Kim Jin Chul <ji...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Jan 2018 11:52:04 +0000
Gerrit-HasComments: Yes