You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Noah Kawasaki (Jira)" <ji...@apache.org> on 2021/01/14 15:21:00 UTC

[jira] [Updated] (SPARK-34102) Spark SQL cannot escape both \ and other special characters

     [ https://issues.apache.org/jira/browse/SPARK-34102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Noah Kawasaki updated SPARK-34102:
----------------------------------
    Labels: escaping filter string-manipulation  (was: )

> Spark SQL cannot escape both \ and other special characters 
> ------------------------------------------------------------
>
>                 Key: SPARK-34102
>                 URL: https://issues.apache.org/jira/browse/SPARK-34102
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.2, 2.1.3, 2.2.2, 2.3.0, 2.4.5, 3.0.1
>            Reporter: Noah Kawasaki
>            Priority: Minor
>              Labels: escaping, filter, string-manipulation
>
> Spark literal string parsing does not properly escape backslashes or other special characters. This is an extension of this issue: https://issues.apache.org/jira/browse/SPARK-17647#
>  
> The issue is that depending on how spark.sql.parser.escapedStringLiterals is set, you will either be able to correctly get escaped backslashes in a string literal, but not escaped other special characters, OR, you can have correctly escaped other special characters, but not correctly escaped backslashes.
> So you have to choose which configuration you care about more.
> I have tested Spark versions 2.1, 2.2, 2.3, 2.4, and 3.0 and they all experience the issue:
> {code:java}
> # These do not return the expected backslash
> SET spark.sql.parser.escapedStringLiterals=false;
> SELECT '\\';
> > \
> (should return \\)
> SELECT 'hi\hi';
> > hihi
> (should return hi\hi) 
> # These are correctly escaped
> SELECT '\"';
> > "
>  SELECT '\'';
> > '{code}
> If I switch this: 
> {code:java}
> # These now work
> SET spark.sql.parser.escapedStringLiterals=true;
> SELECT '\\';
> > \\
> SELECT 'hi\hi';
> > hi\hi
> # These are now not correctly escaped
> SELECT '\"';
> > \"
> (should return ")
> SELECT '\'';
> > \'
> (should return ' ){code}
>  So basically we have to choose:
> SET spark.sql.parser.escapedStringLiterals=false; if we want backslashes correctly escaped but not other special characters
> SET spark.sql.parser.escapedStringLiterals=true; if we want other special characters correctly escaped but not backslashes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org