You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Willi Raschkowski (Jira)" <ji...@apache.org> on 2023/04/14 13:18:00 UTC

[jira] [Commented] (SPARK-43142) DSL expressions fail on attribute with special characters

    [ https://issues.apache.org/jira/browse/SPARK-43142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712382#comment-17712382 ] 

Willi Raschkowski commented on SPARK-43142:
-------------------------------------------

Here's what's happening: {{ImplicitOperators}} methods like {{asc}} rely on a call to {{expr}} [(Github)|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L149]. 

The {{UnresolvedAttribute}} returned by {{.attr}} is implicitly converted to {{DslAttr}}. But {{DslAttr}} does not implement {{expr}} by returning the attribute it's already wrapping. Instead, it only implements how to convert the attribute it's wrapping to a string name [(Github)|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L273-L275].

Returning an attribute for an implicitly wrapped attribute is implemented on the super class {{ImplicitAttribute}} by creating a new {{UnresolvedAttribute}} on the string name return by {{DslAttr}} (the method call {{s}}, [Github|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L278-L280]).

The problem is that this string name returned by {{DslAttr}} no longer has the quotes and thus the new {{UnresolvedAttribute}} parses an unquoted identifier.

{code}
scala> "`col/slash`".attr.name
res1: String = col/slash
{code}

> DSL expressions fail on attribute with special characters
> ---------------------------------------------------------
>
>                 Key: SPARK-43142
>                 URL: https://issues.apache.org/jira/browse/SPARK-43142
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.4.0
>            Reporter: Willi Raschkowski
>            Priority: Major
>
> Expressions on implicitly converted attributes fail if the attributes have names containing special characters. They fail even if the attributes are backtick-quoted:
> {code:java}
> scala> import org.apache.spark.sql.catalyst.dsl.expressions._
> import org.apache.spark.sql.catalyst.dsl.expressions._
> scala> "`slashed/col`".attr
> res0: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = 'slashed/col
> scala> "`slashed/col`".attr.asc
> org.apache.spark.sql.catalyst.parser.ParseException:
> mismatched input '/' expecting {<EOF>, '.', '-'}(line 1, pos 7)
> == SQL ==
> slashed/col
> -------^^^
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org