You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Bjørn Jørgensen (Jira)" <ji...@apache.org> on 2022/12/05 19:37:00 UTC
[jira] [Comment Edited] (SPARK-18502) Spark does not handle columns that contain backquote (`)
[ https://issues.apache.org/jira/browse/SPARK-18502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643524#comment-17643524 ]
Bjørn Jørgensen edited comment on SPARK-18502 at 12/5/22 7:36 PM:
------------------------------------------------------------------
I just answered this problem in user@spark.org
df = spark.createDataFrame(
[("china", "asia"), ("colombia", "south america`")],
["country", "continent`"]
)
df.show()
{code:java}
// +--------+--------------+
| country| continent`|
+--------+--------------+
| china| asia|
|colombia|south america`|
+--------+--------------+ {code}
df.select("continent`").show(1)
(...)
AnalysisException: Syntax error in attribute name: continent`.
clean_df = df.toDF(*(c.replace('`', '_') for c in df.columns))
clean_df.show()
{code:java}
// +--------+--------------+
| country| continent_|
+--------+--------------+
| china| asia|
|colombia|south america`|
+--------+--------------+ {code}
clean_df.select("continent_").show(2)
{code:java}
// +--------------+
| continent_|
+--------------+
| asia|
|south america`|
+--------------+ {code}
Examples are from [MungingData Avoiding Dots / Periods in PySpark Column Names|https://mungingdata.com/pyspark/avoid-dots-periods-column-names/]
was (Author: bjornjorgensen):
I just answered this problem in user@spark.org
df = spark.createDataFrame(
[("china", "asia"), ("colombia", "south america`")],
["country", "continent`"]
)
df.show()
+--------+--------------+ | country| continent`| +--------+--------------+ | china| asia| |colombia|south america`| +--------+--------------+
df.select("continent`").show(1)
(...)AnalysisException: Syntax error in attribute name: continent`.
clean_df = df.toDF(*(c.replace('`', '_') for c in df.columns))
clean_df.show()
+--------+--------------+ | country| continent_| +--------+--------------+ | china| asia| |colombia|south america`| +--------+--------------+
clean_df.select("continent_").show(2)
+--------------+ | continent_| +--------------+ | asia| |south america`| +--------------+
Examples are from [MungingData Avoiding Dots / Periods in PySpark Column Names|https://mungingdata.com/pyspark/avoid-dots-periods-column-names/]
> Spark does not handle columns that contain backquote (`)
> --------------------------------------------------------
>
> Key: SPARK-18502
> URL: https://issues.apache.org/jira/browse/SPARK-18502
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Barry Becker
> Priority: Minor
> Labels: bulk-closed
>
> I know that if a column contains dots or hyphens we can put backquotes/backticks around it, but what if the column contains a backtick (`)? Can the back tick be escaped by some means?
> Here is an example of the sort of error I see
> {code}
> org.apache.spark.sql.AnalysisException: syntax error in attribute name: `Invoice`Date`;org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute$.e$1(unresolved.scala:99) org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute$.parseAttributeName(unresolved.scala:109) org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute$.quotedString(unresolved.scala:90) org.apache.spark.sql.Column.(Column.scala:113) org.apache.spark.sql.Column$.apply(Column.scala:36) org.apache.spark.sql.functions$.min(functions.scala:407) com.mineset.spark.vizagg.vizbin.strategies.DateBinStrategy.getDateExtent(DateBinStrategy.scala:158)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org