[jira] [Resolved] (SPARK-32618) ORC writer doesn't support colon in column names

["Sean R. Owen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-32618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean R. Owen resolved SPARK-32618.
----------------------------------
    Resolution: Invalid

Maybe, this is likely a duplicate. You'd have to test, and at least on 2.4.6 if not 3.0, not vs 2.2.x

> ORC writer doesn't support colon in column names
> ------------------------------------------------
>
>                 Key: SPARK-32618
>                 URL: https://issues.apache.org/jira/browse/SPARK-32618
>             Project: Spark
>          Issue Type: Bug
>          Components: Input/Output
>    Affects Versions: 2.3.0
>            Reporter: Pierre Gramme
>            Priority: Major
>
> Hi,
> I'm getting an {{IllegalArgumentException: Can't parse category at 'struct<a:b^:int>'}} when exporting to ORC a dataframe whose column names contain colon ({{:}}). Reproducible as hereunder. Same problem also occurs if the name with colon appears nested as member of a struct.
> Seems related with SPARK-21791(which was solved in 2.3.0).
> In my real-life case, the column was actually {{xsi:type}}, coming from some parsed xml. Thus other users may be affected too.
> Has it been fixed after Spark 2.3.0? (sorry, can't test easily)
> Any workaround? Would be acceptable for me to find and replace all colons with underscore in column names, but not easy to do in a big set of nested struct columns...
> Thanks
>  
>  
> {code:java}
>  spark.conf.set("spark.sql.orc.impl", "native")
>  val dfColon = Seq(1).toDF("a:b")
>  dfColon.printSchema()
>  dfColon.show()
>  dfColon.write.orc("test_colon")
>  // Fails with IllegalArgumentException: Can't parse category at 'struct<a:b^:int>'
>  
>  import org.apache.spark.sql.functions.struct
>  val dfColonStruct = dfColon.withColumn("x", struct($"a:b")).drop("a:b")
>  dfColonStruct.printSchema()
>  dfColonStruct.show()
>  dfColon.write.orc("test_colon_struct")
>  // Fails with IllegalArgumentException: Can't parse category at 'struct<x:struct<a:b^:int>>'
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org