You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "L. C. Hsieh (Jira)" <ji...@apache.org> on 2021/05/15 00:00:00 UTC
[jira] [Updated] (SPARK-34750) Parquet with invalid chars on column
name reads double as null when a clean schema is applied
[ https://issues.apache.org/jira/browse/SPARK-34750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
L. C. Hsieh updated SPARK-34750:
--------------------------------
Fix Version/s: (was: 2.4.8)
> Parquet with invalid chars on column name reads double as null when a clean schema is applied
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-34750
> URL: https://issues.apache.org/jira/browse/SPARK-34750
> Project: Spark
> Issue Type: Bug
> Components: Input/Output
> Affects Versions: 2.4.3
> Environment: Pyspark 2.4.3
> AWS Glue Dev Endpoint EMR
> Reporter: Nivas Umapathy
> Priority: Major
>
> I have a parquet file that has data with invalid column names on it. [#Reference](https://issues.apache.org/jira/browse/SPARK-27442) Here is the file [Invalid Header Parquet|https://drive.google.com/file/d/101WNWXnPwhjocSMVjkhn5jo85Ri_NydP/view?usp=sharing].
> I tried to load this file with
> {{df = glue_context.read.parquet('invalid_columns_double.parquet')}}
> {{df = df.withColumnRenamed('COL 1', 'COL_1')}}
> {{df = df.withColumnRenamed('COL,2', 'COL_2')}}
> {{df = df.withColumnRenamed('COL;3', 'COL_3') }}
> and so on.
> Now if i call
> {{df.show()}}
> it throws this exception that is still pointing to the old column name.
> {{pyspark.sql.utils.AnalysisException: 'Attribute name "COL 1" contains invalid character(s) among " ,;{}()\\n\\t=". Please use alias to rename it.;'}}
>
> When i read about it in some blogs, there was suggestion to re-read the same parquet with new schema applied. So i did
> {{df = glue_context.read.schema(df.schema).parquet(}}{{'invalid_columns_double.parquet')}}{{}}
>
> and it works, but all the data in the dataframe are null. The same works for Strings
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org