spark.read.schema return null for dataframe column values

[Mohamed Samir <mo...@gmail.com>]

Hi,

I have small question and issue which I hope spark gurus to help me in

I have parquet file person.parquet that has multiple column with one row.
one of the column "Middle Name" has space which cause issue with spark when
writing it to parquet format
[image: image.png]

what i have done is to renaming the column to remove the space as below
SourceData = SourceData.withColumnRenamed("Middle Name","MiddleName")

if i tried to write SourceData to parquet file, it still returns error
Caused by: org.apache.spark.sql.AnalysisException: Attribute name "Middle
Name" contains invalid character(s) among " ,;{}()\n\t=". Please use alias
to rename it.

so i use below which solve the issue but unfortunately the file generated
has null value for column MiddleName
SourceData = spark.read.schema(SourceData.schema).parquet(TestingPath)

[image: image.png]

Any suggestion on how to solve this issue?