Reading parquet strips non-nullability from schema

[Greg Kopff <gr...@q10stats.com>]

Hi.

I’ve spent the last couple of hours trying to chase down an issue with writing/reading parquet files.  I was trying to save (and then read in) a parquet file with a schema that sets my non-nullability details correctly.  After having no success for some time, I posted to Stack Overflow about it:

https://stackoverflow.com/q/72877780/1212960 <https://stackoverflow.com/q/72877780/1212960>

If you read the question, you’ll see that I can trace writing a schema with the correct (non-)nullability information.  And I can even see that when reading in the parquet file, the correct nullability information is found.  However, by the time the data frame gets handed back to me, the (non-)nullability information has been thrown away.

Only after posting the question did I find this sentence in the Parquet Files (https://spark.apache.org/docs/3.3.0/sql-data-sources-parquet.html <https://spark.apache.org/docs/3.3.0/sql-data-sources-parquet.html>) documentation: 

When reading Parquet files, all columns are automatically converted to be nullable for compatibility reasons.

It seems odd to me that:

 - Spark tracks nullable and non-nullable columns
 - Parquet tracks nullable and non-nullable columns
 - Spark can write a parquet file with correctly annotated non-nullable columns
 - Spark can correctly read a parquet file and identify non-nullable columns

… and yet, Spark deliberately discards this information.  

I understand (from this article <https://medium.com/@weshoffman/apache-spark-parquet-and-troublesome-nulls-28712b06f836#:~:text=A%20column%E2%80%99s%20nullable%20characteristic%20is%20a%20contract%20with%20the%20Catalyst%20Optimizer%20that%20null%20data%20will%20not%20be%20produced>) that Spark itself doesn’t enforce the non-nullability - it’s simply information used by the Catalyst optimiser.  And I also understand that Bad Things™ would happen if you inserted null values when the schema says it’s non-null.

But I don’t understand why I can’t accept those caveats and have Spark retain my schema properly, with the non-nullability information maintained.  What are these compatibility reasons that the documentation alludes to?

Is this behaviour configurable?  How can I create a dataframe from a parquet file that uses the schema as-is (i.e. honouring the nullability information)?  I want the Catalyst optimiser to have this information available, and I control my data to ensure that it meets the nullability constraints.

Thanks for your time.

Kindest regards,

—
Greg