[GitHub] [incubator-iceberg] waterlx edited a comment on issue #457: DataFrame generated by Seq() might have schema conflict with Iceberg

[GitBox <gi...@apache.org>]

waterlx edited a comment on issue #457: DataFrame generated by Seq() might have schema conflict with Iceberg
URL: https://github.com/apache/incubator-iceberg/issues/457#issuecomment-528725863
 
 
   String in Spark has default nullable setting as "true", and Iceberg has the following code to decide required or optional with respect to nullable()
   ``` java
   // public Type struct(StructType struct, List<Type> types) in SparkTypeToType.java
   if (field.nullable()) {
           newFields.add(Types.NestedField.optional(id, field.name(), type, doc));
   } else {
           newFields.add(Types.NestedField.required(id, field.name(), type, doc));
   }
   ```
   When in Iceberg, we add a " required" field as StringType, the error message is reported.
   This issue could be resolved by explicitly specifying "nullable" for schema in Spark, like:
   ``` scala
   
   val schema = StructType(List(
       StructField("integer_column", IntegerType, nullable = false),
       StructField("string_column", StringType, nullable = true),
       StructField("date_column", DateType, nullable = true)
   ```
   But I have no idea on how to resolve it when DataFrame is generated by Seq() or loading from a file

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org