[jira] [Created] (SPARK-25709) Can't append to a data source V2 unless target already has schema information or fixed schema matching dataframe

["Assaf Mendelson (JIRA)" <ji...@apache.org>]

Assaf Mendelson created SPARK-25709:
---------------------------------------

             Summary: Can't append to a data source V2 unless target already has schema information or fixed schema matching dataframe
                 Key: SPARK-25709
                 URL: https://issues.apache.org/jira/browse/SPARK-25709
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.4.0
            Reporter: Assaf Mendelson


Let's assume we have a writer data source which writes to a dynamic schema (e.g. writing to a database table). If we have mode append but the table is missing then writing will fail.

 

The reason for this is that in the dataframeWriter save method we have val relation = DataSourceV2Relation.create(source, options) which in turn contains: val reader = source.createReader(options, userSpecifiedSchema)

The problem is that this sets the schema from the reader portion of the data source. If the data source can't infer the schema then we will fail to write (even thought the schema is actually part of the dataframe).

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org