[jira] [Comment Edited] (PHOENIX-6667) Spark3 connector requires that all columns are specified when writing

["Istvan Toth (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-6667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603969#comment-17603969 ] 

Istvan Toth edited comment on PHOENIX-6667 at 9/14/22 8:54 AM:
---------------------------------------------------------------

-In theory, we could process the queries in the connector, and andd NULLs for the non-specified columns automatically, but this would be a performance hit, and is not really necessary, as Phoenix doesn't have a problem with missing nullable columns, it's just a new check added to Spark3 that is causing the problem.-

 


was (Author: stoty):
In theory, we could process the queries in the connector, and andd NULLs for the non-specified columns automatically, but this would be a performance hit, and is not really necessary, as Phoenix doesn't have a problem with missing nullable columns, it's just a new check added to Spark3 that is causing the problem.



 

> Spark3 connector requires that all columns are specified when writing
> ---------------------------------------------------------------------
>
>                 Key: PHOENIX-6667
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-6667
>             Project: Phoenix
>          Issue Type: Bug
>          Components: connectors, spark-connector
>    Affects Versions: connectors-6.0.0
>            Reporter: Istvan Toth
>            Priority: Major
>
> For Spark 2, it was possible to omit some columns from the dataframe, the same way it is not mandatory to specify all columns when upserting via SQL.
> Spark3 has added new checks, which require that EVERY sql column is specifed in the DataFrame.
> Consequently, when using the current API, writing will fail unless you specify all columns.
> This is a loss of functionality WRT Phoenix (and other SQL datastores) compared to Spark2.
> I don't think that we can do anything from the Phoenix side, just documenting the regression here.
> Maybe future Spark versions will make this configurable.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)