[jira] [Created] (SPARK-28151) ByteType is not correctly mapped for read/write of SQLServer tables

["Shiv Prashant Sood (JIRA)" <ji...@apache.org>]

Shiv Prashant Sood created SPARK-28151:
------------------------------------------

             Summary: ByteType is not correctly mapped for read/write of SQLServer tables
                 Key: SPARK-28151
                 URL: https://issues.apache.org/jira/browse/SPARK-28151
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.4.3, 3.0.0
            Reporter: Shiv Prashant Sood


Writing dataframe with column type BYTETYPE fails when using JDBC connector for SQL Server. Append and Read of tables also fail. The problem is due 

1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped to TINYINT
{color:#cc7832}case {color}ByteType => Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832}, {color}java.sql.Types.{color:#9876aa}TINYINT{color}))

In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from the point of view of upcasting, but will lead to 4 byte allocation rather than 1 byte for BYTETYPE.



2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata: Metadata). The function sets the value in RDD row. The value is set per the data type. Here there is no mapping for BYTETYPE and thus results will result in an error when getCatalystType() is fixed.

Note : These issues were found when reading/writing with SQLServer. Will be submitting a PR soon to fix these mappings in MSSQLServerDialect.

Error seen when writing table

(JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable #2: *Cannot find data type BYTE*.)
com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable #2: Cannot find data type BYTE.
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
 ..

 

 

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org