You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/09/19 19:00:00 UTC
[jira] [Commented] (SPARK-36801) Document change for Spark sql jdbc
[ https://issues.apache.org/jira/browse/SPARK-36801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417382#comment-17417382 ]
Apache Spark commented on SPARK-36801:
--------------------------------------
User 'senthh' has created a pull request for this issue:
https://github.com/apache/spark/pull/34042
> Document change for Spark sql jdbc
> ----------------------------------
>
> Key: SPARK-36801
> URL: https://issues.apache.org/jira/browse/SPARK-36801
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
> Affects Versions: 3.0.0, 3.0.2, 3.0.3, 3.1.0, 3.1.1, 3.1.2
> Reporter: Senthil Kumar
> Priority: Trivial
>
> Reading using Spark SQL jdbc DataSource does not maintain nullable type and changes "non nullable" columns to "nullable".
>
> For example:
> mysql> CREATE TABLE Persons(Id int NOT NULL, FirstName varchar(255), LastName varchar(255), Age int);
> Query OK, 0 rows affected (0.04 sec)
> mysql> show tables;
> +-------------------+
> | Tables_in_test_db |
> +-------------------+
> | Persons |
> +-------------------+
> 1 row in set (0.00 sec)
> mysql> desc Persons;
> +-----------+--------------+------+-----+---------+-------+
> | Field | Type | Null | Key | Default | Extra |
> +-----------+--------------+------+-----+---------+-------+
> | Id | int | NO | | NULL | |
> | FirstName | varchar(255) | YES | | NULL | |
> | LastName | varchar(255) | YES | | NULL | |
> | Age | int | YES | | NULL | |
> +-----------+--------------+------+-----+---------+-------+
>
>
> {color:#cc7832}val {color}df = spark.read.format({color:#6a8759}"jdbc"{color})
> .option({color:#6a8759}"database"{color}{color:#cc7832},{color}{color:#6a8759}"Test_DB"{color})
> .option({color:#6a8759}"user"{color}{color:#cc7832}, {color}{color:#6a8759}"root"{color})
> .option({color:#6a8759}"password"{color}{color:#cc7832}, {color}{color:#6a8759}""{color})
> .option({color:#6a8759}"driver"{color}{color:#cc7832}, {color}{color:#6a8759}"com.mysql.cj.jdbc.Driver"{color})
> .option({color:#6a8759}"url"{color}{color:#cc7832}, {color}{color:#6a8759}"jdbc:mysql://localhost:3306/Test_DB"{color})
> .option({color:#6a8759}"query"{color}{color:#cc7832}, {color}{color:#6a8759}"(select * from Persons)"{color})
> .load()
> df.printSchema()
>
> *output:*
>
> root
> |-- Id: integer (nullable = true)
> |-- FirstName: string (nullable = true)
> |-- LastName: string (nullable = true)
> |-- Age: integer (nullable = true)
>
>
> So we need to add a note, in Documentation[1], "All columns are automatically converted to be nullable for compatibility reasons."
> Ref:
> [1 ][https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html#jdbc-to-other-databases]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org