[jira] [Resolved] (SPARK-13167) JDBC data source does not include null value partition columns rows in the result.

["Reynold Xin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-13167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Reynold Xin resolved SPARK-13167.
---------------------------------
       Resolution: Fixed
         Assignee: Suresh Thalamati
    Fix Version/s: 2.0.0

> JDBC data source does not include null value partition columns rows in the result.
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-13167
>                 URL: https://issues.apache.org/jira/browse/SPARK-13167
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.0, 2.0.0
>            Reporter: Suresh Thalamati
>            Assignee: Suresh Thalamati
>             Fix For: 2.0.0
>
>
> Reading from am JDBC data source using a partition column that is nullable can return incorrect number of rows, if there are rows with null value for partition column.
> {code}
> val emp = sqlContext.read.jdbc("jdbc:h2:mem:testdb0;user=testUser;password=testPass", "TEST.EMP", "theid", 0, 4, 3, new Properties)
> emp.count()
> {code}
> Above jdbc read call sets up the partitions of the following form. It does not include null predicate.
> {code}
> JDBCPartition(THEID < 1,0),JDBCPartition(THEID >= 1 AND THEID < 2,1),JDBCPartition(THEID >= 2,2)
> {code}
> Rows with null values in partition column are not included in the results because the partition predicate does not specify is null predicates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org