[jira] [Updated] (SPARK-18859) Catalyst codegen does not mark column as nullable when it should. Causes NPE

["Mykhailo Osypov (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mykhailo Osypov updated SPARK-18859:
------------------------------------
    Description: 
When joining two tables via LEFT JOIN, columns in right table may be NULLs, however catalyst codegen cannot recognize it.
Example:
{code:title=schema.sql|borderStyle=solid}
create table masterdata.testtable(
  id int not null,
  age int
);
create table masterdata.jointable(
  id int not null,
  name text not null
);
(select t.id, t.age, j.name from masterdata.testtable t left join masterdata.jointable j on t.id = j.id) as testtable;
{code}

{code:title=master code|borderStyle=solid}
val df = sqlContext
      .read
      .format("jdbc")
      ..option("dbTable", "query to select)
     ....
      .load
//df generated schema
/*
root
 |-- id: integer (nullable = false)
 |-- age: integer (nullable = true)
 |-- name: string (nullable = false)
*/
{code}
{code:title=Codegen|borderStyle=solid}
/* 038 */       scan_rowWriter.write(0, scan_value);
/* 039 */
/* 040 */       if (scan_isNull1) {
/* 041 */         scan_rowWriter.setNullAt(1);
/* 042 */       } else {
/* 043 */         scan_rowWriter.write(1, scan_value1);
/* 044 */       }
/* 045 */
/* 046 */       scan_rowWriter.write(2, scan_value2);
{code}

Since j.name is from right table of left join query, it may be null. However generated schema doesn't think so (probably because it defined as name text not null)

  was:
When joining two tables via LEFT JOIN, columns in right table may be NULLs, however catalyst codegen cannot recognize it.
Example:
```
create table masterdata.testtable(
  id int not null,
  age int
);
create table masterdata.jointable(
  id int not null,
  name text not null
);

(select t.id, t.age, j.name from masterdata.testtable t left join masterdata.jointable j on t.id = j.id) as testtable;
```
```
val df = sqlContext
      .read
      .format("jdbc")
      ..option("dbTable", "query to select)
     ....
      .load
//df generated schema
/*
root
 |-- id: integer (nullable = false)
 |-- age: integer (nullable = true)
 |-- name: string (nullable = false)
*/
```
CODEGEN
```
/* 038 */       scan_rowWriter.write(0, scan_value);
/* 039 */
/* 040 */       if (scan_isNull1) {
/* 041 */         scan_rowWriter.setNullAt(1);
/* 042 */       } else {
/* 043 */         scan_rowWriter.write(1, scan_value1);
/* 044 */       }
/* 045 */
/* 046 */       scan_rowWriter.write(2, scan_value2);
```

Since j.name is from right table of left join query, it may be null. However generated schema doesn't think so (probably because it defined as name text not null)


> Catalyst codegen does not mark column as nullable when it should. Causes NPE
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-18859
>                 URL: https://issues.apache.org/jira/browse/SPARK-18859
>             Project: Spark
>          Issue Type: Bug
>          Components: Optimizer, SQL
>    Affects Versions: 2.0.2
>            Reporter: Mykhailo Osypov
>            Priority: Critical
>
> When joining two tables via LEFT JOIN, columns in right table may be NULLs, however catalyst codegen cannot recognize it.
> Example:
> {code:title=schema.sql|borderStyle=solid}
> create table masterdata.testtable(
>   id int not null,
>   age int
> );
> create table masterdata.jointable(
>   id int not null,
>   name text not null
> );
> (select t.id, t.age, j.name from masterdata.testtable t left join masterdata.jointable j on t.id = j.id) as testtable;
> {code}
> {code:title=master code|borderStyle=solid}
> val df = sqlContext
>       .read
>       .format("jdbc")
>       ..option("dbTable", "query to select)
>      ....
>       .load
> //df generated schema
> /*
> root
>  |-- id: integer (nullable = false)
>  |-- age: integer (nullable = true)
>  |-- name: string (nullable = false)
> */
> {code}
> {code:title=Codegen|borderStyle=solid}
> /* 038 */       scan_rowWriter.write(0, scan_value);
> /* 039 */
> /* 040 */       if (scan_isNull1) {
> /* 041 */         scan_rowWriter.setNullAt(1);
> /* 042 */       } else {
> /* 043 */         scan_rowWriter.write(1, scan_value1);
> /* 044 */       }
> /* 045 */
> /* 046 */       scan_rowWriter.write(2, scan_value2);
> {code}
> Since j.name is from right table of left join query, it may be null. However generated schema doesn't think so (probably because it defined as name text not null)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org