You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mykhailo Osypov (JIRA)" <ji...@apache.org> on 2016/12/14 10:33:59 UTC
[jira] [Created] (SPARK-18859) Catalyst codegen does not mark
column as nullable when it should. Causes NPE
Mykhailo Osypov created SPARK-18859:
---------------------------------------
Summary: Catalyst codegen does not mark column as nullable when it should. Causes NPE
Key: SPARK-18859
URL: https://issues.apache.org/jira/browse/SPARK-18859
Project: Spark
Issue Type: Bug
Components: Optimizer, SQL
Affects Versions: 2.0.2
Reporter: Mykhailo Osypov
Priority: Critical
When joining two tables via LEFT JOIN, columns in right table may be NULLs, however catalyst codegen cannot recognize it.
Example:
```
create table masterdata.testtable(
id int not null,
age int
);
create table masterdata.jointable(
id int not null,
name text not null
);
(select t.id, t.age, j.name from masterdata.testtable t left join masterdata.jointable j on t.id = j.id) as testtable;
```
```
val df = sqlContext
.read
.format("jdbc")
..option("dbTable", "query to select)
....
.load
//df generated schema
/*
root
|-- id: integer (nullable = false)
|-- age: integer (nullable = true)
|-- name: string (nullable = false)
*/
```
CODEGEN
```
/* 038 */ scan_rowWriter.write(0, scan_value);
/* 039 */
/* 040 */ if (scan_isNull1) {
/* 041 */ scan_rowWriter.setNullAt(1);
/* 042 */ } else {
/* 043 */ scan_rowWriter.write(1, scan_value1);
/* 044 */ }
/* 045 */
/* 046 */ scan_rowWriter.write(2, scan_value2);
```
Since j.name is from right table of left join query, it may be null. However generated schema doesn't think so (probably because it defined as name text not null)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org