[jira] [Created] (SPARK-18558) spark-csv: infer data type for mixed integer/null columns causes exception

["Peter Rose (JIRA)" <ji...@apache.org>]

Peter Rose created SPARK-18558:
----------------------------------

             Summary: spark-csv: infer data type for mixed integer/null columns causes exception
                 Key: SPARK-18558
                 URL: https://issues.apache.org/jira/browse/SPARK-18558
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.0.2
            Reporter: Peter Rose


Null pointer exception when using the following csv file:

example.csv:
column1
"1"
"2"
""

 Dataset<Row> df = spark
	    		.read()
	    		.option("header", "true")
	    		.option("inferSchema", "true")
	    	        .format("csv")
	    		.load(example.csv);

 df.printSchema();

The type is correctly inferred:

root
 |-- col1: integer (nullable = true)

df.show(5);

The show method leads to this exception:

java.lang.NumberFormatException: null
	at java.lang.Integer.parseInt(Integer.java:542) ~[?:1.8.0_25]
	at java.lang.Integer.parseInt(Integer.java:615) ~[?:1.8.0_25]
	at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272) ~[scala-library-2.11.8.jar:?]
	at scala.collection.immutable.StringOps.toInt(StringOps.scala:29) ~[scala-library-2.11.8.jar:?]
	at org.apache.spark.sql.execution.datasources.csv.CSVTypeCast$.castTo(CSVInferSchema.scala:241) ~[spark-sql_2.11-2.0.2.jar:2.0.2]






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org