You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jayadevan M (JIRA)" <ji...@apache.org> on 2016/11/28 10:05:58 UTC
[jira] [Commented] (SPARK-18558) spark-csv: infer data type for
mixed integer/null columns causes exception
[ https://issues.apache.org/jira/browse/SPARK-18558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15701508#comment-15701508 ]
Jayadevan M commented on SPARK-18558:
-------------------------------------
[~PeterRose]
I tried to replicate this issue, But did not get null pointer exception
spark.read.option("header", "true").option("inferSchema", "true").format("csv").load("example.csv").show(5);
+-------+
|column1|
+-------+
| 1|
| 2|
| null|
+-------+
> spark-csv: infer data type for mixed integer/null columns causes exception
> --------------------------------------------------------------------------
>
> Key: SPARK-18558
> URL: https://issues.apache.org/jira/browse/SPARK-18558
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.0.2
> Reporter: Peter Rose
>
> Null pointer exception when using the following csv file:
> example.csv:
> column1
> "1"
> "2"
> ""
> Dataset<Row> df = spark
> .read()
> .option("header", "true")
> .option("inferSchema", "true")
> .format("csv")
> .load(example.csv);
> df.printSchema();
> The type is correctly inferred:
> root
> |-- col1: integer (nullable = true)
> df.show(5);
> The show method leads to this exception:
> java.lang.NumberFormatException: null
> at java.lang.Integer.parseInt(Integer.java:542) ~[?:1.8.0_25]
> at java.lang.Integer.parseInt(Integer.java:615) ~[?:1.8.0_25]
> at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272) ~[scala-library-2.11.8.jar:?]
> at scala.collection.immutable.StringOps.toInt(StringOps.scala:29) ~[scala-library-2.11.8.jar:?]
> at org.apache.spark.sql.execution.datasources.csv.CSVTypeCast$.castTo(CSVInferSchema.scala:241) ~[spark-sql_2.11-2.0.2.jar:2.0.2]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org