You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by jeff saremi <je...@hotmail.com> on 2017/09/12 22:38:00 UTC

Multiple Sources found for csv

I have this line which works in the spark interactive console but it fails in Intellij

Using Spark 2.1.1 in both cases:

Exception in thread "main" java.lang.RuntimeException: Multiple sources found for csv (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat, com.databricks.spark.csv.DefaultSource15), please specify the fully qualified class name.



source:

val ie8df = sqlContext.read.schema(SomeSchema).option("mode","dropmalformed").option("sep", "\t").format("csv").load(somepath)


Re: Multiple Sources found for csv

Posted by jeff saremi <je...@hotmail.com>.
sorry just found this which answers my question:


https://stackoverflow.com/questions/41726340/spark-2-0-csv-error

[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded]<https://stackoverflow.com/questions/41726340/spark-2-0-csv-error>

Spark 2.0 CSV Error<https://stackoverflow.com/questions/41726340/spark-2-0-csv-error>
stackoverflow.com
I am upgrading to spark 2 from 1.6 and am having an issue reading in CSV files. In spark 1.6 I would have something like this to read in a CSV file. val df = sqlContext.read.format("com.databricks...



________________________________
From: jeff saremi <je...@hotmail.com>
Sent: Tuesday, September 12, 2017 3:38:00 PM
To: user@spark.apache.org
Subject: Multiple Sources found for csv


I have this line which works in the spark interactive console but it fails in Intellij

Using Spark 2.1.1 in both cases:

Exception in thread "main" java.lang.RuntimeException: Multiple sources found for csv (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat, com.databricks.spark.csv.DefaultSource15), please specify the fully qualified class name.



source:

val ie8df = sqlContext.read.schema(SomeSchema).option("mode","dropmalformed").option("sep", "\t").format("csv").load(somepath)