You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2017/02/17 18:35:13 UTC
[GitHub] spark pull request #16976: [WIP][SPARK-19610][SQL] Support parsing multiline...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/16976
[WIP][SPARK-19610][SQL] Support parsing multiline CSV files
## What changes were proposed in this pull request?
This PR proposes the support for multiple lines for CSV.
## How was this patch tested?
Unit tests in `CSVSuite` and `tests.py`
Manual tests with a single 9GB CSV file in local file system as below:
```scala
spark.read.option("wholeFile", true).option("inferSchema", true).csv("tmp.csv").count()
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-19610
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16976.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16976
----
commit 3a303c7ac34163291aacd684546dbb30fd5dee8b
Author: hyukjinkwon <gu...@gmail.com>
Date: 2017-02-17T15:52:30Z
Support parsing multiline json files
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73456 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73456/testReport)** for PR 16976 at commit [`6187f6c`](https://github.com/apache/spark/commit/6187f6c0617fa0aec75d1eb638a00c6e7a350d61).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [WIP][SPARK-19610][SQL] Support parsing multiline CSV fi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73063 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73063/testReport)** for PR 16976 at commit [`8275d12`](https://github.com/apache/spark/commit/8275d123742bc0b339fd83c28ce1e16c75960a73).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class CSVDataSource extends Serializable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102649816
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
--- End diff --
I assume this is exactly same with https://github.com/apache/spark/pull/16976/files#diff-56fbd53c6ada276cb4930affe3720be3L77
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102658272
--- Diff: python/test_support/sql/ages_newlines.csv ---
@@ -0,0 +1,6 @@
+Joe,20,"Hi,
+I am Jeo"
+Tom,30,"My name is Tom"
+Hyukjin,25,"I am Hyukjin
--- End diff --
I was.. in the last year :).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [WIP][SPARK-19610][SQL] Support parsing multiline CSV fi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73062/testReport)** for PR 16976 at commit [`3a303c7`](https://github.com/apache/spark/commit/3a303c7ac34163291aacd684546dbb30fd5dee8b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102658245
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
--- End diff --
Yup, just copied and pasted. Just checked that they are same line by line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120034049
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -961,56 +968,121 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
test("SPARK-18699 put malformed records in a `columnNameOfCorruptRecord` field") {
- val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
- val df1 = spark
- .read
- .option("mode", "PERMISSIVE")
- .schema(schema)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df1,
- Row(null, null) ::
- Row(1, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
- val columnNameOfCorruptRecord = "_unparsed"
- val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
- val df2 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField1)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df2,
- Row(null, null, "0,2013-111-11 12:13:14") ::
- Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
- Nil)
-
- // We put a `columnNameOfCorruptRecord` field in the middle of a schema
- val schemaWithCorrField2 = new StructType()
- .add("a", IntegerType)
- .add(columnNameOfCorruptRecord, StringType)
- .add("b", TimestampType)
- val df3 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField2)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df3,
- Row(null, "0,2013-111-11 12:13:14", null) ::
- Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- val errMsg = intercept[AnalysisException] {
- spark
+ Seq(false, true).foreach { wholeFile =>
+ val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
+ val df1 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("wholeFile", wholeFile)
+ .schema(schema)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df1,
+ Row(null, null) ::
+ Row(1, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
+ val columnNameOfCorruptRecord = "_unparsed"
+ val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
+ val df2 = spark
.read
.option("mode", "PERMISSIVE")
.option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField1)
.csv(testFile(valueMalformedFile))
- .collect
- }.getMessage
- assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ checkAnswer(df2,
+ Row(null, null, "0,2013-111-11 12:13:14") ::
+ Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
+ Nil)
+
+ // We put a `columnNameOfCorruptRecord` field in the middle of a schema
+ val schemaWithCorrField2 = new StructType()
+ .add("a", IntegerType)
+ .add(columnNameOfCorruptRecord, StringType)
+ .add("b", TimestampType)
+ val df3 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField2)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df3,
+ Row(null, "0,2013-111-11 12:13:14", null) ::
+ Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ val errMsg = intercept[AnalysisException] {
+ spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .csv(testFile(valueMalformedFile))
+ .collect
+ }.getMessage
+ assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ }
+ }
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
--- End diff --
Ah, sorry, I got you. So, you mean what if other files still have the headers? It will skip. Univocity parser handles this (did I understood correctly?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102685769
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val firstRow = csv.mapPartitions { iter =>
+ // We need this header line always because even if `inferSchema` option is disabled,
+ // it needs to know the number of columns that are string types without schema inference.
+ iter.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ parsedOptions.asParserSettings)
+ }
+ }.first()
--- End diff --
Filed in https://issues.apache.org/jira/browse/SPARK-19709
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73422/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73454/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103015384
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -961,56 +978,135 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
test("SPARK-18699 put malformed records in a `columnNameOfCorruptRecord` field") {
- val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
- val df1 = spark
- .read
- .option("mode", "PERMISSIVE")
- .schema(schema)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df1,
- Row(null, null) ::
- Row(1, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
- val columnNameOfCorruptRecord = "_unparsed"
- val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
- val df2 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField1)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df2,
- Row(null, null, "0,2013-111-11 12:13:14") ::
- Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
- Nil)
-
- // We put a `columnNameOfCorruptRecord` field in the middle of a schema
- val schemaWithCorrField2 = new StructType()
- .add("a", IntegerType)
- .add(columnNameOfCorruptRecord, StringType)
- .add("b", TimestampType)
- val df3 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField2)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df3,
- Row(null, "0,2013-111-11 12:13:14", null) ::
- Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- val errMsg = intercept[AnalysisException] {
- spark
+ Seq(false, true).foreach { bool =>
--- End diff --
nit: `wholeFile` instead of `bool`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103067531
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -243,12 +244,15 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
test("test for DROPMALFORMED parsing mode") {
- val cars = spark.read
- .format("csv")
- .options(Map("header" -> "true", "mode" -> "dropmalformed"))
- .load(testFile(carsFile))
+ Seq(false, true).foreach { wholeFile =>
--- End diff --
Here also simply I added..
```scala
Seq(false, true).foreach { wholeFile =>
...
.option("wholeFile", wholeFile)
...
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by NathanHowell <gi...@git.apache.org>.
Github user NathanHowell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102663016
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala ---
@@ -43,23 +37,26 @@ class CSVFileFormat extends TextBasedFileFormat with DataSourceRegister {
override def shortName(): String = "csv"
- override def toString: String = "CSV"
-
- override def hashCode(): Int = getClass.hashCode()
-
- override def equals(other: Any): Boolean = other.isInstanceOf[CSVFileFormat]
+ override def isSplitable(
+ sparkSession: SparkSession,
+ options: Map[String, String],
+ path: Path): Boolean = {
+ val parsedOptions =
+ new CSVOptions(options, sparkSession.sessionState.conf.sessionLocalTimeZone)
+ val csvDataSource = CSVDataSource(parsedOptions)
+ csvDataSource.isSplitable && super.isSplitable(sparkSession, options, path)
+ }
override def inferSchema(
sparkSession: SparkSession,
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
require(files.nonEmpty, "Cannot infer schema from an empty set of files")
--- End diff --
Should this return `None` instead of throwing an exception? The JSON parser does, though I'm not sure which approach is recommended. They should be consistent though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73341/testReport)** for PR 16976 at commit [`8474933`](https://github.com/apache/spark/commit/84749338ec59e9c27dec3e1bbe8a35d1e5fd3fe7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73334 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73334/testReport)** for PR 16976 at commit [`4dc48e1`](https://github.com/apache/spark/commit/4dc48e19266be8dc39c89936e0957c768c112d8a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73336/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120033528
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -961,56 +968,121 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
test("SPARK-18699 put malformed records in a `columnNameOfCorruptRecord` field") {
- val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
- val df1 = spark
- .read
- .option("mode", "PERMISSIVE")
- .schema(schema)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df1,
- Row(null, null) ::
- Row(1, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
- val columnNameOfCorruptRecord = "_unparsed"
- val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
- val df2 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField1)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df2,
- Row(null, null, "0,2013-111-11 12:13:14") ::
- Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
- Nil)
-
- // We put a `columnNameOfCorruptRecord` field in the middle of a schema
- val schemaWithCorrField2 = new StructType()
- .add("a", IntegerType)
- .add(columnNameOfCorruptRecord, StringType)
- .add("b", TimestampType)
- val df3 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField2)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df3,
- Row(null, "0,2013-111-11 12:13:14", null) ::
- Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- val errMsg = intercept[AnalysisException] {
- spark
+ Seq(false, true).foreach { wholeFile =>
+ val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
+ val df1 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("wholeFile", wholeFile)
+ .schema(schema)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df1,
+ Row(null, null) ::
+ Row(1, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
+ val columnNameOfCorruptRecord = "_unparsed"
+ val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
+ val df2 = spark
.read
.option("mode", "PERMISSIVE")
.option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField1)
.csv(testFile(valueMalformedFile))
- .collect
- }.getMessage
- assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ checkAnswer(df2,
+ Row(null, null, "0,2013-111-11 12:13:14") ::
+ Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
+ Nil)
+
+ // We put a `columnNameOfCorruptRecord` field in the middle of a schema
+ val schemaWithCorrField2 = new StructType()
+ .add("a", IntegerType)
+ .add(columnNameOfCorruptRecord, StringType)
+ .add("b", TimestampType)
+ val df3 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField2)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df3,
+ Row(null, "0,2013-111-11 12:13:14", null) ::
+ Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ val errMsg = intercept[AnalysisException] {
+ spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .csv(testFile(valueMalformedFile))
+ .collect
+ }.getMessage
+ assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ }
+ }
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
--- End diff --
For non-whole-file parsing, we should skip the first line for each csv file (after we combine the partitioned files to a single file), right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73455/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by NathanHowell <gi...@git.apache.org>.
Github user NathanHowell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102662330
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val firstRow = csv.mapPartitions { iter =>
+ // We need this header line always because even if `inferSchema` option is disabled,
+ // it needs to know the number of columns that are string types without schema inference.
+ iter.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ parsedOptions.asParserSettings)
+ }
+ }.first()
--- End diff --
Does this work on 0 byte files?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103015905
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -958,4 +975,77 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
checkAnswer(df, Row(1, null))
}
}
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
+ .option("wholeFile", true)
+ .csv(path.getAbsolutePath)
+
+ // Check if headers have new lines in the names.
+ val actualFields = df.schema.fieldNames.toSeq
+ val expectedFields =
+ Seq("\nstring", "integer\n\n\n", "long\n\n", "bigInteger", "double", "boolean", "null")
--- End diff --
so the `string"` will just become `string`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by NathanHowell <gi...@git.apache.org>.
Github user NathanHowell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102668816
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
--- End diff --
Sounds good to me. Reducing code duplication between the JSON and CSV parsers would be great.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73088/testReport)** for PR 16976 at commit [`44c9465`](https://github.com/apache/spark/commit/44c946541f2b0bbeb00736f4b28db00a6169257f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16976
Can you open a JIRA with more description, expected input, and expected output?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103012984
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CodecStreams.scala ---
@@ -86,4 +88,11 @@ object CodecStreams {
.map(_.getDefaultExtension)
.getOrElse("")
}
+
+ /** Creates an input stream from the path. */
+ def createInputStream(config: Configuration, path: String): InputStream = {
+ val inputStream = CodecStreams.createInputStream(config, new Path(path))
--- End diff --
nit: we can write `createInputStream(config, new Path(path))` directly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102913767
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val shouldDropHeader = parsedOptions.headerFlag && file.start == 0
+ UnivocityParser.parseIterator(lines, shouldDropHeader, parser)
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.parseStream(
+ CodecStreams.createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parser)
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val maybeFirstRow: Option[Array[String]] = csv.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ CodecStreams.createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ new CsvParser(parsedOptions.asParserSettings))
+ }.take(1).headOption
+
+ if (maybeFirstRow.isDefined) {
--- End diff --
I just used `isDefined` and `get` here rather than `map` and `getOrElse` because it looks easier to read to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120030581
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -463,6 +463,7 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* <li>`columnNameOfCorruptRecord` (default is the value specified in
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field having malformed string
* created by `PERMISSIVE` mode. This overrides `spark.sql.columnNameOfCorruptRecord`.</li>
+ * <li>`wholeFile` (default `false`): parse one record, which may span multiple lines.</li>
--- End diff --
No, they are different. JSON reads whole file as a record whereas CSV reads each record when it meets multiple lines in a column.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102650051
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
--- End diff --
nit: I'd prefer `xxx.flatMap(line => parser.parse(line))`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73105/testReport)** for PR 16976 at commit [`2ddbf15`](https://github.com/apache/spark/commit/2ddbf153bd2860aa7d5cab8ed6e84295a02e7986).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102666649
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
--- End diff --
This was copied from `JsonDataSource.infer`. I would like to keep this in case when we deal with https://github.com/apache/spark/pull/16976#discussion_r102663016 and maybe introduce another parent or at least for consistency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73456/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102661723
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -958,4 +975,46 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
checkAnswer(df, Row(1, null))
}
}
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
+ .option("wholeFile", true)
+ .csv(path.getAbsolutePath)
--- End diff --
can we just check the schema and result of `df`?
```
assert(df.schema.map(_.name) == Seq("string\"", "integer\n\n", ...))
checkAnswer(df, Row("this is a\nsimple\nstring", ...) :: Row ...)
```
writing `df` out and reading it back and comparing the result doesn't mean `df` has corrected result.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73454 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73454/testReport)** for PR 16976 at commit [`321d082`](https://github.com/apache/spark/commit/321d082e4f0dbac34c39618adbded99d119e06ac).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73423/testReport)** for PR 16976 at commit [`793a3eb`](https://github.com/apache/spark/commit/793a3eb3f1b19c20aa5991c773c87bcd316d35a4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by vishnusram <gi...@git.apache.org>.
Github user vishnusram commented on the issue:
https://github.com/apache/spark/pull/16976
Thanks Kwon. It worked
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102916301
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala ---
@@ -36,7 +37,7 @@ import org.apache.spark.unsafe.types.UTF8String
private[csv] class UnivocityParser(
schema: StructType,
requiredSchema: StructType,
- options: CSVOptions) extends Logging {
+ private val options: CSVOptions) extends Logging {
--- End diff --
This one is accessed in the companion object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103014118
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val shouldDropHeader = parsedOptions.headerFlag && file.start == 0
+ UnivocityParser.parseIterator(lines, shouldDropHeader, parser)
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.parseStream(
+ CodecStreams.createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parser)
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val maybeFirstRow: Option[Array[String]] = csv.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ CodecStreams.createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ new CsvParser(parsedOptions.asParserSettings))
+ }.take(1).headOption
+
+ if (maybeFirstRow.isDefined) {
+ val firstRow = maybeFirstRow.get
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ CodecStreams.createInputStream(lines.getConfiguration, lines.getPath()),
+ parsedOptions.headerFlag,
+ new CsvParser(parsedOptions.asParserSettings))
+ }
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ } else {
+ // If the first row could not be read, just return the empty schema.
+ Some(StructType(Nil))
+ }
+ }
+
+ private def createBaseRdd(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): RDD[PortableDataStream] = {
+ val paths = inputPaths.map(_.getPath)
+ val name = paths.mkString(",")
+ val job = Job.getInstance(sparkSession.sessionState.newHadoopConf())
+ FileInputFormat.setInputPaths(job, paths: _*)
+ val conf = job.getConfiguration
+
+ new BinaryFileRDD(
+ sparkSession.sparkContext,
+ classOf[StreamInputFormat],
+ classOf[String],
+ classOf[PortableDataStream],
+ conf,
+ sparkSession.sparkContext.defaultMinPartitions)
+ .setName(s"CSVFile: $name")
+ .values
--- End diff --
BTW, where is `values` defined?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120030307
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -463,6 +463,7 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* <li>`columnNameOfCorruptRecord` (default is the value specified in
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field having malformed string
* created by `PERMISSIVE` mode. This overrides `spark.sql.columnNameOfCorruptRecord`.</li>
+ * <li>`wholeFile` (default `false`): parse one record, which may span multiple lines.</li>
--- End diff --
`multiple lines.` -> `multiple lines, per file`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16976
cc @NathanHowell, this basically resembles your implementation. Could you take a look please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73105/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102681341
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -958,4 +975,77 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
checkAnswer(df, Row(1, null))
}
}
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
+ .option("wholeFile", true)
+ .csv(path.getAbsolutePath)
+
+ // Check if headers have new lines in the names.
+ val actualFields = df.schema.fieldNames.toSeq
+ val expectedFields =
+ Seq("\nstring", "integer\n\n\n", "long\n\n", "bigInteger", "double", "boolean", "null")
--- End diff --
Default quote is `"`. This behaviour is definately with original CSV.
```scala
scala> Seq("\"1\"").toDF.write.text("/tmp/abc")
scala> spark.read.csv("/tmp/abc").show()
+---+
|_c0|
+---+
| 1|
+---+
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102664118
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val firstRow = csv.mapPartitions { iter =>
+ // We need this header line always because even if `inferSchema` option is disabled,
+ // it needs to know the number of columns that are string types without schema inference.
+ iter.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ parsedOptions.asParserSettings)
+ }
+ }.first()
--- End diff --
Let me check and add a test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [WIP][SPARK-19610][SQL] Support parsing multiline CSV fi...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16976
(I should clean it up more and take a look into this deeper. I left open this for running tests.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73422 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73422/testReport)** for PR 16976 at commit [`5d0bab4`](https://github.com/apache/spark/commit/5d0bab48ee8c862b9b0612f8a4c28a54e5243e09).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120030347
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala ---
@@ -261,6 +261,7 @@ final class DataStreamReader private[sql](sparkSession: SparkSession) extends Lo
* <li>`columnNameOfCorruptRecord` (default is the value specified in
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field having malformed string
* created by `PERMISSIVE` mode. This overrides `spark.sql.columnNameOfCorruptRecord`.</li>
+ * <li>`wholeFile` (default `false`): parse one record, which may span multiple lines.</li>
--- End diff --
The same here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73341 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73341/testReport)** for PR 16976 at commit [`8474933`](https://github.com/apache/spark/commit/84749338ec59e9c27dec3e1bbe8a35d1e5fd3fe7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102663902
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val firstRow = csv.mapPartitions { iter =>
--- End diff --
I think we have so far promoted the usage of `mapPartitions` and iterator for a better performance (not checked by myself though). At least, these are being used more commonly up to my knowledge and it makes it consistent. I would like to keep this way if you are fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73089/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73105 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73105/testReport)** for PR 16976 at commit [`2ddbf15`](https://github.com/apache/spark/commit/2ddbf153bd2860aa7d5cab8ed6e84295a02e7986).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by NathanHowell <gi...@git.apache.org>.
Github user NathanHowell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102665872
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
--- End diff --
This should also be tested on empty files, if it's not already.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102660255
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala ---
@@ -233,3 +236,31 @@ private[csv] class UnivocityParser(
}
}
}
+
+private[csv] object UnivocityParser {
+ def tokenizeStream(
+ inputStream: InputStream,
+ header: Boolean,
+ settings: CsvParserSettings): Iterator[Array[String]] = new Iterator[Array[String]] {
+ private val parser = new CsvParser(settings)
+ // Note that, here we assume `inputStream` is the whole file that might include the header.
+ parser.beginParsing(inputStream)
+ private var nextRecord = {
+ if (header) {
+ parser.parseNext()
--- End diff --
the normal code path has a more complex way to drop the header, is it ok to simply drop the first parsed record here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16976
It was renamed `multiLine` before the release. Could we try out it instead?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [WIP][SPARK-19610][SQL] Support parsing multiline CSV fi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102665288
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val firstRow = csv.mapPartitions { iter =>
--- End diff --
I see. Thank you. I didn't know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73341/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73423/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102651748
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala ---
@@ -233,3 +236,28 @@ private[csv] class UnivocityParser(
}
}
}
+
+private[csv] object UnivocityParser {
+ def tokenizeStream(
+ inputStream: InputStream,
+ header: Boolean,
+ settings: CsvParserSettings): Iterator[Array[String]] = new Iterator[Array[String]] {
+ private val parser = new CsvParser(settings)
+ // Note that, here we assume `inputStream` is the whole file that might include the header.
+ parser.beginParsing(inputStream)
+ private var nextRecord = {
+ if (header) {
+ parser.parseNext()
+ }
+ parser.parseNext()
+ }
+
+ override def hasNext: Boolean = nextRecord != null
+
+ override def next(): Array[String] = {
+ val curRecord = nextRecord
--- End diff --
nit: it's safer to add a check
```
if (!hasNext) {
throw new NoSuchElementException("End of stream")
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16976
@cloud-fan, I am running a build with 2.10 per b4e6983. I think it is ready for another look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120038036
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -463,6 +463,7 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* <li>`columnNameOfCorruptRecord` (default is the value specified in
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field having malformed string
* created by `PERMISSIVE` mode. This overrides `spark.sql.columnNameOfCorruptRecord`.</li>
+ * <li>`wholeFile` (default `false`): parse one record, which may span multiple lines.</li>
--- End diff --
Then, the option `wholeFile` is misleading to end users for `csv`. We should rename it to `multiLine` or others.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102666323
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val firstRow = csv.mapPartitions { iter =>
+ // We need this header line always because even if `inferSchema` option is disabled,
+ // it needs to know the number of columns that are string types without schema inference.
+ iter.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ parsedOptions.asParserSettings)
+ }
+ }.first()
--- End diff --
It seems the original one is not working too. Can I maybe open another JIRA and PR both together (or a follow-up)?
```scala
scala> spark.read.csv("a")
java.util.NoSuchElementException: next on empty iterator
at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.s
```
It seems we need a bit of fix including other CSV paths.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [WIP][SPARK-19610][SQL] Support parsing multiline CSV fi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73062 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73062/testReport)** for PR 16976 at commit [`3a303c7`](https://github.com/apache/spark/commit/3a303c7ac34163291aacd684546dbb30fd5dee8b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class CSVDataSource extends Serializable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102913028
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -961,56 +978,135 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
test("SPARK-18699 put malformed records in a `columnNameOfCorruptRecord` field") {
- val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
--- End diff --
I simple added
```scala
Seq(false, true).foreach { bool =>
...
.option("wholeFile", bool)
...
.option("wholeFile", bool)
...
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by NathanHowell <gi...@git.apache.org>.
Github user NathanHowell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102665619
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
--- End diff --
Both subclasses return `Some(...)` so it would be more clear to change this to `def infer(...): StructType` and do the option wrapping only in `CSVFileFormat.inferSchema`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73457/testReport)** for PR 16976 at commit [`b7d56c6`](https://github.com/apache/spark/commit/b7d56c6b6db7df7fe12831efb8338787b3108c0e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73336/testReport)** for PR 16976 at commit [`e00bdee`](https://github.com/apache/spark/commit/e00bdee2eb118d2b80b909cfb40c778b59ff8d4c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73417/testReport)** for PR 16976 at commit [`fbdfc61`](https://github.com/apache/spark/commit/fbdfc6189033d7c408108687ddb23f5e7435e945).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73417/testReport)** for PR 16976 at commit [`fbdfc61`](https://github.com/apache/spark/commit/fbdfc6189033d7c408108687ddb23f5e7435e945).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120033204
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -463,6 +463,7 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* <li>`columnNameOfCorruptRecord` (default is the value specified in
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field having malformed string
* created by `PERMISSIVE` mode. This overrides `spark.sql.columnNameOfCorruptRecord`.</li>
+ * <li>`wholeFile` (default `false`): parse one record, which may span multiple lines.</li>
--- End diff --
Still unable to get your point. Given an example?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16976
I initially left the similar comment in the same JIRAs. However, that there are a quite bit of similar JIRAs complaining about this and the original CSV (as the thirdparty library) tries to supports this although that is incorrectly implemented. This tries to match it with JSON one at least and it might be better to provide a way to process such CSV files. Actually, current implementation require quotes :).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by NathanHowell <gi...@git.apache.org>.
Github user NathanHowell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102667812
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val firstRow = csv.mapPartitions { iter =>
+ // We need this header line always because even if `inferSchema` option is disabled,
+ // it needs to know the number of columns that are string types without schema inference.
+ iter.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ parsedOptions.asParserSettings)
+ }
+ }.first()
--- End diff --
Another JIRA is a good idea for the line based parser, it could be fixed in this PR or seperately. `WholeFileCSVDataSource` should be fixed in this PR before merging, since it's new functionality.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120040302
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -463,6 +463,7 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* <li>`columnNameOfCorruptRecord` (default is the value specified in
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field having malformed string
* created by `PERMISSIVE` mode. This overrides `spark.sql.columnNameOfCorruptRecord`.</li>
+ * <li>`wholeFile` (default `false`): parse one record, which may span multiple lines.</li>
--- End diff --
In JSON, when `wholeFile` is on, we only parse one and only one record (i.e., table rows). The semantics are different. Let us fix the option name for CSV in 2.2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73089 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73089/testReport)** for PR 16976 at commit [`373eec9`](https://github.com/apache/spark/commit/373eec9ca8ac0ebef0186ac3dc9b9089a8dbdf35).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16976
Thanks for your review @NathanHowell and @cloud-fan. I just addressed them and ran a build with Scala 2.10 per 8474933.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102649654
--- Diff: python/test_support/sql/ages_newlines.csv ---
@@ -0,0 +1,6 @@
+Joe,20,"Hi,
+I am Jeo"
+Tom,30,"My name is Tom"
+Hyukjin,25,"I am Hyukjin
--- End diff --
wow you are only 25?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16976
Per 4dc48e1, I ran a build with Scala 2.10.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by NathanHowell <gi...@git.apache.org>.
Github user NathanHowell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102662637
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val firstRow = csv.mapPartitions { iter =>
+ // We need this header line always because even if `inferSchema` option is disabled,
+ // it needs to know the number of columns that are string types without schema inference.
+ iter.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ parsedOptions.asParserSettings)
+ }
+ }.first()
+
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.mapPartitions { iter =>
+ iter.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ createInputStream(lines.getConfiguration, lines.getPath()),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings)
+ }
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseRdd(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): RDD[PortableDataStream] = {
+ val paths = inputPaths.map(_.getPath)
+ val name = paths.mkString(",")
+ val job = Job.getInstance(sparkSession.sessionState.newHadoopConf())
+ FileInputFormat.setInputPaths(job, paths: _*)
+ val conf = job.getConfiguration
+
+ new BinaryFileRDD(
+ sparkSession.sparkContext,
+ classOf[StreamInputFormat],
+ classOf[String],
+ classOf[PortableDataStream],
+ conf,
+ sparkSession.sparkContext.defaultMinPartitions)
+ .setName(s"CSVFile: $name")
+ .values
+ }
+
+ private def createInputStream(config: Configuration, path: String): InputStream = {
--- End diff --
Might be nice to move this over to `CodecStreams` (with an appropriate name) and fix up other classes that use this pattern.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103015057
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala ---
@@ -269,3 +273,89 @@ private[csv] class UnivocityParser(
}
}
}
+
+private[csv] object UnivocityParser {
+
+ /**
+ * Parses a stream that contains CSV strings and turns it into an iterator of tokens.
+ */
+ def tokenizeStream(
+ inputStream: InputStream,
+ shouldDropHeader: Boolean,
+ tokenizer: CsvParser): Iterator[Array[String]] = new Iterator[Array[String]] {
+ // Note that, here we assume `inputStream` is the whole file that might include the header.
+ tokenizer.beginParsing(inputStream)
+ private var nextRecord = {
+ if (shouldDropHeader) {
+ tokenizer.parseNext()
+ }
+ tokenizer.parseNext()
+ }
+
+ override def hasNext: Boolean = nextRecord != null
+
+ override def next(): Array[String] = {
+ if (!hasNext) {
+ throw new NoSuchElementException("End of stream")
+ }
+ val curRecord = nextRecord
+ nextRecord = tokenizer.parseNext()
+ curRecord
+ }
+ }
+
+ /**
+ * Parses a stream that contains CSV strings and turns it into an iterator of rows.
+ */
+ def parseStream(
+ inputStream: InputStream,
+ shouldDropHeader: Boolean,
+ parser: UnivocityParser): Iterator[InternalRow] = {
+ val options = parser.options
+ val tokenizer = parser.tokenizer
+ val corruptFieldIndex = parser.corruptFieldIndex
+ val shouldKeepCurrentInput = options.permissive && corruptFieldIndex.isDefined
+
+ val tokenizedIter = tokenizeStream(inputStream, shouldDropHeader, tokenizer)
+
+ val parsedIter = new Iterator[Option[InternalRow]] {
+ private var currentInput: String = _
+
+ override def hasNext: Boolean = tokenizedIter.hasNext
+
+ override def next(): Option[InternalRow] = {
+ if (shouldKeepCurrentInput) {
+ // We should keep the current content before actually parsing in this case.
+ // Otherwise, we lose the string.
+ currentInput = tokenizer.getContext.currentParsedContent().stripLineEnd
--- End diff --
what's expected?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73456 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73456/testReport)** for PR 16976 at commit [`6187f6c`](https://github.com/apache/spark/commit/6187f6c0617fa0aec75d1eb638a00c6e7a350d61).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102671921
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val firstRow = csv.mapPartitions { iter =>
+ // We need this header line always because even if `inferSchema` option is disabled,
+ // it needs to know the number of columns that are string types without schema inference.
+ iter.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ parsedOptions.asParserSettings)
+ }
+ }.first()
--- End diff --
Sure, let me try.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102666948
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
--- End diff --
Yea, indeed it is an issue. https://github.com/apache/spark/pull/16976#discussion_r102662330
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102664832
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala ---
@@ -43,23 +37,26 @@ class CSVFileFormat extends TextBasedFileFormat with DataSourceRegister {
override def shortName(): String = "csv"
- override def toString: String = "CSV"
-
- override def hashCode(): Int = getClass.hashCode()
-
- override def equals(other: Any): Boolean = other.isInstanceOf[CSVFileFormat]
+ override def isSplitable(
+ sparkSession: SparkSession,
+ options: Map[String, String],
+ path: Path): Boolean = {
+ val parsedOptions =
+ new CSVOptions(options, sparkSession.sessionState.conf.sessionLocalTimeZone)
+ val csvDataSource = CSVDataSource(parsedOptions)
+ csvDataSource.isSplitable && super.isSplitable(sparkSession, options, path)
+ }
override def inferSchema(
sparkSession: SparkSession,
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
require(files.nonEmpty, "Cannot infer schema from an empty set of files")
--- End diff --
I think returning `None` is better, the upper-level will handle it and throw exception
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by NathanHowell <gi...@git.apache.org>.
Github user NathanHowell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102662258
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val firstRow = csv.mapPartitions { iter =>
--- End diff --
Could this be `csv.flatMap { lines => ...`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103057146
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CodecStreams.scala ---
@@ -86,4 +88,11 @@ object CodecStreams {
.map(_.getDefaultExtension)
.getOrElse("")
}
+
+ /** Creates an input stream from the path. */
--- End diff --
Sure, let me try to clean up this and other comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102666106
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala ---
@@ -43,23 +37,26 @@ class CSVFileFormat extends TextBasedFileFormat with DataSourceRegister {
override def shortName(): String = "csv"
- override def toString: String = "CSV"
-
- override def hashCode(): Int = getClass.hashCode()
-
- override def equals(other: Any): Boolean = other.isInstanceOf[CSVFileFormat]
+ override def isSplitable(
+ sparkSession: SparkSession,
+ options: Map[String, String],
+ path: Path): Boolean = {
+ val parsedOptions =
+ new CSVOptions(options, sparkSession.sessionState.conf.sessionLocalTimeZone)
+ val csvDataSource = CSVDataSource(parsedOptions)
+ csvDataSource.isSplitable && super.isSplitable(sparkSession, options, path)
+ }
override def inferSchema(
sparkSession: SparkSession,
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
require(files.nonEmpty, "Cannot infer schema from an empty set of files")
--- End diff --
This is not related with this JIRA itself. I would like to avoid other additional changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73417/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [WIP][SPARK-19610][SQL] Support parsing multiline CSV fi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73063/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120030318
--- Diff: python/pyspark/sql/streaming.py ---
@@ -637,6 +637,8 @@ def csv(self, path, schema=None, sep=None, encoding=None, quote=None, escape=Non
``spark.sql.columnNameOfCorruptRecord``. If None is set,
it uses the value specified in
``spark.sql.columnNameOfCorruptRecord``.
+ :param wholeFile: parse one record, which may span multiple lines. If None is
--- End diff --
The same here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16976
thanks, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73334/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73088/testReport)** for PR 16976 at commit [`44c9465`](https://github.com/apache/spark/commit/44c946541f2b0bbeb00736f4b28db00a6169257f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120033115
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -961,56 +968,121 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
test("SPARK-18699 put malformed records in a `columnNameOfCorruptRecord` field") {
- val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
- val df1 = spark
- .read
- .option("mode", "PERMISSIVE")
- .schema(schema)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df1,
- Row(null, null) ::
- Row(1, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
- val columnNameOfCorruptRecord = "_unparsed"
- val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
- val df2 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField1)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df2,
- Row(null, null, "0,2013-111-11 12:13:14") ::
- Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
- Nil)
-
- // We put a `columnNameOfCorruptRecord` field in the middle of a schema
- val schemaWithCorrField2 = new StructType()
- .add("a", IntegerType)
- .add(columnNameOfCorruptRecord, StringType)
- .add("b", TimestampType)
- val df3 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField2)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df3,
- Row(null, "0,2013-111-11 12:13:14", null) ::
- Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- val errMsg = intercept[AnalysisException] {
- spark
+ Seq(false, true).foreach { wholeFile =>
+ val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
+ val df1 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("wholeFile", wholeFile)
+ .schema(schema)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df1,
+ Row(null, null) ::
+ Row(1, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
+ val columnNameOfCorruptRecord = "_unparsed"
+ val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
+ val df2 = spark
.read
.option("mode", "PERMISSIVE")
.option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField1)
.csv(testFile(valueMalformedFile))
- .collect
- }.getMessage
- assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ checkAnswer(df2,
+ Row(null, null, "0,2013-111-11 12:13:14") ::
+ Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
+ Nil)
+
+ // We put a `columnNameOfCorruptRecord` field in the middle of a schema
+ val schemaWithCorrField2 = new StructType()
+ .add("a", IntegerType)
+ .add(columnNameOfCorruptRecord, StringType)
+ .add("b", TimestampType)
+ val df3 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField2)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df3,
+ Row(null, "0,2013-111-11 12:13:14", null) ::
+ Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ val errMsg = intercept[AnalysisException] {
+ spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .csv(testFile(valueMalformedFile))
+ .collect
+ }.getMessage
+ assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ }
+ }
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
--- End diff --
Let us check out and open an issue. I will try to help verify it and resolve at my best together.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120030339
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -385,6 +385,8 @@ def csv(self, path, schema=None, sep=None, encoding=None, quote=None, escape=Non
``spark.sql.columnNameOfCorruptRecord``. If None is set,
it uses the value specified in
``spark.sql.columnNameOfCorruptRecord``.
+ :param wholeFile: parse records, which may span multiple lines. If None is
--- End diff --
The same here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by vishnusram <gi...@git.apache.org>.
Github user vishnusram commented on the issue:
https://github.com/apache/spark/pull/16976
The wholeFile option doesn't seem to be working.
**Test file content:**
"num_col1","txt_col","num_col2"
10001,"regular string",20001
10002,"string with
newline",20002
**Command and result:**
>>> dfu = sqlContext.read.format('com.databricks.spark.csv').option("header","true").option("inferschema","true").option("delimiter",",").option("quote",'"').option("parserLib","univocity").option("wholeFile","true").load('new_line.csv')
>>> dfu.show(3,False)
+--------+--------------+--------+
|num_col1|txt_col |num_col2|
+--------+--------------+--------+
|10001 |regular string|20001 |
|10002 |string with |null |
|newline"|20002 |null |
+--------+--------------+--------+
**Expected result:**
+--------+--------------+--------+
|num_col1|txt_col |num_col2|
+--------+--------------+--------+
|10001 |regular string|20001 |
|10002 |string with\nnewline||20002 |
+--------+--------------+--------+
**Spark version used 2.2**
17/11/16 17:15:37 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.2.0
/_/
Using Python version 2.7.5 (default, May 3 2017 07:55:04)
SparkSession available as 'spark'.
Please let me know if I am missing something here
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73493 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73493/testReport)** for PR 16976 at commit [`22eb29e`](https://github.com/apache/spark/commit/22eb29eddfde484fbd887a32bf8be0b681c4d714).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class CSVDataSource extends Serializable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73454 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73454/testReport)** for PR 16976 at commit [`321d082`](https://github.com/apache/spark/commit/321d082e4f0dbac34c39618adbded99d119e06ac).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120038226
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -463,6 +463,7 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* <li>`columnNameOfCorruptRecord` (default is the value specified in
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field having malformed string
* created by `PERMISSIVE` mode. This overrides `spark.sql.columnNameOfCorruptRecord`.</li>
+ * <li>`wholeFile` (default `false`): parse one record, which may span multiple lines.</li>
--- End diff --
Yea, they are similar and different. I wouldn't mind opening a JIRA for this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73455 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73455/testReport)** for PR 16976 at commit [`b4e6983`](https://github.com/apache/spark/commit/b4e6983351192b788708ba3cd2a4c8e7321f34b0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73334 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73334/testReport)** for PR 16976 at commit [`4dc48e1`](https://github.com/apache/spark/commit/4dc48e19266be8dc39c89936e0957c768c112d8a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73457/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73089 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73089/testReport)** for PR 16976 at commit [`373eec9`](https://github.com/apache/spark/commit/373eec9ca8ac0ebef0186ac3dc9b9089a8dbdf35).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73422/testReport)** for PR 16976 at commit [`5d0bab4`](https://github.com/apache/spark/commit/5d0bab48ee8c862b9b0612f8a4c28a54e5243e09).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103059746
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -958,4 +975,77 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
checkAnswer(df, Row(1, null))
}
}
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
+ .option("wholeFile", true)
+ .csv(path.getAbsolutePath)
+
+ // Check if headers have new lines in the names.
+ val actualFields = df.schema.fieldNames.toSeq
+ val expectedFields =
+ Seq("\nstring", "integer\n\n\n", "long\n\n", "bigInteger", "double", "boolean", "null")
--- End diff --
Yup.
```
"
string"
```
becomes `\nstring`. It requires quotes for newlines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103060062
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -961,56 +978,135 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
test("SPARK-18699 put malformed records in a `columnNameOfCorruptRecord` field") {
- val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
- val df1 = spark
- .read
- .option("mode", "PERMISSIVE")
- .schema(schema)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df1,
- Row(null, null) ::
- Row(1, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
- val columnNameOfCorruptRecord = "_unparsed"
- val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
- val df2 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField1)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df2,
- Row(null, null, "0,2013-111-11 12:13:14") ::
- Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
- Nil)
-
- // We put a `columnNameOfCorruptRecord` field in the middle of a schema
- val schemaWithCorrField2 = new StructType()
- .add("a", IntegerType)
- .add(columnNameOfCorruptRecord, StringType)
- .add("b", TimestampType)
- val df3 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField2)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df3,
- Row(null, "0,2013-111-11 12:13:14", null) ::
- Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- val errMsg = intercept[AnalysisException] {
- spark
+ Seq(false, true).foreach { bool =>
+ val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
+ val df1 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("wholeFile", bool)
+ .schema(schema)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df1,
+ Row(null, null) ::
+ Row(1, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
+ val columnNameOfCorruptRecord = "_unparsed"
+ val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
+ val df2 = spark
.read
.option("mode", "PERMISSIVE")
.option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .option("wholeFile", bool)
+ .schema(schemaWithCorrField1)
.csv(testFile(valueMalformedFile))
- .collect
- }.getMessage
- assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ checkAnswer(df2,
+ Row(null, null, "0,2013-111-11 12:13:14") ::
+ Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
+ Nil)
+
+ // We put a `columnNameOfCorruptRecord` field in the middle of a schema
+ val schemaWithCorrField2 = new StructType()
+ .add("a", IntegerType)
+ .add(columnNameOfCorruptRecord, StringType)
+ .add("b", TimestampType)
+ val df3 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", bool)
+ .schema(schemaWithCorrField2)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df3,
+ Row(null, "0,2013-111-11 12:13:14", null) ::
+ Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ val errMsg = intercept[AnalysisException] {
+ spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", bool)
+ .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .csv(testFile(valueMalformedFile))
+ .collect
+ }.getMessage
+ assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ }
+ }
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
+ .option("wholeFile", true)
+ .csv(path.getAbsolutePath)
+
+ // Check if headers have new lines in the names.
+ val actualFields = df.schema.fieldNames.toSeq
+ val expectedFields =
+ Seq("\nstring", "integer\n\n\n", "long\n\n", "bigInteger", "double", "boolean", "null")
+ assert(actualFields === expectedFields)
+
+ // Check if the rows have new lines in the values.
+ val expected = Row(
+ "this is a\nsimple\nstring.",
+ "\n\n10",
+ "\n21474836470",
+ "92233720368547758070",
+ "\n\n1.7976931348623157E308",
+ "true",
+ null)
+ checkAnswer(df, expected)
+
+ val csvDir = new File(path.getAbsolutePath, "csv").getAbsolutePath
+ val cleanCols = df.schema
+ .map(f => regexp_replace(col(f.name), "[\\r\\n]", ""))
+ val dfWithoutWhiteSpaces = df.select(cleanCols: _*)
+ dfWithoutWhiteSpaces.write.csv(csvDir)
--- End diff --
I meant to test that we are able to read this back without `wholeFile` option if we remove all newlines.
```scala
// Check if the rows are the same if we remove all white spaces.
val readBack = spark.read.csv(csvDir)
checkAnswer(dfWithoutWhiteSpaces, readBack)
```
Let me clean up this one too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103556809
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -961,56 +968,121 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
test("SPARK-18699 put malformed records in a `columnNameOfCorruptRecord` field") {
- val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
- val df1 = spark
- .read
- .option("mode", "PERMISSIVE")
- .schema(schema)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df1,
- Row(null, null) ::
- Row(1, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
- val columnNameOfCorruptRecord = "_unparsed"
- val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
- val df2 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField1)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df2,
- Row(null, null, "0,2013-111-11 12:13:14") ::
- Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
- Nil)
-
- // We put a `columnNameOfCorruptRecord` field in the middle of a schema
- val schemaWithCorrField2 = new StructType()
- .add("a", IntegerType)
- .add(columnNameOfCorruptRecord, StringType)
- .add("b", TimestampType)
- val df3 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField2)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df3,
- Row(null, "0,2013-111-11 12:13:14", null) ::
- Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- val errMsg = intercept[AnalysisException] {
- spark
+ Seq(false, true).foreach { wholeFile =>
+ val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
+ val df1 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("wholeFile", wholeFile)
+ .schema(schema)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df1,
+ Row(null, null) ::
+ Row(1, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
+ val columnNameOfCorruptRecord = "_unparsed"
+ val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
+ val df2 = spark
.read
.option("mode", "PERMISSIVE")
.option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField1)
.csv(testFile(valueMalformedFile))
- .collect
- }.getMessage
- assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ checkAnswer(df2,
+ Row(null, null, "0,2013-111-11 12:13:14") ::
+ Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
+ Nil)
+
+ // We put a `columnNameOfCorruptRecord` field in the middle of a schema
+ val schemaWithCorrField2 = new StructType()
+ .add("a", IntegerType)
+ .add(columnNameOfCorruptRecord, StringType)
+ .add("b", TimestampType)
+ val df3 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField2)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df3,
+ Row(null, "0,2013-111-11 12:13:14", null) ::
+ Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ val errMsg = intercept[AnalysisException] {
+ spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .csv(testFile(valueMalformedFile))
+ .collect
+ }.getMessage
+ assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ }
+ }
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
+ .option("wholeFile", true)
+ .csv(path.getAbsolutePath)
+
+ // Check if headers have new lines in the names.
+ val actualFields = df.schema.fieldNames.toSeq
+ val expectedFields =
+ Seq("\nstring", "integer\n\n\n", "long\n\n", "bigInteger", "double", "boolean", "null")
+ assert(actualFields === expectedFields)
+
+ // Check if the rows have new lines in the values.
+ val expected = Row(
+ "this is a\nsimple\nstring.",
+ "\n\n10",
+ "\n21474836470",
+ "92233720368547758070",
+ "\n\n1.7976931348623157E308",
+ "true",
+ null)
+ checkAnswer(df, expected)
+ }
+ }
+
+ test("Empty file produces empty dataframe with empty schema - wholeFile option") {
+ withTempPath { path =>
+ path.createNewFile()
--- End diff --
actually we want `withTempDir`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [WIP][SPARK-19610][SQL] Support parsing multiline CSV fi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103012905
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CodecStreams.scala ---
@@ -86,4 +88,11 @@ object CodecStreams {
.map(_.getDefaultExtension)
.getOrElse("")
}
+
+ /** Creates an input stream from the path. */
--- End diff --
this doesn't tell the difference between this method and `createInputStream(config: Configuration, file: Path)`.
We also need a better name for this method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102663212
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala ---
@@ -233,3 +236,31 @@ private[csv] class UnivocityParser(
}
}
}
+
+private[csv] object UnivocityParser {
+ def tokenizeStream(
+ inputStream: InputStream,
+ header: Boolean,
+ settings: CsvParserSettings): Iterator[Array[String]] = new Iterator[Array[String]] {
+ private val parser = new CsvParser(settings)
+ // Note that, here we assume `inputStream` is the whole file that might include the header.
+ parser.beginParsing(inputStream)
+ private var nextRecord = {
+ if (header) {
+ parser.parseNext()
--- End diff --
Yup, existing code path processes line by line so comments and empty lines had to manually skipped because it threw an error if each line has only commented line or empty. This is nicely handled in Univocity and the initial version of CSV actually did this in this way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103060555
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val shouldDropHeader = parsedOptions.headerFlag && file.start == 0
+ UnivocityParser.parseIterator(lines, shouldDropHeader, parser)
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.parseStream(
+ CodecStreams.createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parser)
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val maybeFirstRow: Option[Array[String]] = csv.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ CodecStreams.createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ new CsvParser(parsedOptions.asParserSettings))
+ }.take(1).headOption
+
+ if (maybeFirstRow.isDefined) {
+ val firstRow = maybeFirstRow.get
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ CodecStreams.createInputStream(lines.getConfiguration, lines.getPath()),
+ parsedOptions.headerFlag,
+ new CsvParser(parsedOptions.asParserSettings))
+ }
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ } else {
+ // If the first row could not be read, just return the empty schema.
+ Some(StructType(Nil))
+ }
+ }
+
+ private def createBaseRdd(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): RDD[PortableDataStream] = {
+ val paths = inputPaths.map(_.getPath)
+ val name = paths.mkString(",")
+ val job = Job.getInstance(sparkSession.sessionState.newHadoopConf())
+ FileInputFormat.setInputPaths(job, paths: _*)
+ val conf = job.getConfiguration
+
+ new BinaryFileRDD(
+ sparkSession.sparkContext,
+ classOf[StreamInputFormat],
+ classOf[String],
+ classOf[PortableDataStream],
+ conf,
+ sparkSession.sparkContext.defaultMinPartitions)
+ .setName(s"CSVFile: $name")
+ .values
--- End diff --
(it seems defined [here](https://github.com/apache/spark/blob/0077bfcb93832d93009f73f4b80f2e3d98fd2fa4/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala#L1161))
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73457 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73457/testReport)** for PR 16976 at commit [`b7d56c6`](https://github.com/apache/spark/commit/b7d56c6b6db7df7fe12831efb8338787b3108c0e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102914694
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala ---
@@ -269,3 +273,89 @@ private[csv] class UnivocityParser(
}
}
}
+
+private[csv] object UnivocityParser {
+
+ /**
+ * Parses a stream that contains CSV strings and turns it into an iterator of tokens.
+ */
+ def tokenizeStream(
+ inputStream: InputStream,
+ shouldDropHeader: Boolean,
+ tokenizer: CsvParser): Iterator[Array[String]] = new Iterator[Array[String]] {
+ // Note that, here we assume `inputStream` is the whole file that might include the header.
+ tokenizer.beginParsing(inputStream)
+ private var nextRecord = {
+ if (shouldDropHeader) {
+ tokenizer.parseNext()
+ }
+ tokenizer.parseNext()
+ }
+
+ override def hasNext: Boolean = nextRecord != null
+
+ override def next(): Array[String] = {
+ if (!hasNext) {
+ throw new NoSuchElementException("End of stream")
+ }
+ val curRecord = nextRecord
+ nextRecord = tokenizer.parseNext()
+ curRecord
+ }
+ }
+
+ /**
+ * Parses a stream that contains CSV strings and turns it into an iterator of rows.
+ */
+ def parseStream(
+ inputStream: InputStream,
+ shouldDropHeader: Boolean,
+ parser: UnivocityParser): Iterator[InternalRow] = {
+ val options = parser.options
+ val tokenizer = parser.tokenizer
+ val corruptFieldIndex = parser.corruptFieldIndex
+ val shouldKeepCurrentInput = options.permissive && corruptFieldIndex.isDefined
+
+ val tokenizedIter = tokenizeStream(inputStream, shouldDropHeader, tokenizer)
+
+ val parsedIter = new Iterator[Option[InternalRow]] {
+ private var currentInput: String = _
+
+ override def hasNext: Boolean = tokenizedIter.hasNext
+
+ override def next(): Option[InternalRow] = {
+ if (shouldKeepCurrentInput) {
+ // We should keep the current content before actually parsing in this case.
+ // Otherwise, we lose the string.
+ currentInput = tokenizer.getContext.currentParsedContent().stripLineEnd
--- End diff --
It should strip the line end because Univocity parser seems reads the whole line including CRLF. It seems it does not strip following newlines as actual contents.
```scala
scala> "\n\n"
res0: String =
"
"
scala> "\n\n".stripLineEnd
res1: String =
"
"
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73455 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73455/testReport)** for PR 16976 at commit [`b4e6983`](https://github.com/apache/spark/commit/b4e6983351192b788708ba3cd2a4c8e7321f34b0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73088/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120031829
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -961,56 +968,121 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
test("SPARK-18699 put malformed records in a `columnNameOfCorruptRecord` field") {
- val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
- val df1 = spark
- .read
- .option("mode", "PERMISSIVE")
- .schema(schema)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df1,
- Row(null, null) ::
- Row(1, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
- val columnNameOfCorruptRecord = "_unparsed"
- val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
- val df2 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField1)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df2,
- Row(null, null, "0,2013-111-11 12:13:14") ::
- Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
- Nil)
-
- // We put a `columnNameOfCorruptRecord` field in the middle of a schema
- val schemaWithCorrField2 = new StructType()
- .add("a", IntegerType)
- .add(columnNameOfCorruptRecord, StringType)
- .add("b", TimestampType)
- val df3 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField2)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df3,
- Row(null, "0,2013-111-11 12:13:14", null) ::
- Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- val errMsg = intercept[AnalysisException] {
- spark
+ Seq(false, true).foreach { wholeFile =>
+ val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
+ val df1 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("wholeFile", wholeFile)
+ .schema(schema)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df1,
+ Row(null, null) ::
+ Row(1, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
+ val columnNameOfCorruptRecord = "_unparsed"
+ val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
+ val df2 = spark
.read
.option("mode", "PERMISSIVE")
.option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField1)
.csv(testFile(valueMalformedFile))
- .collect
- }.getMessage
- assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ checkAnswer(df2,
+ Row(null, null, "0,2013-111-11 12:13:14") ::
+ Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
+ Nil)
+
+ // We put a `columnNameOfCorruptRecord` field in the middle of a schema
+ val schemaWithCorrField2 = new StructType()
+ .add("a", IntegerType)
+ .add(columnNameOfCorruptRecord, StringType)
+ .add("b", TimestampType)
+ val df3 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField2)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df3,
+ Row(null, "0,2013-111-11 12:13:14", null) ::
+ Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ val errMsg = intercept[AnalysisException] {
+ spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .csv(testFile(valueMalformedFile))
+ .collect
+ }.getMessage
+ assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ }
+ }
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
--- End diff --
When we have a set of files, how can we guarantee the reading order is what the users like? That means, the first file we read is the file that contains header?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [WIP][SPARK-19610][SQL] Support parsing multiline CSV fi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73063 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73063/testReport)** for PR 16976 at commit [`8275d12`](https://github.com/apache/spark/commit/8275d123742bc0b339fd83c28ce1e16c75960a73).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103016252
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -961,56 +978,135 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
test("SPARK-18699 put malformed records in a `columnNameOfCorruptRecord` field") {
- val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
- val df1 = spark
- .read
- .option("mode", "PERMISSIVE")
- .schema(schema)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df1,
- Row(null, null) ::
- Row(1, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
- val columnNameOfCorruptRecord = "_unparsed"
- val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
- val df2 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField1)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df2,
- Row(null, null, "0,2013-111-11 12:13:14") ::
- Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
- Nil)
-
- // We put a `columnNameOfCorruptRecord` field in the middle of a schema
- val schemaWithCorrField2 = new StructType()
- .add("a", IntegerType)
- .add(columnNameOfCorruptRecord, StringType)
- .add("b", TimestampType)
- val df3 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField2)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df3,
- Row(null, "0,2013-111-11 12:13:14", null) ::
- Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- val errMsg = intercept[AnalysisException] {
- spark
+ Seq(false, true).foreach { bool =>
+ val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
+ val df1 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("wholeFile", bool)
+ .schema(schema)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df1,
+ Row(null, null) ::
+ Row(1, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
+ val columnNameOfCorruptRecord = "_unparsed"
+ val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
+ val df2 = spark
.read
.option("mode", "PERMISSIVE")
.option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .option("wholeFile", bool)
+ .schema(schemaWithCorrField1)
.csv(testFile(valueMalformedFile))
- .collect
- }.getMessage
- assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ checkAnswer(df2,
+ Row(null, null, "0,2013-111-11 12:13:14") ::
+ Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
+ Nil)
+
+ // We put a `columnNameOfCorruptRecord` field in the middle of a schema
+ val schemaWithCorrField2 = new StructType()
+ .add("a", IntegerType)
+ .add(columnNameOfCorruptRecord, StringType)
+ .add("b", TimestampType)
+ val df3 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", bool)
+ .schema(schemaWithCorrField2)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df3,
+ Row(null, "0,2013-111-11 12:13:14", null) ::
+ Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ val errMsg = intercept[AnalysisException] {
+ spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", bool)
+ .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .csv(testFile(valueMalformedFile))
+ .collect
+ }.getMessage
+ assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ }
+ }
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
+ .option("wholeFile", true)
+ .csv(path.getAbsolutePath)
+
+ // Check if headers have new lines in the names.
+ val actualFields = df.schema.fieldNames.toSeq
+ val expectedFields =
+ Seq("\nstring", "integer\n\n\n", "long\n\n", "bigInteger", "double", "boolean", "null")
+ assert(actualFields === expectedFields)
+
+ // Check if the rows have new lines in the values.
+ val expected = Row(
+ "this is a\nsimple\nstring.",
+ "\n\n10",
+ "\n21474836470",
+ "92233720368547758070",
+ "\n\n1.7976931348623157E308",
+ "true",
+ null)
+ checkAnswer(df, expected)
+
+ val csvDir = new File(path.getAbsolutePath, "csv").getAbsolutePath
+ val cleanCols = df.schema
+ .map(f => regexp_replace(col(f.name), "[\\r\\n]", ""))
+ val dfWithoutWhiteSpaces = df.select(cleanCols: _*)
+ dfWithoutWhiteSpaces.write.csv(csvDir)
--- End diff --
why we test the write path here? `wholeFile` is only about read path
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73336/testReport)** for PR 16976 at commit [`e00bdee`](https://github.com/apache/spark/commit/e00bdee2eb118d2b80b909cfb40c778b59ff8d4c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73423/testReport)** for PR 16976 at commit [`793a3eb`](https://github.com/apache/spark/commit/793a3eb3f1b19c20aa5991c773c87bcd316d35a4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120033012
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -961,56 +968,121 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
}
test("SPARK-18699 put malformed records in a `columnNameOfCorruptRecord` field") {
- val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
- val df1 = spark
- .read
- .option("mode", "PERMISSIVE")
- .schema(schema)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df1,
- Row(null, null) ::
- Row(1, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
- val columnNameOfCorruptRecord = "_unparsed"
- val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
- val df2 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField1)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df2,
- Row(null, null, "0,2013-111-11 12:13:14") ::
- Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
- Nil)
-
- // We put a `columnNameOfCorruptRecord` field in the middle of a schema
- val schemaWithCorrField2 = new StructType()
- .add("a", IntegerType)
- .add(columnNameOfCorruptRecord, StringType)
- .add("b", TimestampType)
- val df3 = spark
- .read
- .option("mode", "PERMISSIVE")
- .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schemaWithCorrField2)
- .csv(testFile(valueMalformedFile))
- checkAnswer(df3,
- Row(null, "0,2013-111-11 12:13:14", null) ::
- Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
- Nil)
-
- val errMsg = intercept[AnalysisException] {
- spark
+ Seq(false, true).foreach { wholeFile =>
+ val schema = new StructType().add("a", IntegerType).add("b", TimestampType)
+ val df1 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("wholeFile", wholeFile)
+ .schema(schema)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df1,
+ Row(null, null) ::
+ Row(1, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ // If `schema` has `columnNameOfCorruptRecord`, it should handle corrupt records
+ val columnNameOfCorruptRecord = "_unparsed"
+ val schemaWithCorrField1 = schema.add(columnNameOfCorruptRecord, StringType)
+ val df2 = spark
.read
.option("mode", "PERMISSIVE")
.option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
- .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField1)
.csv(testFile(valueMalformedFile))
- .collect
- }.getMessage
- assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ checkAnswer(df2,
+ Row(null, null, "0,2013-111-11 12:13:14") ::
+ Row(1, java.sql.Date.valueOf("1983-08-04"), null) ::
+ Nil)
+
+ // We put a `columnNameOfCorruptRecord` field in the middle of a schema
+ val schemaWithCorrField2 = new StructType()
+ .add("a", IntegerType)
+ .add(columnNameOfCorruptRecord, StringType)
+ .add("b", TimestampType)
+ val df3 = spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schemaWithCorrField2)
+ .csv(testFile(valueMalformedFile))
+ checkAnswer(df3,
+ Row(null, "0,2013-111-11 12:13:14", null) ::
+ Row(1, null, java.sql.Date.valueOf("1983-08-04")) ::
+ Nil)
+
+ val errMsg = intercept[AnalysisException] {
+ spark
+ .read
+ .option("mode", "PERMISSIVE")
+ .option("columnNameOfCorruptRecord", columnNameOfCorruptRecord)
+ .option("wholeFile", wholeFile)
+ .schema(schema.add(columnNameOfCorruptRecord, IntegerType))
+ .csv(testFile(valueMalformedFile))
+ .collect
+ }.getMessage
+ assert(errMsg.startsWith("The field for corrupt records must be string type and nullable"))
+ }
+ }
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
--- End diff --
Yea, that might be an issue. I guess this could be a general issue for CSV datasource itself. It sounds worth checking how it works.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16976
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102679211
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(line => parser.parse(line))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ CodecStreams.createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val maybeFirstRow: Option[Array[String]] = csv.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ CodecStreams.createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ parsedOptions.asParserSettings)
+ }.take(1).headOption
+
+ if (maybeFirstRow.isDefined) {
--- End diff --
I just used if-else here because `getOrElse` things seems making this hard to read to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r120034550
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -463,6 +463,7 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* <li>`columnNameOfCorruptRecord` (default is the value specified in
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field having malformed string
* created by `PERMISSIVE` mode. This overrides `spark.sql.columnNameOfCorruptRecord`.</li>
+ * <li>`wholeFile` (default `false`): parse one record, which may span multiple lines.</li>
--- End diff --
Definitely. I wanted to emphasize multiple lines is not _per file_.
For example, CSV reads multiple records (multiple lines) per file (newline is replaced to `\n` manually for readability).
```csv
"I am
Hyukjin Kwon"
"Hyukjin Kwon
I love Spark!"
```
```scala
scala> spark.read.option("wholeFile", true).csv("test.csv").show()
+---------------------+
| _c0|
+---------------------+
| I am\nHyukjin Kwon|
|Hyukjin Kwon\nI lo...|
+---------------------+
```
Whereas JSON reads the record _per file_. I am pretty sure object root support is primary.
```json
{
"I am": "HyukjinKwon",
"HyukjinKwon": "I love Spark!"
}
```
```scala
scala> spark.read.option("wholeFile", true).json("test.json").show()
+-------------+-----------+
| HyukjinKwon| I am|
+-------------+-----------+
|I love Spark!|HyukjinKwon|
+-------------+-----------+
```
but note that it could (in terms of input/output), work similarly with CSV when the input is a json array.
```json
[{
"I am": "HyukjinKwon",
"HyukjinKwon": "I love Spark!"
},{
"I am": "HyukjinKwon",
"HyukjinKwon": "I love Spark!"
}]
```
```scala
scala> spark.read.option("wholeFile", true).json("test.json").show()
+-------------+-----------+
| HyukjinKwon| I am|
+-------------+-----------+
|I love Spark!|HyukjinKwon|
|I love Spark!|HyukjinKwon|
+-------------+-----------+
```
Comparing array case and CSV, they work still differently. JSON, up to my knowledge, parses whole files and produces each record (in case or an array) whereas CSV parses record by record from the stream.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16976
Guys, please use the mailing list or JIRA ...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16976
cc @cloud-fan too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16976
hmm, I understand the motivation for this, though my understanding with csv generally either avoid having newline in field or some implementation would require quotes around field value with newline
https://en.wikipedia.org/wiki/Comma-separated_values#General_functionality
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by smram <gi...@git.apache.org>.
Github user smram commented on the issue:
https://github.com/apache/spark/pull/16976
> don't think it quite works
+1 i could not get this to work either @raviolli
Does it need the univocity parser to be specified as suggested initially here https://github.com/databricks/spark-csv/issues/175 ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73493/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103059540
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala ---
@@ -269,3 +273,89 @@ private[csv] class UnivocityParser(
}
}
}
+
+private[csv] object UnivocityParser {
+
+ /**
+ * Parses a stream that contains CSV strings and turns it into an iterator of tokens.
+ */
+ def tokenizeStream(
+ inputStream: InputStream,
+ shouldDropHeader: Boolean,
+ tokenizer: CsvParser): Iterator[Array[String]] = new Iterator[Array[String]] {
+ // Note that, here we assume `inputStream` is the whole file that might include the header.
+ tokenizer.beginParsing(inputStream)
+ private var nextRecord = {
+ if (shouldDropHeader) {
+ tokenizer.parseNext()
+ }
+ tokenizer.parseNext()
+ }
+
+ override def hasNext: Boolean = nextRecord != null
+
+ override def next(): Array[String] = {
+ if (!hasNext) {
+ throw new NoSuchElementException("End of stream")
+ }
+ val curRecord = nextRecord
+ nextRecord = tokenizer.parseNext()
+ curRecord
+ }
+ }
+
+ /**
+ * Parses a stream that contains CSV strings and turns it into an iterator of rows.
+ */
+ def parseStream(
+ inputStream: InputStream,
+ shouldDropHeader: Boolean,
+ parser: UnivocityParser): Iterator[InternalRow] = {
+ val options = parser.options
+ val tokenizer = parser.tokenizer
+ val corruptFieldIndex = parser.corruptFieldIndex
+ val shouldKeepCurrentInput = options.permissive && corruptFieldIndex.isDefined
+
+ val tokenizedIter = tokenizeStream(inputStream, shouldDropHeader, tokenizer)
+
+ val parsedIter = new Iterator[Option[InternalRow]] {
+ private var currentInput: String = _
+
+ override def hasNext: Boolean = tokenizedIter.hasNext
+
+ override def next(): Option[InternalRow] = {
+ if (shouldKeepCurrentInput) {
+ // We should keep the current content before actually parsing in this case.
+ // Otherwise, we lose the string.
+ currentInput = tokenizer.getContext.currentParsedContent().stripLineEnd
--- End diff --
It expects,
> Returns a String with the input character sequence parsed to produce the current record.
For example,
input
```csv
0,2013-111-11 12:13:14
1,1983-08-04
```
output (sequentially)
- `CsvParser.parseNext()`
```scala
Array("0", "2013-111-11 12:13:14")
```
- `CsvParser.getContext.currentParsedContent()`
```
0,2013-111-11 12:13:14\n
```
- `CsvParser.parseNext()`
```scala
Array("1", "1983-08-04")
```
- `CsvParser.getContext.currentParsedContent()`
```
1,1983-08-04\n
```
It seems re-use the internal buffer but it requires an overhead. Let me try to avoid this extra call.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [WIP][SPARK-19610][SQL] Support parsing multiline CSV fi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73062/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16976
**[Test build #73493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73493/testReport)** for PR 16976 at commit [`22eb29e`](https://github.com/apache/spark/commit/22eb29eddfde484fbd887a32bf8be0b681c4d714).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16976
I see, fair enough. cc @falaki
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102664433
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val linesWithoutHeader = if (parsedOptions.headerFlag && file.start == 0) {
+ // Note that if there are only comments in the first block, the header would probably
+ // be not dropped.
+ CSVUtils.dropHeaderLine(lines, parsedOptions)
+ } else {
+ lines
+ }
+
+ val filteredLines: Iterator[String] =
+ CSVUtils.filterCommentAndEmpty(linesWithoutHeader, parsedOptions)
+ filteredLines.flatMap(parser.parse(_: String))
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.tokenizeStream(
+ createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parsedOptions.asParserSettings).flatMap { tokens =>
+ parser.convert(tokens)
+ }
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val firstRow = csv.mapPartitions { iter =>
--- End diff --
it has better performance if you use `mapPartitionsInternal`, otherwise these 2 are exactly same...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Posted by raviolli <gi...@git.apache.org>.
Github user raviolli commented on the issue:
https://github.com/apache/spark/pull/16976
Dont think it quite works.
spark.read.option("wholeFile", "true").option("header", "true").option("inferSchema", "true").csv("FILE.csv").show()
+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+-----+-------+
| article_id| date| goldstein_scale| avg_tone| source| domain| text|title|authors|
+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+-----+-------+
| 671907787| 20170710| -2| -2.75049115913556|http://www.blackl...|http://www.blackl...|As much as the U....| null| null|
|Russian President...| 2014| celebrating the ...| null| null| null| null| null| null|
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r103014004
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala ---
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.csv
+
+import java.io.InputStream
+import java.nio.charset.{Charset, StandardCharsets}
+
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.io.{LongWritable, Text}
+import org.apache.hadoop.mapred.TextInputFormat
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
+
+import org.apache.spark.TaskContext
+import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
+import org.apache.spark.rdd.{BinaryFileRDD, RDD}
+import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.text.TextFileFormat
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Common functions for parsing CSV files
+ */
+abstract class CSVDataSource extends Serializable {
+ def isSplitable: Boolean
+
+ /**
+ * Parse a [[PartitionedFile]] into [[InternalRow]] instances.
+ */
+ def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow]
+
+ /**
+ * Infers the schema from `inputPaths` files.
+ */
+ def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType]
+
+ /**
+ * Generates a header from the given row which is null-safe and duplicate-safe.
+ */
+ protected def makeSafeHeader(
+ row: Array[String],
+ caseSensitive: Boolean,
+ options: CSVOptions): Array[String] = {
+ if (options.headerFlag) {
+ val duplicates = {
+ val headerNames = row.filter(_ != null)
+ .map(name => if (caseSensitive) name else name.toLowerCase)
+ headerNames.diff(headerNames.distinct).distinct
+ }
+
+ row.zipWithIndex.map { case (value, index) =>
+ if (value == null || value.isEmpty || value == options.nullValue) {
+ // When there are empty strings or the values set in `nullValue`, put the
+ // index as the suffix.
+ s"_c$index"
+ } else if (!caseSensitive && duplicates.contains(value.toLowerCase)) {
+ // When there are case-insensitive duplicates, put the index as the suffix.
+ s"$value$index"
+ } else if (duplicates.contains(value)) {
+ // When there are duplicates, put the index as the suffix.
+ s"$value$index"
+ } else {
+ value
+ }
+ }
+ } else {
+ row.zipWithIndex.map { case (_, index) =>
+ // Uses default column names, "_c#" where # is its position of fields
+ // when header option is disabled.
+ s"_c$index"
+ }
+ }
+ }
+}
+
+object CSVDataSource {
+ def apply(options: CSVOptions): CSVDataSource = {
+ if (options.wholeFile) {
+ WholeFileCSVDataSource
+ } else {
+ TextInputCSVDataSource
+ }
+ }
+}
+
+object TextInputCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = true
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ val lines = {
+ val linesReader = new HadoopFileLinesReader(file, conf)
+ Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => linesReader.close()))
+ linesReader.map { line =>
+ new String(line.getBytes, 0, line.getLength, parsedOptions.charset)
+ }
+ }
+
+ val shouldDropHeader = parsedOptions.headerFlag && file.start == 0
+ UnivocityParser.parseIterator(lines, shouldDropHeader, parser)
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: Dataset[String] = createBaseDataset(sparkSession, inputPaths, parsedOptions)
+ val firstLine: String = CSVUtils.filterCommentAndEmpty(csv, parsedOptions).first()
+ val firstRow = new CsvParser(parsedOptions.asParserSettings).parseLine(firstLine)
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.rdd.mapPartitions { iter =>
+ val filteredLines = CSVUtils.filterCommentAndEmpty(iter, parsedOptions)
+ val linesWithoutHeader =
+ CSVUtils.filterHeaderLine(filteredLines, firstLine, parsedOptions)
+ val parser = new CsvParser(parsedOptions.asParserSettings)
+ linesWithoutHeader.map(parser.parseLine)
+ }
+
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ }
+
+ private def createBaseDataset(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): Dataset[String] = {
+ val paths = inputPaths.map(_.getPath.toString)
+ if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
+ sparkSession.baseRelationToDataFrame(
+ DataSource.apply(
+ sparkSession,
+ paths = paths,
+ className = classOf[TextFileFormat].getName
+ ).resolveRelation(checkFilesExist = false))
+ .select("value").as[String](Encoders.STRING)
+ } else {
+ val charset = options.charset
+ val rdd = sparkSession.sparkContext
+ .hadoopFile[LongWritable, Text, TextInputFormat](paths.mkString(","))
+ .mapPartitions(_.map(pair => new String(pair._2.getBytes, 0, pair._2.getLength, charset)))
+ sparkSession.createDataset(rdd)(Encoders.STRING)
+ }
+ }
+}
+
+object WholeFileCSVDataSource extends CSVDataSource {
+ override val isSplitable: Boolean = false
+
+ override def readFile(
+ conf: Configuration,
+ file: PartitionedFile,
+ parser: UnivocityParser,
+ parsedOptions: CSVOptions): Iterator[InternalRow] = {
+ UnivocityParser.parseStream(
+ CodecStreams.createInputStream(conf, file.filePath),
+ parsedOptions.headerFlag,
+ parser)
+ }
+
+ override def infer(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ parsedOptions: CSVOptions): Option[StructType] = {
+ val csv: RDD[PortableDataStream] = createBaseRdd(sparkSession, inputPaths, parsedOptions)
+ val maybeFirstRow: Option[Array[String]] = csv.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ CodecStreams.createInputStream(lines.getConfiguration, lines.getPath()),
+ false,
+ new CsvParser(parsedOptions.asParserSettings))
+ }.take(1).headOption
+
+ if (maybeFirstRow.isDefined) {
+ val firstRow = maybeFirstRow.get
+ val caseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val header = makeSafeHeader(firstRow, caseSensitive, parsedOptions)
+ val tokenRDD = csv.flatMap { lines =>
+ UnivocityParser.tokenizeStream(
+ CodecStreams.createInputStream(lines.getConfiguration, lines.getPath()),
+ parsedOptions.headerFlag,
+ new CsvParser(parsedOptions.asParserSettings))
+ }
+ Some(CSVInferSchema.infer(tokenRDD, header, parsedOptions))
+ } else {
+ // If the first row could not be read, just return the empty schema.
+ Some(StructType(Nil))
+ }
+ }
+
+ private def createBaseRdd(
+ sparkSession: SparkSession,
+ inputPaths: Seq[FileStatus],
+ options: CSVOptions): RDD[PortableDataStream] = {
+ val paths = inputPaths.map(_.getPath)
+ val name = paths.mkString(",")
+ val job = Job.getInstance(sparkSession.sessionState.newHadoopConf())
+ FileInputFormat.setInputPaths(job, paths: _*)
+ val conf = job.getConfiguration
+
+ new BinaryFileRDD(
+ sparkSession.sparkContext,
+ classOf[StreamInputFormat],
+ classOf[String],
+ classOf[PortableDataStream],
+ conf,
+ sparkSession.sparkContext.defaultMinPartitions)
+ .setName(s"CSVFile: $name")
+ .values
--- End diff --
nit:
```
val rdd = new BinaryFileRDD ...
rdd.setName(..).values
```
is more readable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16976: [SPARK-19610][SQL] Support parsing multiline CSV ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16976#discussion_r102680021
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala ---
@@ -958,4 +975,46 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
checkAnswer(df, Row(1, null))
}
}
+
+ test("SPARK-19610: Parse normal multi-line CSV files") {
+ val primitiveFieldAndType = Seq(
+ """"
+ |string","integer
+ |
+ |
+ |","long
+ |
+ |","bigInteger",double,boolean,null""".stripMargin,
+ """"this is a
+ |simple
+ |string.","
+ |
+ |10","
+ |21474836470","92233720368547758070","
+ |
+ |1.7976931348623157E308",true,""".stripMargin)
+
+ withTempPath { path =>
+ primitiveFieldAndType.toDF("value").coalesce(1).write.text(path.getAbsolutePath)
+
+ val df = spark.read
+ .option("header", true)
+ .option("wholeFile", true)
+ .csv(path.getAbsolutePath)
--- End diff --
Added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org