You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2018/12/17 16:30:41 UTC

[GitHub] bersprockets commented on a change in pull request #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables

bersprockets commented on a change in pull request #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables
URL: https://github.com/apache/spark/pull/23336#discussion_r242218860
 
 

 ##########
 File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
 ##########
 @@ -82,7 +82,12 @@ class UnivocityParser(
 
   // Retrieve the raw record string.
   private def getCurrentInput: UTF8String = {
-    UTF8String.fromString(tokenizer.getContext.currentParsedContent().stripLineEnd)
+    val currentParsedContent = tokenizer.getContext.currentParsedContent()
+    if (currentParsedContent == null) {
 
 Review comment:
   This was to get around a problem with CsvExpressionsSuite tests, which intentionally produces some bad records but also bypasses the tokenizer and calls UnivocityParser#convert directly. This was not a problem before, because those tests did not specify a corrupt records field.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org