You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/04/13 02:56:40 UTC

[GitHub] [spark] panbingkun opened a new pull request, #36169: [SPARK-38721][SQL][TESTS] Test the error class: CANNOT_PARSE_DECIMAL

panbingkun opened a new pull request, #36169:
URL: https://github.com/apache/spark/pull/36169

   ## What changes were proposed in this pull request?
   This pr aims to Add one test for the error class CANNOT_PARSE_DECIMAL to QueryExecutionErrorsSuite, it's a followup of [SPARK-37935](https://issues.apache.org/jira/browse/SPARK-37935).
   
   ### Why are the changes needed?
   Add one test for the error class CANNOT_PARSE_DECIMAL to QueryExecutionErrorsSuite. 
   
   ### Does this PR introduce any user-facing change?
   No
   
   ### How was this patch tested?
   - Manual test:
   
   ```
   build/sbt "sql/testOnly *QueryExecutionErrorsSuite*"
   ```
   
   All tests passed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on a diff in pull request #36169: [SPARK-38721][SQL][TESTS] Test the error class: CANNOT_PARSE_DECIMAL

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #36169:
URL: https://github.com/apache/spark/pull/36169#discussion_r849649086


##########
sql/core/src/test/resources/test-data/unparseable_decimal.csv:
##########
@@ -0,0 +1,2 @@
+money

Review Comment:
   Could you write the file in the test like:
   ```scala
       withTempPath { path =>
         Seq(
           "money",
           "$92,807.99"
         ).toDF("data")
           .coalesce(1)
           .write.text(path.getAbsolutePath)
   ```
   see example in CSVSuite. We don't need unnecessary dependencies in the resource folder if we can easily form the intput.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] panbingkun commented on pull request #36169: [SPARK-38721][SQL][TESTS] Test the error class: CANNOT_PARSE_DECIMAL

Posted by GitBox <gi...@apache.org>.
panbingkun commented on PR #36169:
URL: https://github.com/apache/spark/pull/36169#issuecomment-1098721037

   new pr: https://github.com/apache/spark/pull/36192
   ### why new pr?
   I accidentally deleted old repo on GitHub。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] LuciferYang commented on pull request #36169: [SPARK-38721][SQL][TESTS] Test the error class: CANNOT_PARSE_DECIMAL

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36169:
URL: https://github.com/apache/spark/pull/36169#issuecomment-1097501753

   cc @MaxGekk 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] panbingkun commented on a diff in pull request #36169: [SPARK-38721][SQL][TESTS] Test the error class: CANNOT_PARSE_DECIMAL

Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #36169:
URL: https://github.com/apache/spark/pull/36169#discussion_r849382118


##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala:
##########
@@ -278,4 +281,14 @@ class QueryExecutionErrorsSuite extends QueryTest
     assert(e.getMessage ===
       "Datetime operation overflow: add 1000000 YEAR to TIMESTAMP '2022-03-09 01:02:03'.")
   }
+
+  test("CANNOT_PARSE_DECIMAL: unparseable decimal") {
+    val e = intercept[SparkIllegalStateException] {
+      val decimalParser = ExprUtils.getDecimalParser(Locale.ROOT)
+      decimalParser("$92,807.99")

Review Comment:
   I do it, pls review again!Unusually deep exception



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on a diff in pull request #36169: [SPARK-38721][SQL][TESTS] Test the error class: CANNOT_PARSE_DECIMAL

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #36169:
URL: https://github.com/apache/spark/pull/36169#discussion_r849091336


##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala:
##########
@@ -278,4 +281,14 @@ class QueryExecutionErrorsSuite extends QueryTest
     assert(e.getMessage ===
       "Datetime operation overflow: add 1000000 YEAR to TIMESTAMP '2022-03-09 01:02:03'.")
   }
+
+  test("CANNOT_PARSE_DECIMAL: unparseable decimal") {
+    val e = intercept[SparkIllegalStateException] {
+      val decimalParser = ExprUtils.getDecimalParser(Locale.ROOT)
+      decimalParser("$92,807.99")

Review Comment:
   Could you raise the exception from user space (not using internal methods), please. We assign error classes not for user visible errors, just in case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] panbingkun commented on a diff in pull request #36169: [SPARK-38721][SQL][TESTS] Test the error class: CANNOT_PARSE_DECIMAL

Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #36169:
URL: https://github.com/apache/spark/pull/36169#discussion_r850075349


##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala:
##########
@@ -278,4 +284,35 @@ class QueryExecutionErrorsSuite extends QueryTest
     assert(e.getMessage ===
       "Datetime operation overflow: add 1000000 YEAR to TIMESTAMP '2022-03-09 01:02:03'.")
   }
+
+  test("CANNOT_PARSE_DECIMAL: unparseable decimal") {
+    val e1 = intercept[SparkException] {
+      val schema = new StructType().add("money", DecimalType.DoubleDecimal)
+      spark
+        .read
+        .schema(schema)
+        .format("csv")
+        .option("header", "true")
+        .option("locale", Locale.ROOT.toLanguageTag)
+        .option("multiLine", "true")
+        .option("inferSchema", "false")
+        .option("mode", "FAILFAST")

Review Comment:
   ### from_csv don't support FailFastMode, code as folllow:
   if (mode != PermissiveMode && mode != FailFastMode) {
         throw QueryCompilationErrors.parseModeUnsupportedError("from_csv", mode)
   }
   
   ### In FailureSafeParser:
   def parse(input: IN): Iterator[InternalRow] = {
       try {
         rawParser.apply(input).iterator.map(row => toResultRow(Some(row), () => null))
       } catch {
         case e: BadRecordException => mode match {
           case PermissiveMode =>
             Iterator(toResultRow(e.partialResult(), e.record))
           case DropMalformedMode =>
             Iterator.empty
           case FailFastMode =>
             throw QueryExecutionErrors.malformedRecordsDetectedInRecordParsingError(e)
         }
       }
     }
   if Mode is not FailFastMode, the deep exception: CANNOT_PARSE_DECIMAL, can't throw。
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on a diff in pull request #36169: [SPARK-38721][SQL][TESTS] Test the error class: CANNOT_PARSE_DECIMAL

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #36169:
URL: https://github.com/apache/spark/pull/36169#discussion_r849649086


##########
sql/core/src/test/resources/test-data/unparseable_decimal.csv:
##########
@@ -0,0 +1,2 @@
+money

Review Comment:
   Could you write the file in the test like:
   ```scala
       withTempPath { path =>
         Seq(
           "money",
           "$92,807.99"
         ).toDF().coalesce(1).write.text(path.getAbsolutePath)
   ```
   see examples in CSVSuite. We don't need unnecessary dependencies in the resource folder if we can easily form the input.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] panbingkun commented on a diff in pull request #36169: [SPARK-38721][SQL][TESTS] Test the error class: CANNOT_PARSE_DECIMAL

Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #36169:
URL: https://github.com/apache/spark/pull/36169#discussion_r850075349


##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala:
##########
@@ -278,4 +284,35 @@ class QueryExecutionErrorsSuite extends QueryTest
     assert(e.getMessage ===
       "Datetime operation overflow: add 1000000 YEAR to TIMESTAMP '2022-03-09 01:02:03'.")
   }
+
+  test("CANNOT_PARSE_DECIMAL: unparseable decimal") {
+    val e1 = intercept[SparkException] {
+      val schema = new StructType().add("money", DecimalType.DoubleDecimal)
+      spark
+        .read
+        .schema(schema)
+        .format("csv")
+        .option("header", "true")
+        .option("locale", Locale.ROOT.toLanguageTag)
+        .option("multiLine", "true")
+        .option("inferSchema", "false")
+        .option("mode", "FAILFAST")

Review Comment:
   ### from_csv don't support FailFastMode, code as folllow:
   #### In CsvToStructs.parser
   if (mode != PermissiveMode && mode != FailFastMode) {
         throw QueryCompilationErrors.parseModeUnsupportedError("from_csv", mode)
   }
   ##
   ### if Mode is not FailFastMode, the deep exception: CANNOT_PARSE_DECIMAL, can't throw。
   #### In FailureSafeParser:
   def parse(input: IN): Iterator[InternalRow] = {
       try {
         rawParser.apply(input).iterator.map(row => toResultRow(Some(row), () => null))
       } catch {
         case e: BadRecordException => mode match {
           case PermissiveMode =>
             Iterator(toResultRow(e.partialResult(), e.record))
           case DropMalformedMode =>
             Iterator.empty
           case FailFastMode =>
             throw QueryExecutionErrors.malformedRecordsDetectedInRecordParsingError(e)
         }
       }
     }
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on a diff in pull request #36169: [SPARK-38721][SQL][TESTS] Test the error class: CANNOT_PARSE_DECIMAL

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #36169:
URL: https://github.com/apache/spark/pull/36169#discussion_r849091336


##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala:
##########
@@ -278,4 +281,14 @@ class QueryExecutionErrorsSuite extends QueryTest
     assert(e.getMessage ===
       "Datetime operation overflow: add 1000000 YEAR to TIMESTAMP '2022-03-09 01:02:03'.")
   }
+
+  test("CANNOT_PARSE_DECIMAL: unparseable decimal") {
+    val e = intercept[SparkIllegalStateException] {
+      val decimalParser = ExprUtils.getDecimalParser(Locale.ROOT)
+      decimalParser("$92,807.99")

Review Comment:
   Could you raise the exception from user space/code (not using internal methods), please. We assign error classes **only** for user visible errors, just in case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on a diff in pull request #36169: [SPARK-38721][SQL][TESTS] Test the error class: CANNOT_PARSE_DECIMAL

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #36169:
URL: https://github.com/apache/spark/pull/36169#discussion_r849650320


##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala:
##########
@@ -278,4 +284,35 @@ class QueryExecutionErrorsSuite extends QueryTest
     assert(e.getMessage ===
       "Datetime operation overflow: add 1000000 YEAR to TIMESTAMP '2022-03-09 01:02:03'.")
   }
+
+  test("CANNOT_PARSE_DECIMAL: unparseable decimal") {
+    val e1 = intercept[SparkException] {
+      val schema = new StructType().add("money", DecimalType.DoubleDecimal)
+      spark
+        .read
+        .schema(schema)
+        .format("csv")
+        .option("header", "true")
+        .option("locale", Locale.ROOT.toLanguageTag)
+        .option("multiLine", "true")
+        .option("inferSchema", "false")
+        .option("mode", "FAILFAST")

Review Comment:
   Just wonder, is it possible to use the `from_csv` function? If so, we could avoid access to the file system.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org