Re: [PR] [SPARK-58458][SQL][TESTS] Add coverage for the record split when multiLine is disabled [spark]

["Joorgem (via GitHub)" <gi...@apache.org>]

Joorgem commented on PR #57671:
URL: https://github.com/apache/spark/pull/57671#issuecomment-5152542839

   Added in 3a9ef31 — thanks. FAILFAST now asserts `FAILED_READ_FILE.NO_HINT` via `checkErrorMatchPVals`, matching the suite's existing `test for FAILFAST parsing mode`.
   
   While doing it I had to fix something of my own. @szehon-ho pointed out on #57608 that "both halves are malformed" is too strong — each physical line is parsed alone, univocity's default `STOP_AT_DELIMITER` does not fail on an unclosed quote, and token count is what decides corruptness. That condemned this test's *comment*, which blamed the unbalanced quote. The assertions were right; the explanation was not, and nothing catches that.
   
   So the test now pins both shapes, because they differ in the way that matters:
   
   | schema | input | leading half | trailing half | `DROPMALFORMED` |
   |---|---|---|---|---|
   | 3 col | `1,"a⏎b",c` | malformed | malformed | drops both — whole record lost |
   | 2 col | `1,"hello⏎world"` | `[1, hello]` — **clean, unflagged** | malformed | **keeps the truncated row** |
   
   The second one is the reason the case is worth a test rather than only a doc note: `DROPMALFORMED` retains a row that looks valid and is not, precisely because its token count still matches. The first case is the benign one — nothing silently wrong survives it.
   
   Verified green on the full `sql` matrix (`slow`, `extended`, `other`) against `master` before pushing, so it holds in `CSVv1Suite`, `CSVv2Suite` and `CSVLegacyTimeParserSuite`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org