You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/08/13 06:06:08 UTC
(texera) 01/01: feat(csv-scan): keep a numeric column's type when one of its cells is blank (#7568)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7568-19fb99fc89d70eabea84feca712002c2ee8ecd77
in repository https://gitbox.apache.org/repos/asf/texera.git
commit 21ec0a011dd15659e97e63defae52f9293d552f3
Author: Kary Zheng <15...@users.noreply.github.com>
AuthorDate: Thu Aug 13 06:05:52 2026 +0000
feat(csv-scan): keep a numeric column's type when one of its cells is blank (#7568)
### What changes were proposed in this PR?
Schema inference and execution disagreed about what a blank cell is, and
the schema side was the one that lost information. Inference set
`nullValue("")` on its parser, so a blank read as an empty string:
`tryParseDouble("")` fails, `tryParseBoolean("")` fails, and
`inferField` lands on `tryParseString()`. One empty cell was enough to
type a whole numeric column as STRING. Execution builds its parser
without `nullValue`, so the same blank read as null there, which is what
`AttributeTypeUtils.parseField` is written to pass through.
The effect reaches well past the scan. Every downstream operator that
does arithmetic on such a column then receives strings and fails on rows
whose values are perfectly good numbers, not on the blank one. Hugging
Face Iris Logistic Regression on a three-row file fails at the first
row, where numpy is handed `array([['2.6', '0.75']], dtype='<U32')`.
Dropping the setting leaves both sides reading a blank as null, and
`tryParseDouble(null)` already answers DOUBLE, so the column keeps the
type its values give it.
One corner changes with it: a column that is blank in every sampled row
now infers as INTEGER rather than STRING. Its values are null either
way, so this renames the empty rather than reinterpreting anything.
### Any related issues, documentation, discussions?
Closes #7550
### How was this PR tested?
`CSVScanSourceOpDescSpec` gains a case that writes a two-column file
whose numeric column is blank on one row and asserts the inferred type
is DOUBLE. It fails on the previous behavior, 16 passed / 1 failed
before the change and 17 / 0 after. Since this touches inference every
scan goes through, the whole module was run as well: 2187 passed, 0
failed.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
Co-authored-by: Claude Opus 5 (1M context) <no...@anthropic.com>
---
.../source/scan/csv/CSVScanSourceOpDesc.scala | 5 ++++-
.../source/scan/csv/CSVScanSourceOpDescSpec.scala | 24 ++++++++++++++++++++++
2 files changed, 28 insertions(+), 1 deletion(-)
diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
index 1a784a44ee..69088abb6c 100644
--- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
+++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
@@ -107,7 +107,10 @@ class CSVScanSourceOpDesc extends ScanSourceOpDesc {
csvSetting.setMaxColumns(maxColumns)
csvSetting.setFormat(csvFormat)
csvSetting.setHeaderExtractionEnabled(hasHeader)
- csvSetting.setNullValue("")
+ // No setNullValue here, so a blank cell reads as null exactly as it does at
+ // execution time (CSVScanSourceOpExec builds its parser without one). Reading it
+ // as "" instead made inferField fall through to STRING, which typed a numeric
+ // column by its one empty cell rather than by its values.
val parser = new CsvParser(csvSetting)
parser.beginParsing(inputReader)
diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala
index 3049d7f02a..fe426552a9 100644
--- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala
+++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala
@@ -97,6 +97,30 @@ class CSVScanSourceOpDescSpec extends AnyFlatSpec with BeforeAndAfter {
opDesc.sourceSchema().getAttributes.map(_.getName).toList
}
+ // Writes a numeric column with one blank cell and returns the absolute path.
+ private def writeCsvWithBlankNumericCell(): String = {
+ val tmpFile = Files.createTempFile("blank-cell-", ".csv")
+ tmpFile.toFile.deleteOnExit()
+ Files.write(
+ tmpFile,
+ "id,measure\n1,2.5\n2,\n3,4.5\n".getBytes(StandardCharsets.UTF_8)
+ )
+ tmpFile.toString
+ }
+
+ it should "infer a numeric column as DOUBLE even when one of its cells is blank" in {
+ val path = writeCsvWithBlankNumericCell()
+ csvScanSourceOpDesc.fileName = Some(path)
+ csvScanSourceOpDesc.setResolvedFileName(FileResolver.resolve(path))
+
+ // A blank used to read as "" while inferring and as null while executing. The ""
+ // fell through inferField to STRING, so one empty cell retyped the whole column
+ // and every downstream numeric operator then received strings.
+ assert(
+ csvScanSourceOpDesc.sourceSchema().getAttribute("measure").getType == AttributeType.DOUBLE
+ )
+ }
+
it should "infer schema from single-line-data csv" in {
parallelCsvScanSourceOpDesc.fileName = Some(TestOperators.CountrySalesSmallCsvPath)