(texera) branch gh-readonly-queue/main/pr-7568-19fb99fc89d70eabea84feca712002c2ee8ecd77 created (now 21ec0a011d)

[gi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-7568-19fb99fc89d70eabea84feca712002c2ee8ecd77
in repository https://gitbox.apache.org/repos/asf/texera.git


      at 21ec0a011d feat(csv-scan): keep a numeric column's type when one of its cells is blank (#7568)

This branch includes the following new commits:

     new 21ec0a011d feat(csv-scan): keep a numeric column's type when one of its cells is blank (#7568)

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(texera) 01/01: feat(csv-scan): keep a numeric column's type when one of its cells is blank (#7568)

[gi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7568-19fb99fc89d70eabea84feca712002c2ee8ecd77
in repository https://gitbox.apache.org/repos/asf/texera.git

commit 21ec0a011dd15659e97e63defae52f9293d552f3
Author: Kary Zheng <15...@users.noreply.github.com>
AuthorDate: Thu Aug 13 06:05:52 2026 +0000

    feat(csv-scan): keep a numeric column's type when one of its cells is blank (#7568)
    
    ### What changes were proposed in this PR?
    
    Schema inference and execution disagreed about what a blank cell is, and
    the schema side was the one that lost information. Inference set
    `nullValue("")` on its parser, so a blank read as an empty string:
    `tryParseDouble("")` fails, `tryParseBoolean("")` fails, and
    `inferField` lands on `tryParseString()`. One empty cell was enough to
    type a whole numeric column as STRING. Execution builds its parser
    without `nullValue`, so the same blank read as null there, which is what
    `AttributeTypeUtils.parseField` is written to pass through.
    
    The effect reaches well past the scan. Every downstream operator that
    does arithmetic on such a column then receives strings and fails on rows
    whose values are perfectly good numbers, not on the blank one. Hugging
    Face Iris Logistic Regression on a three-row file fails at the first
    row, where numpy is handed `array([['2.6', '0.75']], dtype='<U32')`.
    
    Dropping the setting leaves both sides reading a blank as null, and
    `tryParseDouble(null)` already answers DOUBLE, so the column keeps the
    type its values give it.
    
    One corner changes with it: a column that is blank in every sampled row
    now infers as INTEGER rather than STRING. Its values are null either
    way, so this renames the empty rather than reinterpreting anything.
    
    ### Any related issues, documentation, discussions?
    
    Closes #7550
    
    ### How was this PR tested?
    
    `CSVScanSourceOpDescSpec` gains a case that writes a two-column file
    whose numeric column is blank on one row and asserts the inferred type
    is DOUBLE. It fails on the previous behavior, 16 passed / 1 failed
    before the change and 17 / 0 after. Since this touches inference every
    scan goes through, the whole module was run as well: 2187 passed, 0
    failed.
    
    ### Was this PR authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5)
    
    Co-authored-by: Claude Opus 5 (1M context) <no...@anthropic.com>
---
 .../source/scan/csv/CSVScanSourceOpDesc.scala      |  5 ++++-
 .../source/scan/csv/CSVScanSourceOpDescSpec.scala  | 24 ++++++++++++++++++++++
 2 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
index 1a784a44ee..69088abb6c 100644
--- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
+++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
@@ -107,7 +107,10 @@ class CSVScanSourceOpDesc extends ScanSourceOpDesc {
     csvSetting.setMaxColumns(maxColumns)
     csvSetting.setFormat(csvFormat)
     csvSetting.setHeaderExtractionEnabled(hasHeader)
-    csvSetting.setNullValue("")
+    // No setNullValue here, so a blank cell reads as null exactly as it does at
+    // execution time (CSVScanSourceOpExec builds its parser without one). Reading it
+    // as "" instead made inferField fall through to STRING, which typed a numeric
+    // column by its one empty cell rather than by its values.
     val parser = new CsvParser(csvSetting)
     parser.beginParsing(inputReader)
 
diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala
index 3049d7f02a..fe426552a9 100644
--- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala
+++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala
@@ -97,6 +97,30 @@ class CSVScanSourceOpDescSpec extends AnyFlatSpec with BeforeAndAfter {
     opDesc.sourceSchema().getAttributes.map(_.getName).toList
   }
 
+  // Writes a numeric column with one blank cell and returns the absolute path.
+  private def writeCsvWithBlankNumericCell(): String = {
+    val tmpFile = Files.createTempFile("blank-cell-", ".csv")
+    tmpFile.toFile.deleteOnExit()
+    Files.write(
+      tmpFile,
+      "id,measure\n1,2.5\n2,\n3,4.5\n".getBytes(StandardCharsets.UTF_8)
+    )
+    tmpFile.toString
+  }
+
+  it should "infer a numeric column as DOUBLE even when one of its cells is blank" in {
+    val path = writeCsvWithBlankNumericCell()
+    csvScanSourceOpDesc.fileName = Some(path)
+    csvScanSourceOpDesc.setResolvedFileName(FileResolver.resolve(path))
+
+    // A blank used to read as "" while inferring and as null while executing. The ""
+    // fell through inferField to STRING, so one empty cell retyped the whole column
+    // and every downstream numeric operator then received strings.
+    assert(
+      csvScanSourceOpDesc.sourceSchema().getAttribute("measure").getType == AttributeType.DOUBLE
+    )
+  }
+
   it should "infer schema from single-line-data csv" in {
 
     parallelCsvScanSourceOpDesc.fileName = Some(TestOperators.CountrySalesSmallCsvPath)