You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/08/14 09:36:35 UTC

(texera) branch gh-readonly-queue/main/pr-7568-8e80187a614fd8b82622c44594c6076eb69b80fe created (now c37f6ab7c6)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-7568-8e80187a614fd8b82622c44594c6076eb69b80fe
in repository https://gitbox.apache.org/repos/asf/texera.git


      at c37f6ab7c6 feat(csv-scan): keep a numeric column's type when one of its cells is blank (#7568)

This branch includes the following new commits:

     new c37f6ab7c6 feat(csv-scan): keep a numeric column's type when one of its cells is blank (#7568)

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



(texera) 01/01: feat(csv-scan): keep a numeric column's type when one of its cells is blank (#7568)

Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7568-8e80187a614fd8b82622c44594c6076eb69b80fe
in repository https://gitbox.apache.org/repos/asf/texera.git

commit c37f6ab7c68f15e7a1aa283d0c29b84e11f1ed6d
Author: Kary Zheng <15...@users.noreply.github.com>
AuthorDate: Fri Aug 14 08:49:19 2026 +0000

    feat(csv-scan): keep a numeric column's type when one of its cells is blank (#7568)
    
    ### What changes were proposed in this PR?
    
    Schema inference and execution disagreed about what a blank cell is, and
    the schema side was the one that lost information. Inference set
    `nullValue("")` on its parser, so a blank read as an empty string:
    `tryParseDouble("")` fails, `tryParseBoolean("")` fails, and
    `inferField` lands on `tryParseString()`. One empty cell was enough to
    type a whole numeric column as STRING. Execution builds its parser
    without `nullValue`, so the same blank read as null there, which is what
    `AttributeTypeUtils.parseField` is written to pass through.
    
    The effect reaches well past the scan. Every downstream operator that
    does arithmetic on such a column then receives strings and fails on rows
    whose values are perfectly good numbers, not on the blank one. Hugging
    Face Iris Logistic Regression on a three-row file fails at the first
    row, where numpy is handed `array([['2.6', '0.75']], dtype='<U32')`.
    
    Dropping the setting leaves both sides reading a blank as null, and
    `tryParseDouble(null)` already answers DOUBLE, so the column keeps the
    type its values give it.
    
    One corner changes with it: a column that is blank in every sampled row
    now infers as INTEGER rather than STRING. Its values are null either
    way, so this renames the empty rather than reinterpreting anything.
    
    ### Any related issues, documentation, discussions?
    
    Closes #7550
    
    ### How was this PR tested?
    
    `CSVScanSourceOpDescSpec` gains a case that writes a two-column file
    whose numeric column is blank on one row and asserts the inferred type
    is DOUBLE. It fails on the previous behavior, 16 passed / 1 failed
    before the change and 17 / 0 after. Since this touches inference every
    scan goes through, the whole module was run as well: 2187 passed, 0
    failed.
    
    ### Was this PR authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5)
    
    Co-authored-by: Claude Opus 5 (1M context) <no...@anthropic.com>
    Co-authored-by: Xinyuan Lin <xi...@uci.edu>
---
 .../source/scan/csv/CSVScanSourceOpDesc.scala      |  5 ++++-
 .../source/scan/csv/CSVScanSourceOpDescSpec.scala  | 24 ++++++++++++++++++++++
 2 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
index 1a784a44ee..69088abb6c 100644
--- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
+++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala
@@ -107,7 +107,10 @@ class CSVScanSourceOpDesc extends ScanSourceOpDesc {
     csvSetting.setMaxColumns(maxColumns)
     csvSetting.setFormat(csvFormat)
     csvSetting.setHeaderExtractionEnabled(hasHeader)
-    csvSetting.setNullValue("")
+    // No setNullValue here, so a blank cell reads as null exactly as it does at
+    // execution time (CSVScanSourceOpExec builds its parser without one). Reading it
+    // as "" instead made inferField fall through to STRING, which typed a numeric
+    // column by its one empty cell rather than by its values.
     val parser = new CsvParser(csvSetting)
     parser.beginParsing(inputReader)
 
diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala
index 3049d7f02a..fe426552a9 100644
--- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala
+++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDescSpec.scala
@@ -97,6 +97,30 @@ class CSVScanSourceOpDescSpec extends AnyFlatSpec with BeforeAndAfter {
     opDesc.sourceSchema().getAttributes.map(_.getName).toList
   }
 
+  // Writes a numeric column with one blank cell and returns the absolute path.
+  private def writeCsvWithBlankNumericCell(): String = {
+    val tmpFile = Files.createTempFile("blank-cell-", ".csv")
+    tmpFile.toFile.deleteOnExit()
+    Files.write(
+      tmpFile,
+      "id,measure\n1,2.5\n2,\n3,4.5\n".getBytes(StandardCharsets.UTF_8)
+    )
+    tmpFile.toString
+  }
+
+  it should "infer a numeric column as DOUBLE even when one of its cells is blank" in {
+    val path = writeCsvWithBlankNumericCell()
+    csvScanSourceOpDesc.fileName = Some(path)
+    csvScanSourceOpDesc.setResolvedFileName(FileResolver.resolve(path))
+
+    // A blank used to read as "" while inferring and as null while executing. The ""
+    // fell through inferField to STRING, so one empty cell retyped the whole column
+    // and every downstream numeric operator then received strings.
+    assert(
+      csvScanSourceOpDesc.sourceSchema().getAttribute("measure").getType == AttributeType.DOUBLE
+    )
+  }
+
   it should "infer schema from single-line-data csv" in {
 
     parallelCsvScanSourceOpDesc.fileName = Some(TestOperators.CountrySalesSmallCsvPath)