You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "kz930 (via GitHub)" <gi...@apache.org> on 2026/08/11 20:44:53 UTC
[PR] fix(csv-scan): keep a numeric column's type when one of its cells is blank [texera]
kz930 opened a new pull request, #7568:
URL: https://github.com/apache/texera/pull/7568
### What changes were proposed in this PR?
Schema inference and execution disagreed about what a blank cell is, and the schema side was the one that lost information. Inference set `nullValue("")` on its parser, so a blank read as an empty string: `tryParseDouble("")` fails, `tryParseBoolean("")` fails, and `inferField` lands on `tryParseString()`. One empty cell was enough to type a whole numeric column as STRING. Execution builds its parser without `nullValue`, so the same blank read as null there, which is what `AttributeTypeUtils.parseField` is written to pass through.
The effect reaches well past the scan. Every downstream operator that does arithmetic on such a column then receives strings and fails on rows whose values are perfectly good numbers, not on the blank one. Hugging Face Iris Logistic Regression on a three-row file fails at the first row, where numpy is handed `array([['2.6', '0.75']], dtype='<U32')`.
Dropping the setting leaves both sides reading a blank as null, and `tryParseDouble(null)` already answers DOUBLE, so the column keeps the type its values give it.
One corner changes with it: a column that is blank in every sampled row now infers as INTEGER rather than STRING. Its values are null either way, so this renames the empty rather than reinterpreting anything.
### Any related issues, documentation, discussions?
Closes #7550
### How was this PR tested?
`CSVScanSourceOpDescSpec` gains a case that writes a two-column file whose numeric column is blank on one row and asserts the inferred type is DOUBLE. It fails on the previous behavior, 16 passed / 1 failed before the change and 17 / 0 after. Since this touches inference every scan goes through, the whole module was run as well: 2187 passed, 0 failed.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csv-scan): keep a numeric column's type when one of its cells is blank [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #7568:
URL: https://github.com/apache/texera/pull/7568#issuecomment-5258746360
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/7568?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 87.88%. Comparing base ([`e878df3`](https://app.codecov.io/gh/apache/texera/commit/e878df3eee7ca3baee7b0c34eceb63548c59e68a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`9f49d90`](https://app.codecov.io/gh/apache/texera/commit/9f49d90dd1e637d8bc91861c6a489c93da501818?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #7568 +/- ##
============================================
- Coverage 87.88% 87.88% -0.01%
Complexity 4276 4276
============================================
Files 1176 1176
Lines 47018 47017 -1
Branches 5245 5245
============================================
- Hits 41323 41322 -1
Misses 3972 3972
Partials 1723 1723
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/7568/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/7568/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.00% <ø> (ø)` | | |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/7568/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.62% <ø> (ø)` | | Carriedforward from [e878df3](https://app.codecov.io/gh/apache/texera/commit/e878df3eee7ca3baee7b0c34eceb63548c59e68a?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/7568/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.88% <ø> (-0.01%)` | :arrow_down: | Carriedforward from [e878df3](https://app.codecov.io/gh/apache/texera/commit/e878df3eee7ca3baee7b0c34eceb63548c59e68a?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/7568/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.38% <ø> (ø)` | | |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/7568/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.97% <ø> (ø)` | | |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/7568/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.05% <ø> (ø)` | | Carriedforward from [e878df3](https://app.codecov.io/gh/apache/texera/commit/e878df3eee7ca3baee7b0c34eceb63548c59e68a?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/7568/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `89.94% <ø> (ø)` | | Carriedforward from [e878df3](https://app.codecov.io/gh/apache/texera/commit/e878df3eee7ca3baee7b0c34eceb63548c59e68a?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/7568/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.89% <ø> (ø)` | | |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/7568/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `97.52% <ø> (ø)` | | Carriedforward from [e878df3](https://app.codecov.io/gh/apache/texera/commit/e878df3eee7ca3baee7b0c34eceb63548c59e68a?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/7568/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `26.31% <ø> (ø)` | | Carriedforward from [e878df3](https://app.codecov.io/gh/apache/texera/commit/e878df3eee7ca3baee7b0c34eceb63548c59e68a?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/7568?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(csv-scan): keep a numeric column's type when one of its cells is blank [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7568:
URL: https://github.com/apache/texera/pull/7568#issuecomment-5258712385
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- No candidates found from `git blame` history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(csv-scan): keep a numeric column's type when one of its cells is blank [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7568:
URL: https://github.com/apache/texera/pull/7568#issuecomment-5258712473
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 0 better · 🔴 6 worse · ⚪ 9 noise (<±5%) · 0 without baseline
> Compared against main `e878df3` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/31534514926)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 398 | 0.243 | 24,724/32,895/32,895 us | 🔴 +17.1% / 🔴 +105.7% |
| 🔴 | bs=100 sw=10 sl=64 | 938 | 0.573 | 106,070/138,290/138,290 us | 🔴 +6.7% / 🔴 +28.0% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,092 | 0.666 | 915,870/996,096/996,096 us | ⚪ within ±5% / 🟢 -7.7% |
<details><summary>Baseline details</summary>
Latest main `e878df3` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 398 tuples/sec | 466 tuples/sec | 784.3 tuples/sec | -14.6% | -49.3% |
| bs=10 sw=10 sl=64 | MB/s | 0.243 MB/s | 0.284 MB/s | 0.479 MB/s | -14.4% | -49.2% |
| bs=10 sw=10 sl=64 | p50 | 24,724 us | 22,647 us | 12,590 us | +9.2% | +96.4% |
| bs=10 sw=10 sl=64 | p95 | 32,895 us | 28,100 us | 15,991 us | +17.1% | +105.7% |
| bs=10 sw=10 sl=64 | p99 | 32,895 us | 28,100 us | 18,694 us | +17.1% | +76.0% |
| bs=100 sw=10 sl=64 | throughput | 938 tuples/sec | 975 tuples/sec | 1,002 tuples/sec | -3.8% | -6.4% |
| bs=100 sw=10 sl=64 | MB/s | 0.573 MB/s | 0.595 MB/s | 0.612 MB/s | -3.7% | -6.3% |
| bs=100 sw=10 sl=64 | p50 | 106,070 us | 99,445 us | 101,285 us | +6.7% | +4.7% |
| bs=100 sw=10 sl=64 | p95 | 138,290 us | 137,286 us | 108,068 us | +0.7% | +28.0% |
| bs=100 sw=10 sl=64 | p99 | 138,290 us | 137,286 us | 118,235 us | +0.7% | +17.0% |
| bs=1000 sw=10 sl=64 | throughput | 1,092 tuples/sec | 1,111 tuples/sec | 1,030 tuples/sec | -1.7% | +6.0% |
| bs=1000 sw=10 sl=64 | MB/s | 0.666 MB/s | 0.678 MB/s | 0.629 MB/s | -1.8% | +5.9% |
| bs=1000 sw=10 sl=64 | p50 | 915,870 us | 902,249 us | 991,882 us | +1.5% | -7.7% |
| bs=1000 sw=10 sl=64 | p95 | 996,096 us | 970,696 us | 1,038,496 us | +2.6% | -4.1% |
| bs=1000 sw=10 sl=64 | p99 | 996,096 us | 970,696 us | 1,068,265 us | +2.6% | -6.8% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,503.04,200,128000,398,0.243,24723.65,32895.05,32895.05
1,100,10,64,20,2131.74,2000,1280000,938,0.573,106070.23,138290.26,138290.26
2,1000,10,64,20,18322.51,20000,12800000,1092,0.666,915870.20,996096.30,996096.30
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(csv-scan): keep a numeric column's type when one of its cells is blank [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7568:
URL: https://github.com/apache/texera/pull/7568#issuecomment-5258649547
<!-- backport-auto-label-report -->
### Backport auto-label report
This `fix:` PR was checked against each actively-supported release branch. `release/*` labels drive the post-merge backport, so add or remove one to change where this fix lands.
| Release branch | Analysis |
| --- | --- |
| ✅ `release/v1.2` | Change detected on this branch — label added; this fix is queued to backport here. Requested review from @xuang7. |
_[Auto-label run](https://github.com/apache/texera/actions/runs/31534491738)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csv-scan): keep a numeric column's type when one of its cells is blank [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7568:
URL: https://github.com/apache/texera/pull/7568#issuecomment-5259320550
@aglinxinyuan May you take a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csv-scan): keep a numeric column's type when one of its cells is blank [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan merged PR #7568:
URL: https://github.com/apache/texera/pull/7568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org