You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Amer-Mukhtar (via GitHub)" <gi...@apache.org> on 2026/10/07 09:25:55 UTC
[PR] perf(workflow-core): speed up CSV field and timestamp parsing [texera]
Amer-Mukhtar opened a new pull request, #8907:
URL: https://github.com/apache/texera/pull/8907
Description:
### What changes were proposed in this PR?
The CSV scan spends most of its time converting text fields to typed values, mostly timestamps. User-facing behavior is unchanged: same values, same errors, same warnings.
- `AttributeTypeUtils.parseFields` fills an array in a plain loop instead of building a collection per row.
- Timestamp parsing uses one `DateParser` per thread instead of `DateParserUtils.parseDate`, which is `static synchronized` and serializes all workers in the JVM.
- New fast path for zone-less ISO timestamps (`yyyy-MM-dd[( |T)HH:mm[:ss[.f]]]`). It sets the same `Calendar` fields the library does, so results are identical; any other string still goes to the library.
- `CSVScanSourceOpExec` and `ParallelCSVScanSourceOpExec` build the column type array once per scan instead of once per row.
| Full CSV scan, 2M rows | Before | After |
|---|---|---|
| int, double, string, bool, long | 876 ms | 553 ms |
| + ISO timestamp column | 3593 ms | 1026 ms |
### Any related issues, documentation, discussions?
Related to #8684 (direction 2, faster per-field parsing). The Arrow parts of that issue depend on #8558.
### How was this PR tested?
Added tests to `AttributeTypeUtilsSpec` (short and null rows, parse-error message and cause, timestamps compared against the date-parser library on ~4,800 valid and invalid strings, parsing from 8 threads at once) and to `CSVScanSourceOpExecSpec` (each field comes out as its inferred type). A one-off check also compared the fast path with the library on 137,760 strings in 7 time zones, with 0 mismatches.
sbt "WorkflowCore/testOnly org.apache.texera.amber.core.tuple.AttributeTypeUtilsSpec" "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.scan.csv.*"
sbt WorkflowCore/test WorkflowOperator/test
sbt "WorkflowCore/scalafixAll --check" "WorkflowOperator/scalafixAll --check" WorkflowCore/scalafmtCheckAll WorkflowOperator/scalafmtCheckAll
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(workflow-core): speed up CSV field and timestamp parsing [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on code in PR #8907:
URL: https://github.com/apache/texera/pull/8907#discussion_r4382381771
##########
common/workflow-core/src/main/scala/org/apache/texera/amber/core/tuple/AttributeTypeUtils.scala:
##########
@@ -228,6 +245,69 @@ object AttributeTypeUtils extends Serializable {
}
+ /**
+ * Epoch millis of a zone-less ISO local date-time, `yyyy-MM-dd` optionally
+ * followed by `[ T]HH:mm[:ss[.f]]` with 1-9 fraction digits, or Long.MinValue
+ * when `s` is not exactly that shape or names no real date-time.
+ *
+ * This is the form a CSV timestamp column usually holds, and the general date
+ * parser spends most of a scan's time on it. For a zone-less string that parser
+ * sets these same fields on a fresh default Calendar, so doing that here gives
+ * the identical instant; anything else falls through to the parser unchanged.
+ */
+ private def parseIsoLocalMillis(s: String): Long = {
+ val len = s.length
+ if (len != 10 && len != 16 && len < 19) return Long.MinValue
+ def digit(i: Int): Int = {
+ val c = s.charAt(i)
+ if (c >= '0' && c <= '9') c - '0' else -1000
+ }
+ def twoDigits(i: Int): Int = digit(i) * 10 + digit(i + 1)
+
+ if (s.charAt(4) != '-' || s.charAt(7) != '-') return Long.MinValue
+ val year = digit(0) * 1000 + digit(1) * 100 + twoDigits(2)
Review Comment:
A non-digit in the 4th year position slips through, because `digit()` returns -1000 and the year stays positive. For example, `202a-01-05` now parses as 1020-01-05, while `DateParser` rejects it, so could the fast path return `Long.MinValue` on any non-digit and add such inputs to the comparison test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(workflow-core): speed up CSV field and timestamp parsing [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8907:
URL: https://github.com/apache/texera/pull/8907#issuecomment-6035240696
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8907?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 92.59%. Comparing base ([`2cd44ee`](https://app.codecov.io/gh/apache/texera/commit/2cd44ee0d8219a7cc0f16e8c4ee869c0bad9feec?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`73a1ead`](https://app.codecov.io/gh/apache/texera/commit/73a1ead40aba128df4ffd4529c5299b1b5749fa0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8907 +/- ##
============================================
- Coverage 92.59% 92.59% -0.01%
Complexity 5043 5043
============================================
Files 1252 1252
Lines 53583 53576 -7
Branches 6672 6672
============================================
- Hits 49617 49610 -7
Misses 2319 2319
Partials 1647 1647
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8907/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8907/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.38% <ø> (ø)` | | |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8907/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.16% <ø> (ø)` | | Carriedforward from [2cd44ee](https://app.codecov.io/gh/apache/texera/commit/2cd44ee0d8219a7cc0f16e8c4ee869c0bad9feec?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8907/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `88.06% <ø> (-0.01%)` | :arrow_down: | Carriedforward from [2cd44ee](https://app.codecov.io/gh/apache/texera/commit/2cd44ee0d8219a7cc0f16e8c4ee869c0bad9feec?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8907/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.41% <ø> (ø)` | | |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8907/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.37% <ø> (ø)` | | |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8907/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.53% <ø> (ø)` | | |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8907/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.59% <ø> (ø)` | | Carriedforward from [2cd44ee](https://app.codecov.io/gh/apache/texera/commit/2cd44ee0d8219a7cc0f16e8c4ee869c0bad9feec?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8907/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.73% <ø> (ø)` | | |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8907/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.58% <ø> (ø)` | | Carriedforward from [2cd44ee](https://app.codecov.io/gh/apache/texera/commit/2cd44ee0d8219a7cc0f16e8c4ee869c0bad9feec?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8907/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.09% <ø> (ø)` | | |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8907?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(workflow-core): speed up CSV field and timestamp parsing [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8907:
URL: https://github.com/apache/texera/pull/8907#issuecomment-6035121130
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 0 better · 🔴 12 worse · ⚪ 3 noise (<±5%) · 0 without baseline
> Compared against main `e2238c4` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/37600568721)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 362 | 0.221 | 26,539/39,448/39,448 us | 🔴 +12.6% / 🔴 +191.9% |
| 🔴 | bs=100 sw=10 sl=64 | 760 | 0.464 | 127,456/177,426/177,426 us | 🔴 +24.8% / 🔴 +89.2% |
| 🔴 | bs=1000 sw=10 sl=64 | 905 | 0.552 | 1,102,002/1,196,348/1,196,348 us | 🔴 +6.4% / 🔴 +32.3% |
<details><summary>Baseline details</summary>
Latest main `e2238c4` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 362 tuples/sec | 393 tuples/sec | 963.37 tuples/sec | -7.9% | -62.4% |
| bs=10 sw=10 sl=64 | MB/s | 0.221 MB/s | 0.24 MB/s | 0.588 MB/s | -7.9% | -62.4% |
| bs=10 sw=10 sl=64 | p50 | 26,539 us | 24,967 us | 10,998 us | +6.3% | +141.3% |
| bs=10 sw=10 sl=64 | p95 | 39,448 us | 35,037 us | 13,515 us | +12.6% | +191.9% |
| bs=10 sw=10 sl=64 | p99 | 39,448 us | 35,037 us | 16,850 us | +12.6% | +134.1% |
| bs=100 sw=10 sl=64 | throughput | 760 tuples/sec | 820 tuples/sec | 1,235 tuples/sec | -7.3% | -38.5% |
| bs=100 sw=10 sl=64 | MB/s | 0.464 MB/s | 0.501 MB/s | 0.754 MB/s | -7.4% | -38.5% |
| bs=100 sw=10 sl=64 | p50 | 127,456 us | 120,278 us | 87,833 us | +6.0% | +45.1% |
| bs=100 sw=10 sl=64 | p95 | 177,426 us | 142,173 us | 93,795 us | +24.8% | +89.2% |
| bs=100 sw=10 sl=64 | p99 | 177,426 us | 142,173 us | 103,718 us | +24.8% | +71.1% |
| bs=1000 sw=10 sl=64 | throughput | 905 tuples/sec | 920 tuples/sec | 1,273 tuples/sec | -1.6% | -28.9% |
| bs=1000 sw=10 sl=64 | MB/s | 0.552 MB/s | 0.561 MB/s | 0.777 MB/s | -1.6% | -28.9% |
| bs=1000 sw=10 sl=64 | p50 | 1,102,002 us | 1,090,880 us | 861,707 us | +1.0% | +27.9% |
| bs=1000 sw=10 sl=64 | p95 | 1,196,348 us | 1,124,788 us | 904,523 us | +6.4% | +32.3% |
| bs=1000 sw=10 sl=64 | p99 | 1,196,348 us | 1,124,788 us | 931,473 us | +6.4% | +28.4% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,552.11,200,128000,362,0.221,26539.20,39447.56,39447.56
1,100,10,64,20,2630.13,2000,1280000,760,0.464,127455.94,177425.90,177425.90
2,1000,10,64,20,22111.11,20000,12800000,905,0.552,1102001.53,1196348.18,1196348.18
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(workflow-core): speed up CSV field and timestamp parsing [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8907:
URL: https://github.com/apache/texera/pull/8907#issuecomment-6035024904
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@eugenegujing`, `@carloea2`, `@aglinxinyuan`
You can notify them by mentioning `@eugenegujing`, `@carloea2`, `@aglinxinyuan` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org