You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "Baunsgaard (via GitHub)" <gi...@apache.org> on 2026/06/25 09:05:37 UTC
[PR] [SYSTEMDS-3949] Add native Delta Lake matrix read/write via Delta Kernel [systemds]
Baunsgaard opened a new pull request, #2511:
URL: https://github.com/apache/systemds/pull/2511
Introduce a DELTA file format that reads and writes Delta Lake tables natively through the Spark-free Delta Kernel library, for matrices on the single-node CP path. DML read/write with format="delta" now operates directly on Delta tables without a Spark DataFrame round-trip.
- Add FileFormat.DELTA and exclude it from the text formats
- Accept format="delta" with unknown dimensions in the parser (like CSV) and set blocksize -1 for the columnar format
- Wire DELTA into the matrix reader and writer factories
- Add DeltaKernelUtils plus ReaderDelta and WriterDelta with column-at-a-time, boxing-free data transfer
- Refresh cached matrix metadata after a Delta read (discovered dimensions)
- Pin parquet 1.13.1 and jackson core/annotations to 2.15.2 to align with Delta Kernel's transitive requirements
Append/overwrite table semantics, distributed execution, frames, and time travel are out of scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SYSTEMDS-3949] Add native Delta Lake matrix read/write via Delta Kernel [systemds]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2511:
URL: https://github.com/apache/systemds/pull/2511#issuecomment-4797852429
## [Codecov](https://app.codecov.io/gh/apache/systemds/pull/2511?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `70.53571%` with `66 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 71.53%. Comparing base ([`0637bf1`](https://app.codecov.io/gh/apache/systemds/commit/0637bf15958df968c08f69771091b79db33421a2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`06deac3`](https://app.codecov.io/gh/apache/systemds/commit/06deac3f8ccd240c43504dd9f60e2c7693c290b6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/systemds/pull/2511?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../java/org/apache/sysds/runtime/io/ReaderDelta.java](https://app.codecov.io/gh/apache/systemds/pull/2511?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Fio%2FReaderDelta.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvaW8vUmVhZGVyRGVsdGEuamF2YQ==) | 62.68% | [18 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2511?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../org/apache/sysds/runtime/io/DeltaKernelUtils.java](https://app.codecov.io/gh/apache/systemds/pull/2511?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Fio%2FDeltaKernelUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvaW8vRGVsdGFLZXJuZWxVdGlscy5qYXZh) | 71.79% | [19 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2511?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/sysds/runtime/io/WriterDelta.java](https://app.codecov.io/gh/apache/systemds/pull/2511?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Fio%2FWriterDelta.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvaW8vV3JpdGVyRGVsdGEuamF2YQ==) | 75.38% | [11 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2511?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/sysds/runtime/io/MatrixReaderFactory.java](https://app.codecov.io/gh/apache/systemds/pull/2511?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Fio%2FMatrixReaderFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvaW8vTWF0cml4UmVhZGVyRmFjdG9yeS5qYXZh) | 50.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2511?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...n/java/org/apache/sysds/parser/DataExpression.java](https://app.codecov.io/gh/apache/systemds/pull/2511?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fparser%2FDataExpression.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3BhcnNlci9EYXRhRXhwcmVzc2lvbi5qYXZh) | 83.33% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2511?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2511 +/- ##
============================================
- Coverage 71.55% 71.53% -0.03%
- Complexity 49062 49066 +4
============================================
Files 1574 1577 +3
Lines 189570 189788 +218
Branches 37188 37227 +39
============================================
+ Hits 135650 135761 +111
- Misses 43442 43526 +84
- Partials 10478 10501 +23
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/systemds/pull/2511?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SYSTEMDS-3949] Add native Delta Lake matrix read/write via Delta Kernel [systemds]
Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #2511:
URL: https://github.com/apache/systemds/pull/2511#issuecomment-4799568110
### Native Delta read performance vs binary reader
Benchmark (`org.apache.sysds.performance` `DeltaIO`), 32 vCPU, JDK 17, `-Xmx16g`,
20 timed repetitions after warmup, isolated (no competing load).
**Workload:** 5,000,000 × 16 dense FP64 matrix (~640 MB in memory, sparsity 1.0).
| Read path | Time (ms) | Disk throughput |
|----------------------|----------------|-----------------|
| Delta (serial) | 5411 ± 79 | 79 MB/s |
| **Delta (parallel)** | **875 ± 19** | **490 MB/s** |
| Binary (parallel) | ~100–170 | 3.7–6.0 GB/s |
**Takeaways**
- The parallel reader decodes data files concurrently and is **~6.2× faster than the
serial Delta reader** (5411 ms → 875 ms).
- The binary reader is still **~5–9× faster** than parallel Delta. This is expected:
binary is a near-raw `double[]` dump, while Delta Kernel 3.3.0 decodes parquet
per-value (row-by-row converters, not vectorized). File-level parallelism is the
main lever available, which is what this reader exploits.
- **On disk Delta is smaller:** 450 MB vs 645 MB for binary (0.70×), thanks to parquet
encoding — so the read-time cost buys a ~30% smaller footprint.
For reference, the write path is encode-bound: Delta write ~7.0 s vs binary ~0.15 s
(parquet encoding dominates); not the focus of this PR but noted for completeness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SYSTEMDS-3949] Add native Delta Lake matrix read/write via Delta Kernel [systemds]
Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard merged PR #2511:
URL: https://github.com/apache/systemds/pull/2511
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org