You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "Baunsgaard (via GitHub)" <gi...@apache.org> on 2026/06/26 16:52:05 UTC

[PR] [SYSTEMDS-3949] Add Demo for Databricks Workspace Execution [systemds]

Baunsgaard opened a new pull request, #2520:
URL: https://github.com/apache/systemds/pull/2520

   Add scripts/databricks: a self-contained kit for deploying and running SystemDS on a Databricks cluster (tested on DBR 16.4 LTS, Spark 3.5.2, Scala 2.12, where the SystemDS jar runs unchanged).
   
   - deploy.sh: create a UC volume, upload SystemDS.jar, create a single-user cluster with the required Vector API / --add-opens JVM flags, install the Delta Kernel Maven libraries, and import the demo notebooks. All settings come from a .env file (template in .env.example); local state (.env, .cluster_id) is git-ignored.
   - SystemDS_MLContext_Demo.scala: Unity Catalog table round-trip via the MLContext (Scala) API with a configurable DML script and execution mode.
   - SystemDS_vs_SparkML_LinReg.scala: linear regression with categorical encoding (transformencode + lm) vs a Spark ML OneHotEncoder + LinearRegression pipeline, timing encode + train.
   - SystemDS_Delta_E2E.scala: end-to-end Delta -> transformencode -> lm, reading a Delta table natively as a frame, compared against the equivalent Spark ML pipeline; prints a per-instruction breakdown.
   - SystemDS_Python_Demo.py and demo.dml: minimal Python API and DML smoke tests.
   - README.md: setup, configuration, node-type guidance, Delta Kernel library requirement, and indicative benchmark numbers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3949] Add Demo for Databricks Workspace Execution [systemds]

Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard merged PR #2520:
URL: https://github.com/apache/systemds/pull/2520


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3949] Add Demo for Databricks Workspace Execution [systemds]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2520:
URL: https://github.com/apache/systemds/pull/2520#issuecomment-4811887448

   ## [Codecov](https://app.codecov.io/gh/apache/systemds/pull/2520?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 71.60%. Comparing base ([`0637bf1`](https://app.codecov.io/gh/apache/systemds/commit/0637bf15958df968c08f69771091b79db33421a2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c1b0ce6`](https://app.codecov.io/gh/apache/systemds/commit/c1b0ce6cee49a97fe6d0faa1461b30000d0e1396?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 5 commits behind head on main.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##               main    #2520      +/-   ##
   ============================================
   + Coverage     71.55%   71.60%   +0.05%     
   - Complexity    49062    49154      +92     
   ============================================
     Files          1574     1575       +1     
     Lines        189570   189793     +223     
     Branches      37188    37235      +47     
   ============================================
   + Hits         135650   135908     +258     
   + Misses        43442    43406      -36     
   - Partials      10478    10479       +1     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/systemds/pull/2520?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org