You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/06 17:56:05 UTC

[I] Performance snapshot: Ballista vs Comet vs Spark @ SF100 — join strategy (SMJ vs ShuffledHashJoin) [datafusion-ballista]

andygrove opened a new issue, #1959:
URL: https://github.com/apache/datafusion-ballista/issues/1959

   Follow-up to #1944, adding the dimension that turns out to explain most of the Ballista↔Comet gap: **join strategy (SortMergeJoin vs ShuffledHashJoin)**. #1944 compared the three engines at their *defaults* — where Ballista uses SMJ and Comet uses SHJ — so it was partly comparing plans, not engines. This snapshot runs **both** strategies on **all three** engines.
   
   > Informational snapshot, not a bug report. Supersedes #1944.
   
   ## Setup
   - **Cluster:** 2-node k3s, each 32 cores / 128 GiB. **2 executors × 8 cores (16 task slots)**, sequential runs (idle engines scaled to 0 — no I/O contention).
   - **Data:** TPC-H **SF100** Parquet in MinIO (S3); `s3://` for Ballista, `s3a://` for Spark/Comet. Identical 22-query set; all row counts match.
   - **Ballista:** `main` + in-flight PRs (shuffle in-flight governor #1951, broadcast-join promotion, exchange reuse #1953). DataFusion 54. **Static planner.** `target_partitions=32`.
   - **Comet:** 1.0.0-SNAPSHOT on Spark 3.5. **Vanilla Spark:** 3.5 with Comet disabled (`spark.comet.enabled=false`, default sort shuffle manager).
   - **Join-strategy toggles:** Ballista `datafusion.optimizer.prefer_hash_join`; Comet `spark.comet.exec.replaceSortMergeJoin`; Spark `spark.sql.join.preferSortMergeJoin`.
   - **Method:** single iteration, no warmup.
   
   ## Totals — the headline
   
   | Engine | SortMergeJoin | ShuffledHashJoin | Δ from hash join |
   |---|---:|---:|---:|
   | **Ballista** | 560 s | **398 s** | **−29 %** |
   | **Comet** | 356 s | 347 s | −3 % |
   | **Vanilla Spark** | 586 s | 590 s | +1 % |
   
   **Matched strategy:**
   
   | | Ballista | Comet | Vanilla Spark |
   |---|---:|---:|---:|
   | SortMergeJoin | 560 | 356 | 586 |
   | ShuffledHashJoin | **398** | **347** | 590 |
   
   ## Observations
   
   1. **Enabling hash join is a −29 % win for Ballista, but a no-op for Comet (−3 %) and Spark (+1 %).** Ballista's SortMergeJoin sorts both sides of every large join (~963 M rows — ~1.6× the fact table — for Q9 alone); those sorts dominate its time. Comet's columnar sorts and Spark's row-based sorts are comparatively cheap, so the strategy toggle barely moves them.
   2. **At matched strategy, Ballista is within ~15 % of Comet** (398 vs 347 s, 1.15×) and **~1.5× faster than vanilla Spark** (398 vs 590 s). Most of the headline Ballista-vs-Comet gap reported in #1944 was Ballista running its SMJ default against Comet's SHJ default — a config difference, not an engine difference.
   3. **Ballista SHJ ran all 22 queries at SF100 with no OOM** (2×8, 28 GB pool). `prefer_hash_join=false` is a conservative default because DataFusion's hash join cannot spill (#1648); at this scale that default costs ~29 %.
   4. The residual matched-SHJ gap to Comet is **query-dependent**: Ballista is *faster* on several (Q1 7.1 vs 10.4, Q4, Q8, Q17, Q22) and Comet is 2×+ on a few (**Q7, Q14, Q19**). Those outliers — not the average — are where the remaining execution/shuffle difference lives. Attributing them needs shuffle-read metrics that Ballista does not yet expose (#1958).
   5. **Vanilla Spark is the slowest of the three** regardless of strategy (590–586 s); both Ballista and Comet are comfortably ahead.
   
   ## Per-query (seconds)
   
   | Q | Bal SMJ | Bal SHJ | Comet SMJ | Comet SHJ | Spark SMJ | Spark SHJ |
   |---|---:|---:|---:|---:|---:|---:|
   | Q1 | 6.6 | 7.1 | 11.0 | 10.4 | 60.6 | 60.0 |
   | Q2 | 10.1 | 7.8 | 8.6 | 6.9 | 10.8 | 10.2 |
   | Q3 | 17.7 | 13.3 | 14.9 | 13.5 | 23.9 | 23.2 |
   | Q4 | 12.1 | 6.4 | 7.5 | 7.7 | 17.0 | 17.2 |
   | Q5 | 45.0 | 30.4 | 28.6 | 24.1 | 37.2 | 33.7 |
   | Q6 | 6.2 | 6.0 | 3.7 | 3.7 | 11.1 | 11.2 |
   | Q7 | 43.8 | 32.5 | 17.2 | 15.7 | 24.7 | 25.3 |
   | Q8 | 53.8 | 30.3 | 32.3 | 32.8 | 32.3 | 32.2 |
   | Q9 | 77.1 | 45.7 | 45.5 | 42.7 | 56.0 | 57.8 |
   | Q10 | 17.2 | 14.8 | 14.8 | 13.6 | 22.3 | 22.1 |
   | Q11 | 8.3 | 7.3 | 7.7 | 7.1 | 10.2 | 10.1 |
   | Q12 | 10.5 | 8.3 | 8.8 | 8.0 | 15.9 | 15.9 |
   | Q13 | 8.4 | 7.7 | 7.4 | 7.1 | 14.1 | 16.9 |
   | Q14 | 14.9 | 14.8 | 7.1 | 6.5 | 12.6 | 12.6 |
   | Q15 | 11.1 | 11.2 | 11.3 | 11.2 | 25.9 | 26.9 |
   | Q16 | 3.3 | 3.0 | 3.2 | 3.2 | 5.5 | 5.5 |
   | Q17 | 48.6 | 35.4 | 38.4 | 39.3 | 60.7 | 60.4 |
   | Q18 | 48.4 | 28.8 | 23.6 | 24.7 | 47.0 | 46.6 |
   | Q19 | 16.9 | 17.3 | 8.0 | 8.0 | 15.1 | 15.0 |
   | Q20 | 17.6 | 15.7 | 9.6 | 9.5 | 16.7 | 16.5 |
   | Q21 | 77.9 | 50.9 | 42.5 | 47.2 | 58.2 | 62.1 |
   | Q22 | 4.4 | 3.5 | 4.3 | 3.9 | 8.2 | 8.8 |
   | **Total** | **560** | **398** | **356** | **347** | **586** | **590** |
   
   ## Caveats
   Single iteration on a shared homelab cluster carries ~5–10 % run-to-run noise (disk/warm-up effects). The −29 % Ballista hash-join effect and the matched-strategy ordering are well above that; the per-query 2× outliers (Q7/Q14/Q19) deserve multi-iteration confirmation before drawing conclusions from them.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Performance snapshot: Ballista vs Comet vs Spark @ SF100 — join strategy (SMJ vs ShuffledHashJoin) [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1959:
URL: https://github.com/apache/datafusion-ballista/issues/1959#issuecomment-4899884190

   > Any idea why such huge difference in SMJ between comet and ballista?
   
   Seems like 32 partitions in Ballista was too low. I'm running more benchmarks with higher partition count.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Performance snapshot: Ballista vs Comet vs Spark @ SF100 — join strategy (SMJ vs ShuffledHashJoin) [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1959:
URL: https://github.com/apache/datafusion-ballista/issues/1959#issuecomment-4900048129

   ## Addendum — `target_partitions` sensitivity (why SMJ looked slower)
   
   The matrix above ran Ballista at `target_partitions=32`. Sweeping it shows that a good chunk of the SMJ gap was **under-partitioning**, and that SMJ and SHJ respond **oppositely**:
   
   | Ballista total (s) | p=32 | p=64 | p=96 |
   |---|---:|---:|---:|
   | **SortMergeJoin** | 560 | **466** | 528 |
   | **ShuffledHashJoin** | **398** | 413 | 492 |
   
   - **SMJ benefits from more partitions** (32 → 64 = **−17 %**). Sort-merge joins sort both sides, and sorting is superlinear, so smaller partitions mean much smaller sorts. The sweet spot is ~64 (≈4× the 16 task slots); 96 is past the overhead knee.
   - **SHJ does not** (32 is best). Hash joins don't sort, so shrinking partitions doesn't reduce a superlinear cost — more partitions just add shuffle-file / scheduling overhead.
   
   **Fair (best-partition) engine comparison:**
   
   | | Ballista (best) | Comet | ratio |
   |---|---:|---:|---:|
   | SMJ | 466 (p=64) | 356 | **1.31×** (was 1.57× at p=32) |
   | SHJ | 398 (p=32) | 347 | 1.15× |
   
   Per-query, the sort-heavy joins improve most going 32 → 64: **Q9 77→55, Q21 78→57, Q18 48→33, Q8 54→39, Q5 45→33 s.**
   
   ### Takeaways
   1. Ballista's best config here is still **ShuffledHashJoin (398 s, within ~15 % of Comet)** — enabling hash join beats tuning SMJ partition count.
   2. The rule-of-thumb "**`target_partitions` = total task slots**" (32 here) is **too low for SortMergeJoin** — SMJ wants ≈4× slots. For SHJ, matching slots is fine.
   3. The residual after tuning — SMJ 1.31× / SHJ 1.15× vs Comet — is the real engine / shuffle-implementation difference, concentrated in a few outliers (**Q7 stays ~2× Comet even at good partitions**). Attributing that needs the shuffle-read metrics tracked in #1958.
   
   _(Single iteration, ~5–10 % run-to-run noise; the −17 % SMJ partition effect and the SMJ-vs-SHJ opposite response are well above it.)_
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Performance snapshot: Ballista vs Comet vs Spark @ SF100 — join strategy (SMJ vs ShuffledHashJoin) [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1959:
URL: https://github.com/apache/datafusion-ballista/issues/1959#issuecomment-4947072784

   I'm going to create a fresh issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Performance snapshot: Ballista vs Comet vs Spark @ SF100 — join strategy (SMJ vs ShuffledHashJoin) [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1959:
URL: https://github.com/apache/datafusion-ballista/issues/1959#issuecomment-4897555064

   Any idea why such huge difference in SMJ between comet and ballista?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Performance snapshot: Ballista vs Comet vs Spark @ SF100 — join strategy (SMJ vs ShuffledHashJoin) [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1959: Performance snapshot: Ballista vs Comet vs Spark @ SF100 — join strategy (SMJ vs ShuffledHashJoin)
URL: https://github.com/apache/datafusion-ballista/issues/1959


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org