You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "David Milicevic (Jira)" <ji...@apache.org> on 2025/11/21 14:04:00 UTC
[jira] [Comment Edited] (SPARK-48456) [M1] Performance benchmark
[ https://issues.apache.org/jira/browse/SPARK-48456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18039865#comment-18039865 ]
David Milicevic edited comment on SPARK-48456 at 11/21/25 2:03 PM:
-------------------------------------------------------------------
Same as with SPARK-48457 (Testing and operational readiness), I'm marking this item as done (cleanup before Spark 4.1 release).
We did benchmarks, I'm not sure how much of those are relevant to OSS.
What's important for sure is the benchmark against equivalent code written in PySpark, for which on average we had a bit lower perf (~10%), but that was expected.
To address these issues, we have produced a set of work items related to how local variables, simple conditions, and similar are to be handled and those should address the gap.
Other than that, before we figure out a way to introduce new concepts, like for example plan caching, or similar, I don't think we need more benchmarks.
Tagging [~milan.dankovic] here as well, to fill-in if any details are missing.
[~cloud_fan] please assign this one to Milan as well, to add more details.
was (Author: JIRAUSER304692):
Same as with SPARK-48457 (Testing and operational readiness), I'm marking this item as done (cleanup before Spark 4.1 release).
We did benchmarks, I'm not sure how much of those are relevant to OSS.
What's important for sure is the benchmark against equivalent code written in PySpark, for which on average we had a bit lower perf (~10%), but that was expected.
To address these issues, we have produced a set of work items related to how local variables, simple conditions, and similar are to be handled and those should address the gap.
Other than that, before we figure out a way to introduce new concepts, like for example plan caching, or similar, I don't think we need more benchmarks.
Tagging [~milan.dankovic] here as well, to fill-in if any details are missing.
[~cloud_fan] please assign this one to Milan as well.
> [M1] Performance benchmark
> --------------------------
>
> Key: SPARK-48456
> URL: https://issues.apache.org/jira/browse/SPARK-48456
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: David Milicevic
> Priority: Major
>
> Performance parity is officially an M2 requirement, but by the end of M0 I think we should start doing some perf benchmarks to figure out where do we stand in the beginning and if we need to change something right from the start before we get to work on a more complex stuff.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org