You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean R. Owen (Jira)" <ji...@apache.org> on 2022/10/12 17:50:00 UTC
[jira] [Resolved] (SPARK-40499) Spark 3.2.1 percentlie_approx query much slower than Spark 2.4.0
[ https://issues.apache.org/jira/browse/SPARK-40499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean R. Owen resolved SPARK-40499.
----------------------------------
Resolution: Invalid
It's hard to action this; this sounds like a question for the mailing list
> Spark 3.2.1 percentlie_approx query much slower than Spark 2.4.0
> ----------------------------------------------------------------
>
> Key: SPARK-40499
> URL: https://issues.apache.org/jira/browse/SPARK-40499
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 3.2.1
> Environment: hadoop: 3.0.0
> spark: 2.4.0 / 3.2.1
> shuffle：spark 2.4.0
> Reporter: xuanzhiang
> Priority: Major
> Attachments: spark2.4-shuffle-data.png, spark3.2-shuffle-data.png
>
>
> spark.sql(
> s"""
> |SELECT
> | Info ,
> | PERCENTILE_APPROX(cost,0.5) cost_p50,
> | PERCENTILE_APPROX(cost,0.9) cost_p90,
> | PERCENTILE_APPROX(cost,0.95) cost_p95,
> | PERCENTILE_APPROX(cost,0.99) cost_p99,
> | PERCENTILE_APPROX(cost,0.999) cost_p999
> |FROM
> | textData
> |""".stripMargin)
> * When we used spark 2.4.0, aggregation adopted objHashAggregator, stage 2 pull shuffle data very quick . but , when we use spark 3.2.1 and use old shuffle , 140M shuffle data cost 3 hours.
> * If we upgrade the Shuffle, will we get performance regression？
> *
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org