You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/01 11:34:30 UTC
[GitHub] [spark] wangyum opened a new pull request, #37048: [SPARK-39655][SQL] Add a config to limit CartesianProductExec's partition number
wangyum opened a new pull request, #37048:
URL: https://github.com/apache/spark/pull/37048
### What changes were proposed in this pull request?
This PR adds a config to limit CartesianProductExec's partition number.
The default value is: `Int.MaxValue`.The recommended value is: `1200000`.
### Why are the changes needed?
A large number of partitions can cause driver instability, especially for Spark Hive thriftserver. It affects other users. A common case is that the join condition is incorrectly, such as:
```sql
SELECT * FROM t1 JOIN t2 ON t1.id = t1.id;
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #37048: [SPARK-39655][CORE] Add a config to limit the number of RDD partitions
Posted by GitBox <gi...@apache.org>.
wangyum commented on PR #37048:
URL: https://github.com/apache/spark/pull/37048#issuecomment-1180460255
> OK, but your approach doesn't fix all the issues. With AQE, a complicated query may have run for a while and the cartesian product fails in the last query stage.
Yes. This is mainly to prevent a bad query affecting the stability of the thrift server. Another case is: `a join b on a.id = b.id = a.name = b.name`, https://github.com/apache/spark/pull/29728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #37048: [SPARK-39655][SQL] Add a config to limit CartesianProductExec's partition number
Posted by GitBox <gi...@apache.org>.
wangyum commented on PR #37048:
URL: https://github.com/apache/spark/pull/37048#issuecomment-1172254561
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #37048: [SPARK-39655][SQL] Add a config to limit CartesianProductExec's partition number
Posted by GitBox <gi...@apache.org>.
wangyum commented on PR #37048:
URL: https://github.com/apache/spark/pull/37048#issuecomment-1179999172
> This reminds me of `CheckCartesianProducts`. In general, it's a bit hard to predict bad queries and fail earlier. Will #37014 solve your issue?
That can not solve the case where the input is very large and can't coalesce shuffle partitions. For example:
```sql
select * from large_table1(10000 partitions) t1 join large_table2(10000 partitions) t2 on t1.id = t1.id;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #37048: [SPARK-39655][SQL] Add a config to limit CartesianProductExec's partition number
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #37048:
URL: https://github.com/apache/spark/pull/37048#issuecomment-1180014906
OK, but your approach doesn't fix all the issues. With AQE, a complicated query may have run for a while and the cartesian product fails in the last query stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #37048: [SPARK-39655][SQL] Add a config to limit CartesianProductExec's partition number
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #37048:
URL: https://github.com/apache/spark/pull/37048#issuecomment-1178653885
This reminds me of `CheckCartesianProducts`. In general, it's a bit hard to predict bad queries and fail earlier. Will https://github.com/apache/spark/pull/37014 solve your issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #37048: [SPARK-39655][CORE] Add a config to limit the number of RDD partitions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #37048:
URL: https://github.com/apache/spark/pull/37048#issuecomment-1181485961
> This is mainly to prevent a bad query affecting the stability of the thrift server.
I think we need better resource isolation here. Cartesian product is not the only place that can generate a super big Spark job with tons of tasks. Shall we have a mechanism in Spark Core to limit the task slots used by one thriftserver user?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #37048: [SPARK-39655][CORE] Add a config to limit the number of RDD partitions
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #37048: [SPARK-39655][CORE] Add a config to limit the number of RDD partitions
URL: https://github.com/apache/spark/pull/37048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #37048: [SPARK-39655][CORE] Add a config to limit the number of RDD partitions
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #37048:
URL: https://github.com/apache/spark/pull/37048#issuecomment-1292806164
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org