You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/03 02:10:38 UTC
[GitHub] [spark] beliefer commented on a diff in pull request #36405: [SPARK-39065][SQL] DS V2 Limit push-down should avoid out of memory
beliefer commented on code in PR #36405:
URL: https://github.com/apache/spark/pull/36405#discussion_r863326711
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCScanBuilder.scala:
##########
@@ -131,7 +131,8 @@ case class JDBCScanBuilder(
}
override def pushLimit(limit: Int): Boolean = {
- if (jdbcOptions.pushDownLimit) {
+ if (jdbcOptions.pushDownLimit &&
+ JdbcDialects.get(jdbcOptions.url).canPushDownLimit(jdbcOptions, limit)) {
Review Comment:
This option just used to solve out-of-memory when `Limit` could be pushed down. Because all the data will pulled by one Task when pushing down `Limit`. If Limit cannot be pushed down, Spark could uses jdbc partition or other method to avoid the problem.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCScanBuilder.scala:
##########
@@ -131,7 +131,8 @@ case class JDBCScanBuilder(
}
override def pushLimit(limit: Int): Boolean = {
- if (jdbcOptions.pushDownLimit) {
+ if (jdbcOptions.pushDownLimit &&
+ JdbcDialects.get(jdbcOptions.url).canPushDownLimit(jdbcOptions, limit)) {
Review Comment:
This option just used to solve out-of-memory when `Limit` could be pushed down. Because all the data will pulled by one Task when pushing down `Limit`. If `Limit` cannot be pushed down, Spark could uses jdbc partition or other method to avoid the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org