[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36405: [SPARK-39065][SQL] DS V2 Limit push-down should avoid out of memory

[GitBox <gi...@apache.org>]

HyukjinKwon commented on code in PR #36405:
URL: https://github.com/apache/spark/pull/36405#discussion_r862561548


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCScanBuilder.scala:
##########
@@ -131,7 +131,8 @@ case class JDBCScanBuilder(
   }
 
   override def pushLimit(limit: Int): Boolean = {
-    if (jdbcOptions.pushDownLimit) {
+    if (jdbcOptions.pushDownLimit &&
+      JdbcDialects.get(jdbcOptions.url).canPushDownLimit(jdbcOptions, limit)) {

Review Comment:
   It will still throw an out-of-memory exception anyway if 1. the limit is very large, or 2. no limit is pushed down (as it will collect all anyway). How does this option solve the problem?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org