You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/04 03:30:58 UTC
[GitHub] [spark] wangyum opened a new pull request, #37069: [SPARK-39667][SQL] Add another workaround when there is not enough memory to build and broadcast the table
wangyum opened a new pull request, #37069:
URL: https://github.com/apache/spark/pull/37069
### What changes were proposed in this pull request?
This PR adds `ANALYZE TABLE tbl COMPUTE STATISTICS;` as a workaround when there is not enough memory to build and broadcast the table.
### Why are the changes needed?
The current workaround has the following disadvantages:
1. Setting `spark.sql.autoBroadcastJoinThreshold` to -1 will disable all broadcast joins.
2. We don't know the specific value of `spark.driver.memory` should be set.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual testing:
```
Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting spark.sql.autoBroadcastJoinThreshold to -1 or increase the spark driver memory by setting spark.driver.memory to a higher value or analyze these tables through: ANALYZE TABLE `tpds5t`.`item` COMPUTE STATISTICS;."
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #37069: [SPARK-39667][SQL] Add another workaround when there is not enough memory to build and broadcast the table
Posted by GitBox <gi...@apache.org>.
wangyum commented on PR #37069:
URL: https://github.com/apache/spark/pull/37069#issuecomment-1180466553
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37069: [SPARK-39667][SQL] Add another workaround when there is not enough memory to build and broadcast the table
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37069:
URL: https://github.com/apache/spark/pull/37069#discussion_r913420730
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala:
##########
@@ -179,8 +179,9 @@ case class BroadcastExchangeExec(
// SparkFatalException, which is a subclass of Exception. ThreadUtils.awaitResult
// will catch this exception and re-throw the wrapped fatal throwable.
case oe: OutOfMemoryError =>
+ val tables = child.collect { case f: FileSourceScanExec => f.tableIdentifier }.flatten
Review Comment:
So we can't give similar suggestions for the V2 table? Need more refactor work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum closed pull request #37069: [SPARK-39667][SQL] Add another workaround when there is not enough memory to build and broadcast the table
Posted by GitBox <gi...@apache.org>.
wangyum closed pull request #37069: [SPARK-39667][SQL] Add another workaround when there is not enough memory to build and broadcast the table
URL: https://github.com/apache/spark/pull/37069
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37069: [SPARK-39667][SQL] Add another workaround when there is not enough memory to build and broadcast the table
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37069:
URL: https://github.com/apache/spark/pull/37069#discussion_r916526901
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala:
##########
@@ -179,8 +179,9 @@ case class BroadcastExchangeExec(
// SparkFatalException, which is a subclass of Exception. ThreadUtils.awaitResult
// will catch this exception and re-throw the wrapped fatal throwable.
case oe: OutOfMemoryError =>
+ val tables = child.collect { case f: FileSourceScanExec => f.tableIdentifier }.flatten
Review Comment:
no, ANALYZE TABLE only works for file source tables I think. And file source v2 is not production-ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #37069: [SPARK-39667][SQL] Add another workaround when there is not enough memory to build and broadcast the table
Posted by GitBox <gi...@apache.org>.
wangyum commented on PR #37069:
URL: https://github.com/apache/spark/pull/37069#issuecomment-1173300088
cc @HyukjinKwon @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #37069: [SPARK-39667][SQL] Add another workaround when there is not enough memory to build and broadcast the table
Posted by GitBox <gi...@apache.org>.
wangyum commented on code in PR #37069:
URL: https://github.com/apache/spark/pull/37069#discussion_r913471423
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala:
##########
@@ -179,8 +179,9 @@ case class BroadcastExchangeExec(
// SparkFatalException, which is a subclass of Exception. ThreadUtils.awaitResult
// will catch this exception and re-throw the wrapped fatal throwable.
case oe: OutOfMemoryError =>
+ val tables = child.collect { case f: FileSourceScanExec => f.tableIdentifier }.flatten
Review Comment:
Does we fully support analyze v2 tables? For example: analyze delta tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37069: [SPARK-39667][SQL] Add another workaround when there is not enough memory to build and broadcast the table
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37069:
URL: https://github.com/apache/spark/pull/37069#discussion_r916527812
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala:
##########
@@ -179,8 +179,9 @@ case class BroadcastExchangeExec(
// SparkFatalException, which is a subclass of Exception. ThreadUtils.awaitResult
// will catch this exception and re-throw the wrapped fatal throwable.
case oe: OutOfMemoryError =>
+ val tables = child.collect { case f: FileSourceScanExec => f.tableIdentifier }.flatten
Review Comment:
Got it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org