[incubator-uniffle] 08/17: [Improvement] Check ADAPTIVE_EXECUTION_ENABLED in RssShuffleManager (#197)

[js...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

jshao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git

commit 6bdf49e1a68131545a8385123da558be287a196f
Author: xunxunmimi5577 <52...@users.noreply.github.com>
AuthorDate: Fri Jun 24 02:12:40 2022 +0800

    [Improvement] Check ADAPTIVE_EXECUTION_ENABLED in RssShuffleManager (#197)
    
    ### What changes were proposed in this pull request?
     1. Add checking of spark.sql.adaptive.enabled=false in RssShuffleManager's constructor for spark2.
     2. Add a description of this parameter in the Deploy Spark Client section of the readme.
    
    ### Why are the changes needed?
     When use firestorm+spark2+spark.sql.adaptive.enabled=true，the result is wrong，but we didn't get any hints.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Manual test
---
 README.md                                                              | 1 +
 .../src/main/java/org/apache/spark/shuffle/RssShuffleManager.java      | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/README.md b/README.md
index 50903ce..0fb65e5 100644
--- a/README.md
+++ b/README.md
@@ -149,6 +149,7 @@ rss-xxx.tgz will be generated for deployment
    ```
    spark.shuffle.manager org.apache.spark.shuffle.RssShuffleManager
    spark.rss.coordinator.quorum <coordinatorIp1>:19999,<coordinatorIp2>:19999
+   # Note: For Spark2, spark.sql.adaptive.enabled should be false because Spark2 doesn't support AQE.
    ```
 
 ### Support Spark dynamic allocation
diff --git a/client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java b/client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java
index 8a2c385..28f1a8d 100644
--- a/client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java
+++ b/client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java
@@ -136,6 +136,9 @@ public class RssShuffleManager implements ShuffleManager {
   };
 
   public RssShuffleManager(SparkConf sparkConf, boolean isDriver) {
+    if (sparkConf.getBoolean("spark.sql.adaptive.enabled", false)) {
+      throw new IllegalArgumentException("Spark2 doesn't support AQE, spark.sql.adaptive.enabled should be false.");
+    }
     this.sparkConf = sparkConf;
 
     // set & check replica config