[GitHub] [spark] dongjoon-hyun opened a new pull request #24339: [SPARK-27436][SQL] Add spark.sql.optimizer.nonExcludedRules

[GitBox <gi...@apache.org>]

dongjoon-hyun opened a new pull request #24339: [SPARK-27436][SQL] Add spark.sql.optimizer.nonExcludedRules
URL: https://github.com/apache/spark/pull/24339
 
 
   ## What changes were proposed in this pull request?
   
   Currently, users can remove optimizer rule dynamically. This PR aims to add a new feature with `spark.sql.optimizer.nonExcludedRules` static configuration to prevent accidental rule exclusion by users in SQL environment dynamically. This is useful for admin to protect custom optimizer rules.
   
   **AS IS**
   ```sql
   $ bin/spark-sql \
   --conf spark.sql.optimizer.planChangeLog.level=ERROR \
   --conf spark.sql.optimizer.planChangeLog.batches=OptimizeLimitZero \
   --conf spark.sql.optimizer.planChangeLog.rules=org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero
   Spark master: local[*], Application Id: local-1554952359955
   spark-sql> select 1 limit 0;
   19/04/10 20:12:46 ERROR BaseSessionStateBuilder$$anon$2:
   === Applying Rule org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero ===
   !GlobalLimit 0              LocalRelation <empty>, [1#0]
   !+- LocalLimit 0
   !   +- Project [1 AS 1#0]
   !      +- OneRowRelation
   
   19/04/10 20:12:46 ERROR BaseSessionStateBuilder$$anon$2:
   === Result of Batch OptimizeLimitZero ===
   !GlobalLimit 0              LocalRelation <empty>, [1#0]
   !+- LocalLimit 0
   !   +- Project [1 AS 1#0]
   !      +- OneRowRelation
   
   Time taken: 1.426 seconds
   spark-sql> set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero;
   19/04/10 20:12:48 ERROR BaseSessionStateBuilder$$anon$2: Batch OptimizeLimitZero has no effect.
   spark.sql.optimizer.excludedRules	org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero
   Time taken: 0.26 seconds, Fetched 1 row(s)
   spark-sql> select 1 limit 0;
   Time taken: 0.226 seconds
   ```
   
   **AFTER (with spark.sql.optimizer.nonExcludedRules)**
   ```sql
   $ bin/spark-shell \
   --conf spark.sql.optimizer.planChangeLog.level=ERROR \
   --conf spark.sql.optimizer.planChangeLog.batches=OptimizeLimitZero \
   --conf spark.sql.optimizer.planChangeLog.rules=org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero \
     --conf spark.sql.optimizer.nonExcludedRules=org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero
   Spark master: local[*], Application Id: local-1554952378462
   spark-sql> set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero;
   19/04/10 20:13:04 ERROR BaseSessionStateBuilder$$anon$2: Batch OptimizeLimitZero has no effect.
   19/04/10 20:13:04 ERROR BaseSessionStateBuilder$$anon$2: Batch OptimizeLimitZero has no effect.
   spark.sql.optimizer.excludedRules	org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero
   Time taken: 1.563 seconds, Fetched 1 row(s)
   spark-sql> select 1 limit 0;
   19/04/10 20:13:07 ERROR BaseSessionStateBuilder$$anon$2:
   === Applying Rule org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero ===
   !GlobalLimit 0              LocalRelation <empty>, [1#4]
   !+- LocalLimit 0
   !   +- Project [1 AS 1#4]
   !      +- OneRowRelation
   
   19/04/10 20:13:07 ERROR BaseSessionStateBuilder$$anon$2:
   === Result of Batch OptimizeLimitZero ===
   !GlobalLimit 0              LocalRelation <empty>, [1#4]
   !+- LocalLimit 0
   !   +- Project [1 AS 1#4]
   !      +- OneRowRelation
   ```
   
   ## How was this patch tested?
   
   Pass the Jenkins with a newly added test case.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org