[spark] branch master updated: [SPARK-39675][SQL] Switch 'spark.sql.codegen.factoryMode' configuration from testing purpose to internal purpose

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 3853ad45376 [SPARK-39675][SQL] Switch 'spark.sql.codegen.factoryMode' configuration from testing purpose to internal purpose
3853ad45376 is described below

commit 3853ad45376629447b1c9dd92953dc0c3e103b52
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Jul 5 12:47:40 2022 +0900

    [SPARK-39675][SQL] Switch 'spark.sql.codegen.factoryMode' configuration from testing purpose to internal purpose
    
    ### What changes were proposed in this pull request?
    
    This PR proposes to switch 'spark.sql.codegen.factoryMode' configuration from testing purpose to internal purpose by removing `Utils.isTesting`.
    
    In addition, it changes the documentation in 'spark.sql.codegen.factoryMode' to say that this is only for internal purpose, and NOT supposed to be set by end users.
    
    ### Why are the changes needed?
    
    We can actually leverage this configuration in other testing beds that mimic production environment to make sure the fallback codegen works in production too.
    
    Another point is about benchmarking in the production enviornment.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Manual testing.
    
    Closes #37076 from HyukjinKwon/SPARK-39675.
    
    Authored-by: Hyukjin Kwon <gu...@apache.org>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 .../catalyst/expressions/CodeGeneratorWithInterpretedFallback.scala  | 5 ++---
 .../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala       | 3 ++-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallback.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallback.scala
index 3b7219477bd..0509b852cfd 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallback.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallback.scala
@@ -21,7 +21,6 @@ import scala.util.control.NonFatal
 
 import org.apache.spark.internal.Logging
 import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.util.Utils
 
 /**
  * Defines values for `SQLConf` config of fallback mode. Use for test only.
@@ -43,9 +42,9 @@ abstract class CodeGeneratorWithInterpretedFallback[IN, OUT] extends Logging {
     val fallbackMode = CodegenObjectFactoryMode.withName(config)
 
     fallbackMode match {
-      case CodegenObjectFactoryMode.CODEGEN_ONLY if Utils.isTesting =>
+      case CodegenObjectFactoryMode.CODEGEN_ONLY =>
         createCodeGeneratedObject(in)
-      case CodegenObjectFactoryMode.NO_CODEGEN if Utils.isTesting =>
+      case CodegenObjectFactoryMode.NO_CODEGEN =>
         createInterpretedObject(in)
       case _ =>
         try {
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index f0ecbd0f9ba..1b7857ead59 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -1548,7 +1548,8 @@ object SQLConf {
       "during tests. `FALLBACK` means trying codegen first and then falling back to " +
       "interpreted if any compile error happens. Disabling fallback if `CODEGEN_ONLY`. " +
       "`NO_CODEGEN` skips codegen and goes interpreted path always. Note that " +
-      "this config works only for tests.")
+      "this configuration is only for the internal usage, and NOT supposed to be set by " +
+      "end users.")
     .version("2.4.0")
     .internal()
     .stringConf


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org