You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/07/09 03:06:52 UTC

(spark) branch master updated: [SPARK-48837][ML] In CountVectorizer, only read binary parameter once per transform, not once per row

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new ba4fb5a50ab6 [SPARK-48837][ML] In CountVectorizer, only read binary parameter once per transform, not once per row
ba4fb5a50ab6 is described below

commit ba4fb5a50ab6942a17b66cb48cac8df6e8d644ec
Author: Josh Rosen <jo...@databricks.com>
AuthorDate: Mon Jul 8 20:06:41 2024 -0700

    [SPARK-48837][ML] In CountVectorizer, only read binary parameter once per transform, not once per row
    
    ### What changes were proposed in this pull request?
    
    https://github.com/apache/spark/pull/11536 added a `binary` toggle parameter to `CountVectorizer`, but the parameter evaluation occurs inside of the vectorizer UDF itself: this causes expensive parameter reading to occur once-per-row instead of once-per-transform.
    
    This PR addresses this issue by updating the code to only read the parameter once, similar to what was already being done for the `minTf` parameter.
    
    ### Why are the changes needed?
    
    Address a performance issue.
    
    I spotted this issue when I saw the stack
    
    ```scala
    [...]
    at scala.collection.mutable.ArrayOps$ofRef.length(ArrayOps.scala:204)
    at scala.collection.IndexedSeqOptimized.exists(IndexedSeqOptimized.scala:49)
    at scala.collection.IndexedSeqOptimized.exists$(IndexedSeqOptimized.scala:49)
    at scala.collection.mutable.ArrayOps$ofRef.exists(ArrayOps.scala:198)
    at org.apache.spark.ml.param.Params.hasParam(params.scala:701)
    at org.apache.spark.ml.param.Params.hasParam$(params.scala:700)
    at org.apache.spark.ml.PipelineStage.hasParam(Pipeline.scala:42)
    at org.apache.spark.ml.param.Params.shouldOwn(params.scala:856)
    at org.apache.spark.ml.param.Params.get(params.scala:739)
    at org.apache.spark.ml.param.Params.get$(params.scala:738)
    at org.apache.spark.ml.PipelineStage.get(Pipeline.scala:42)
    at org.apache.spark.ml.param.Params.getOrDefault(params.scala:759)
    at org.apache.spark.ml.param.Params.getOrDefault$(params.scala:757)
    at org.apache.spark.ml.PipelineStage.getOrDefault(Pipeline.scala:42)
    at org.apache.spark.ml.param.Params.$(params.scala:766)
    at org.apache.spark.ml.param.Params.$$(params.scala:766)
    at org.apache.spark.ml.PipelineStage.$(Pipeline.scala:42)
    at org.apache.spark.ml.feature.CountVectorizerModel.$anonfun$transform$1(CountVectorizer.scala:326)
    at org.apache.spark.ml.feature.CountVectorizerModel$$Lambda$12153/1200761496.apply(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.sort_addToSorter_0$(Unknown Source)
    [...]
    ```
    
    while investigating an unrelated issue.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Existing unit tests.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #47258 from JoshRosen/CountVectorizer-conf.
    
    Authored-by: Josh Rosen <jo...@databricks.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala b/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
index c02b43ceed5e..890266ed7a72 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
@@ -310,7 +310,10 @@ class CountVectorizerModel(
       broadcastDict = Some(dataset.sparkSession.sparkContext.broadcast(dict))
     }
     val dictBr = broadcastDict.get
+    // SPARK-48837: capture parameter values here so that we only evaulate once-per-transform
+    // rather than once-per-row:
     val minTf = $(minTF)
+    val isBinary = $(binary)
     val vectorizer = udf { document: Seq[String] =>
       val termCounts = new OpenHashMap[Int, Double]
       var tokenCount = 0L
@@ -322,7 +325,7 @@ class CountVectorizerModel(
         tokenCount += 1
       }
       val effectiveMinTF = if (minTf >= 1.0) minTf else tokenCount * minTf
-      val effectiveCounts = if ($(binary)) {
+      val effectiveCounts = if (isBinary) {
         termCounts.filter(_._2 >= effectiveMinTF).map(p => (p._1, 1.0)).toSeq
       } else {
         termCounts.filter(_._2 >= effectiveMinTF).toSeq


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org