You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/07/09 03:06:52 UTC
(spark) branch master updated: [SPARK-48837][ML] In CountVectorizer, only read binary parameter once per transform, not once per row
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ba4fb5a50ab6 [SPARK-48837][ML] In CountVectorizer, only read binary parameter once per transform, not once per row
ba4fb5a50ab6 is described below
commit ba4fb5a50ab6942a17b66cb48cac8df6e8d644ec
Author: Josh Rosen <jo...@databricks.com>
AuthorDate: Mon Jul 8 20:06:41 2024 -0700
[SPARK-48837][ML] In CountVectorizer, only read binary parameter once per transform, not once per row
### What changes were proposed in this pull request?
https://github.com/apache/spark/pull/11536 added a `binary` toggle parameter to `CountVectorizer`, but the parameter evaluation occurs inside of the vectorizer UDF itself: this causes expensive parameter reading to occur once-per-row instead of once-per-transform.
This PR addresses this issue by updating the code to only read the parameter once, similar to what was already being done for the `minTf` parameter.
### Why are the changes needed?
Address a performance issue.
I spotted this issue when I saw the stack
```scala
[...]
at scala.collection.mutable.ArrayOps$ofRef.length(ArrayOps.scala:204)
at scala.collection.IndexedSeqOptimized.exists(IndexedSeqOptimized.scala:49)
at scala.collection.IndexedSeqOptimized.exists$(IndexedSeqOptimized.scala:49)
at scala.collection.mutable.ArrayOps$ofRef.exists(ArrayOps.scala:198)
at org.apache.spark.ml.param.Params.hasParam(params.scala:701)
at org.apache.spark.ml.param.Params.hasParam$(params.scala:700)
at org.apache.spark.ml.PipelineStage.hasParam(Pipeline.scala:42)
at org.apache.spark.ml.param.Params.shouldOwn(params.scala:856)
at org.apache.spark.ml.param.Params.get(params.scala:739)
at org.apache.spark.ml.param.Params.get$(params.scala:738)
at org.apache.spark.ml.PipelineStage.get(Pipeline.scala:42)
at org.apache.spark.ml.param.Params.getOrDefault(params.scala:759)
at org.apache.spark.ml.param.Params.getOrDefault$(params.scala:757)
at org.apache.spark.ml.PipelineStage.getOrDefault(Pipeline.scala:42)
at org.apache.spark.ml.param.Params.$(params.scala:766)
at org.apache.spark.ml.param.Params.$$(params.scala:766)
at org.apache.spark.ml.PipelineStage.$(Pipeline.scala:42)
at org.apache.spark.ml.feature.CountVectorizerModel.$anonfun$transform$1(CountVectorizer.scala:326)
at org.apache.spark.ml.feature.CountVectorizerModel$$Lambda$12153/1200761496.apply(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.sort_addToSorter_0$(Unknown Source)
[...]
```
while investigating an unrelated issue.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing unit tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47258 from JoshRosen/CountVectorizer-conf.
Authored-by: Josh Rosen <jo...@databricks.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala b/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
index c02b43ceed5e..890266ed7a72 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
@@ -310,7 +310,10 @@ class CountVectorizerModel(
broadcastDict = Some(dataset.sparkSession.sparkContext.broadcast(dict))
}
val dictBr = broadcastDict.get
+ // SPARK-48837: capture parameter values here so that we only evaulate once-per-transform
+ // rather than once-per-row:
val minTf = $(minTF)
+ val isBinary = $(binary)
val vectorizer = udf { document: Seq[String] =>
val termCounts = new OpenHashMap[Int, Double]
var tokenCount = 0L
@@ -322,7 +325,7 @@ class CountVectorizerModel(
tokenCount += 1
}
val effectiveMinTF = if (minTf >= 1.0) minTf else tokenCount * minTf
- val effectiveCounts = if ($(binary)) {
+ val effectiveCounts = if (isBinary) {
termCounts.filter(_._2 >= effectiveMinTF).map(p => (p._1, 1.0)).toSeq
} else {
termCounts.filter(_._2 >= effectiveMinTF).toSeq
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org