You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mpetruska <gi...@git.apache.org> on 2017/11/04 13:30:58 UTC
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
GitHub user mpetruska opened a pull request:
https://github.com/apache/spark/pull/19659
[SPARK-19668][ML] Multiple NGram sizes
## What changes were proposed in this pull request?
[Jira ticket](https://issues.apache.org/jira/browse/SPARK-19668):
- implements extraction of multiple sizes of ngrams with `feature.NGram`
## How was this patch tested?
- unit tests were added for the implementation (`multiSliding` function)
- test cases were added to `NGramSuite` and `Word2VecSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mpetruska/spark SPARK-19668
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19659.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19659
----
commit 11f26f40b5f3d7325b6da3f7b32b0fd4db0449ee
Author: Mark Petruska <pe...@gmail.com>
Date: 2017-11-04T11:34:39Z
implements NGram over a range of values
commit 74fee5531a605fcc876dd7864713ba28e0d4ad71
Author: Mark Petruska <pe...@gmail.com>
Date: 2017-11-04T13:04:17Z
adds tests for multiSliding
commit 0364a56ef6d71939a4db5d81f8ce0e68c1ecd546
Author: Mark Petruska <pe...@gmail.com>
Date: 2017-11-04T13:17:57Z
adds additional tests for multi-length ngram interactions
commit 96e2a6267002d263927ecb99f5d46642a5c4df4d
Author: Mark Petruska <pe...@gmail.com>
Date: 2017-11-04T13:18:18Z
adds maxN parameter to NGram examples
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/19659
Let me CC @MLnick as well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r149480386
--- Diff: examples/src/main/python/ml/n_gram_example.py ---
@@ -35,7 +35,7 @@
(2, ["Logistic", "regression", "models", "are", "neat"])
], ["id", "words"])
- ngram = NGram(n=2, inputCol="words", outputCol="ngrams")
+ ngram = NGram(n=2, maxN=4, inputCol="words", outputCol="ngrams")
--- End diff --
This generally isn't enough to expose the new functionality in Python. Take a look in `feature.py` & also consider adding some Python tests.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r149696337
--- Diff: mllib/src/main/scala/org/apache/spark/ml/extensions/seq/package.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.extensions
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+
+import org.apache.spark.annotation.Since
+
+package object seq {
+
+ /**
+ * Calculates sliding windows over multiple length parameters simultaneously.
+ * @param x the input sequence over which sliding windows will be collected
+ * @param min the inclusive minimal length of the window to be collected
+ * @param max the inclusive maximal length of the window to be collected
+ * @return the collected windows
+ *
+ * @example {{{
+ * multiSliding(1 to 5, min = 2, max = 4) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5),
+ * Seq(3, 4), Seq(3, 4, 5),
+ * Seq(4, 5)
+ * )
+ *
+ * multiSliding(1 to 10, min = 2, max = 5) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4), Seq(1, 2, 3, 4, 5),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5), Seq(2, 3, 4, 5, 6),
+ * Seq(3, 4), Seq(3, 4, 5), Seq(3, 4, 5, 6), Seq(3, 4, 5, 6, 7),
+ * Seq(4, 5), Seq(4, 5, 6), Seq(4, 5, 6, 7), Seq(4, 5, 6, 7, 8),
+ * Seq(5, 6), Seq(5, 6, 7), Seq(5, 6, 7, 8), Seq(5, 6, 7, 8, 9),
+ * Seq(6, 7), Seq(6, 7, 8), Seq(6, 7, 8, 9), Seq(6, 7, 8, 9, 10),
+ * Seq(7, 8), Seq(7, 8, 9), Seq(7, 8, 9, 10),
+ * Seq(8, 9), Seq(8, 9, 10),
+ * Seq(9, 10)
+ * )
+ * }}}
+ */
+ @Since("From which version?")
+ def multiSliding[A](x: Seq[A], min: Int, max: Int): Seq[Seq[A]] = {
+ type B = Seq[A]
+
+ def addWindowsFromBuffer(acc: List[B], buffer: Queue[A]): List[B] = {
+ buffer.drop(min - 1).foldLeft((acc, buffer.take(min - 1))) {
+ case ((a, b), current) =>
+ val newB = b.enqueue(current)
+ (newB :: a, newB)
+ }._1
+ }
+
+ @tailrec
+ def addWindowsFromFinalBuffer(acc: List[B], buffer: Queue[A]): List[B] = {
+ buffer.dequeueOption match {
+ case Some((_, tail)) => addWindowsFromFinalBuffer(addWindowsFromBuffer(acc, tail), tail)
+ case None => acc
+ }
+ }
+
+ def calculateMultiSliding(): List[B] = {
--- End diff --
Nope, unfortunately it's not the case:
```Scala
scala> def slidingOriginal(x: Seq[String], n: Int): Seq[String] = {
| x.iterator.sliding(n).withPartial(false).map(_.mkString(" ")).toSeq
| }
slidingOriginal: (x: Seq[String], n: Int)Seq[String]
scala> def slidingNew(x: Seq[String], n: Int): Seq[String] = {
| multiSliding(x, n, n).map(_.mkString(" "))
| }
slidingNew: (x: Seq[String], n: Int)Seq[String]
scala> time(slidingOriginal((1 to 10000).map(_.toString), 5))
res9: scala.concurrent.duration.FiniteDuration = 2427473 nanoseconds
scala> time(slidingOriginal((1 to 10000).map(_.toString), 5)).toMillis
res10: Long = 3
scala> time(slidingNew((1 to 10000).map(_.toString), 5)).toMillis
res11: Long = 15
scala> time(slidingOriginal((1 to 100000).map(_.toString), 5)).toMillis
res12: Long = 11
scala> time(slidingNew((1 to 100000).map(_.toString), 5)).toMillis
res13: Long = 479
scala> time(slidingOriginal((1 to 10000).map(_.toString), 50)).toMillis
res14: Long = 1
scala> time(slidingNew((1 to 10000).map(_.toString), 50)).toMillis
res15: Long = 136
```
Adding code to fall back to the original implementation if `n == maxN`...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r149690477
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala ---
@@ -42,11 +42,22 @@ class NGram @Since("1.5.0") (@Since("1.5.0") override val uid: String)
/**
* Minimum n-gram length, greater than or equal to 1.
+ * All values of m such that n <= m <= maxN will be used.
* Default: 2, bigram features
* @group param
*/
@Since("1.5.0")
- val n: IntParam = new IntParam(this, "n", "number elements per n-gram (>=1)",
+ val n: IntParam = new IntParam(this, "n", "minimum number of elements per n-gram (>=1)",
+ ParamValidators.gtEq(1))
+
+ /**
+ * Maximum n-gram length, greater than or equal to `n`.
+ * All values of m such that n <= m <= maxN will be used.
+ * Default: 2, bigram features
+ * @group param
+ */
+ @Since("From which version?")
+ val maxN: IntParam = new IntParam(this, "maxN", "maximum number elements per n-gram (>=n)",
--- End diff --
Added explanation in the scaladoc comment for the case when `maxN` is not set.
The default is set to 2 here: https://github.com/apache/spark/pull/19659
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r149480525
--- Diff: mllib/src/main/scala/org/apache/spark/ml/extensions/seq/package.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.extensions
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+
+import org.apache.spark.annotation.Since
+
+package object seq {
+
+ /**
+ * Calculates sliding windows over multiple length parameters simultaneously.
+ * @param x the input sequence over which sliding windows will be collected
+ * @param min the inclusive minimal length of the window to be collected
+ * @param max the inclusive maximal length of the window to be collected
+ * @return the collected windows
+ *
+ * @example {{{
+ * multiSliding(1 to 5, min = 2, max = 4) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5),
+ * Seq(3, 4), Seq(3, 4, 5),
+ * Seq(4, 5)
+ * )
+ *
+ * multiSliding(1 to 10, min = 2, max = 5) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4), Seq(1, 2, 3, 4, 5),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5), Seq(2, 3, 4, 5, 6),
+ * Seq(3, 4), Seq(3, 4, 5), Seq(3, 4, 5, 6), Seq(3, 4, 5, 6, 7),
+ * Seq(4, 5), Seq(4, 5, 6), Seq(4, 5, 6, 7), Seq(4, 5, 6, 7, 8),
+ * Seq(5, 6), Seq(5, 6, 7), Seq(5, 6, 7, 8), Seq(5, 6, 7, 8, 9),
+ * Seq(6, 7), Seq(6, 7, 8), Seq(6, 7, 8, 9), Seq(6, 7, 8, 9, 10),
+ * Seq(7, 8), Seq(7, 8, 9), Seq(7, 8, 9, 10),
+ * Seq(8, 9), Seq(8, 9, 10),
+ * Seq(9, 10)
+ * )
+ * }}}
+ */
+ @Since("From which version?")
--- End diff --
Also is there a reason not to put this functionality inside of ngram its self for now?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r148930750
--- Diff: mllib/src/main/scala/org/apache/spark/ml/extensions/seq/package.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.extensions
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+
+import org.apache.spark.annotation.Since
+
+package object seq {
+
+ /**
+ * Calculates sliding windows over multiple length parameters simultaneously.
+ * @param x the input sequence over which sliding windows will be collected
+ * @param min the inclusive minimal length of the window to be collected
+ * @param max the inclusive maximal length of the window to be collected
+ * @return the collected windows
+ *
+ * @example {{{
+ * multiSliding(1 to 5, min = 2, max = 4) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5),
+ * Seq(3, 4), Seq(3, 4, 5),
+ * Seq(4, 5)
+ * )
+ *
+ * multiSliding(1 to 10, min = 2, max = 5) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4), Seq(1, 2, 3, 4, 5),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5), Seq(2, 3, 4, 5, 6),
+ * Seq(3, 4), Seq(3, 4, 5), Seq(3, 4, 5, 6), Seq(3, 4, 5, 6, 7),
+ * Seq(4, 5), Seq(4, 5, 6), Seq(4, 5, 6, 7), Seq(4, 5, 6, 7, 8),
+ * Seq(5, 6), Seq(5, 6, 7), Seq(5, 6, 7, 8), Seq(5, 6, 7, 8, 9),
+ * Seq(6, 7), Seq(6, 7, 8), Seq(6, 7, 8, 9), Seq(6, 7, 8, 9, 10),
+ * Seq(7, 8), Seq(7, 8, 9), Seq(7, 8, 9, 10),
+ * Seq(8, 9), Seq(8, 9, 10),
+ * Seq(9, 10)
+ * )
+ * }}}
+ */
+ @Since("From which version?")
+ def multiSliding[A](x: Seq[A], min: Int, max: Int): Seq[Seq[A]] = {
+ type B = Seq[A]
+
+ def addWindowsFromBuffer(acc: List[B], buffer: Queue[A]): List[B] = {
+ buffer.drop(min - 1).foldLeft((acc, buffer.take(min - 1))) {
+ case ((a, b), current) =>
+ val newB = b.enqueue(current)
+ (newB :: a, newB)
+ }._1
+ }
+
+ @tailrec
+ def addWindowsFromFinalBuffer(acc: List[B], buffer: Queue[A]): List[B] = {
+ buffer.dequeueOption match {
+ case Some((_, tail)) => addWindowsFromFinalBuffer(addWindowsFromBuffer(acc, tail), tail)
+ case None => acc
+ }
+ }
+
+ def calculateMultiSliding(): List[B] = {
+ val (accumulated, finalBuffer) = x.foldLeft((List.empty[B], Queue.empty[A])) {
+
+ case ((acc, buffer), current) if buffer.length < min - 1 =>
+ (acc, buffer.enqueue(current))
+
+ case ((acc, buffer), current) if buffer.length == min - 1 =>
+ val newBuffer = buffer.enqueue(current)
+ (newBuffer :: acc, newBuffer)
+
+ case ((acc, buffer), current) if buffer.length >= min && buffer.length < max =>
+ val newBuffer = buffer.enqueue(current)
+ (newBuffer :: acc, newBuffer)
+
+ case ((acc, buffer), current) if buffer.length == max =>
+ val (_, newBuffer) = buffer.enqueue(current).dequeue
+ (addWindowsFromBuffer(acc, newBuffer), newBuffer)
+
+ case ((acc, buffer), _) if buffer.length > max =>
+ (acc, buffer)
+
+ }
+
+ addWindowsFromFinalBuffer(accumulated, finalBuffer).reverse
+ }
+
+ (1 <= min) && (min <= max) match {
+ case true => calculateMultiSliding()
+ case false => Seq.empty
+ }
+ }
+
+ @Since("From which version?")
--- End diff --
Note to self: please don't forget to fill this in.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19659
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r149479801
--- Diff: mllib/src/test/scala/org/apache/spark/ml/feature/NGramSuite.scala ---
@@ -76,11 +76,32 @@ class NGramSuite extends SparkFunSuite with MLlibTestSparkContext with DefaultRe
testNGram(nGram, dataset)
}
+ test("NGramLength in [2, 4] yields length 2, 3, 4 n-grams") {
+ val nGram = new NGram()
+ .setInputCol("inputTokens")
+ .setOutputCol("nGrams")
+ .setN(2)
+ .setMaxN(4)
+ val dataset = Seq(NGramTestData(
+ Array("a", "b", "c", "d", "e", "f", "g"),
+ Array(
+ "a b", "a b c", "a b c d",
+ "b c", "b c d", "b c d e",
+ "c d", "c d e", "c d e f",
+ "d e", "d e f", "d e f g",
+ "e f", "e f g",
+ "f g"
+ )
+ )).toDF()
+ testNGram(nGram, dataset)
+ }
+
test("read/write") {
val t = new NGram()
.setInputCol("myInputCol")
.setOutputCol("myOutputCol")
.setN(3)
+ .setMaxN(5)
testDefaultReadWrite(t)
--- End diff --
Would be good to make sure read/write makes sense even when maxN isn't set perhaps?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r148930761
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala ---
@@ -42,11 +42,22 @@ class NGram @Since("1.5.0") (@Since("1.5.0") override val uid: String)
/**
* Minimum n-gram length, greater than or equal to 1.
+ * All values of m such that n <= m <= maxN will be used.
* Default: 2, bigram features
* @group param
*/
@Since("1.5.0")
- val n: IntParam = new IntParam(this, "n", "number elements per n-gram (>=1)",
+ val n: IntParam = new IntParam(this, "n", "minimum number of elements per n-gram (>=1)",
+ ParamValidators.gtEq(1))
+
+ /**
+ * Maximum n-gram length, greater than or equal to `n`.
+ * All values of m such that n <= m <= maxN will be used.
+ * Default: 2, bigram features
+ * @group param
+ */
+ @Since("From which version?")
--- End diff --
Note to self: please fill this in.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/19659
Thanks for working on this. Jenkins OK to test.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on the issue:
https://github.com/apache/spark/pull/19659
Also, will this work in SparkR, or do I need to add some code/configuration to expose the functionality to [R]?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/19659
Is this a thing you are still working on @mpetruska ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r148930762
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala ---
@@ -57,10 +68,22 @@ class NGram @Since("1.5.0") (@Since("1.5.0") override val uid: String)
@Since("1.5.0")
def getN: Int = $(n)
+ /** @group setParam */
+ @Since("From which version?")
--- End diff --
Note to self: please fill this in.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on the issue:
https://github.com/apache/spark/pull/19659
No, I'm not working on this one, the feature seems to lack any real interest. We can close it, if you like.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r149479380
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala ---
@@ -42,11 +42,22 @@ class NGram @Since("1.5.0") (@Since("1.5.0") override val uid: String)
/**
* Minimum n-gram length, greater than or equal to 1.
+ * All values of m such that n <= m <= maxN will be used.
* Default: 2, bigram features
* @group param
*/
@Since("1.5.0")
- val n: IntParam = new IntParam(this, "n", "number elements per n-gram (>=1)",
+ val n: IntParam = new IntParam(this, "n", "minimum number of elements per n-gram (>=1)",
+ ParamValidators.gtEq(1))
+
+ /**
+ * Maximum n-gram length, greater than or equal to `n`.
+ * All values of m such that n <= m <= maxN will be used.
+ * Default: 2, bigram features
+ * @group param
+ */
+ @Since("From which version?")
+ val maxN: IntParam = new IntParam(this, "maxN", "maximum number elements per n-gram (>=n)",
--- End diff --
Should this have a default value? Also perhaps the explanation that if unset maxN = n
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r149478967
--- Diff: mllib/src/main/scala/org/apache/spark/ml/extensions/seq/package.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.extensions
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+
+import org.apache.spark.annotation.Since
+
+package object seq {
+
+ /**
+ * Calculates sliding windows over multiple length parameters simultaneously.
+ * @param x the input sequence over which sliding windows will be collected
+ * @param min the inclusive minimal length of the window to be collected
+ * @param max the inclusive maximal length of the window to be collected
+ * @return the collected windows
+ *
+ * @example {{{
+ * multiSliding(1 to 5, min = 2, max = 4) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5),
+ * Seq(3, 4), Seq(3, 4, 5),
+ * Seq(4, 5)
+ * )
+ *
+ * multiSliding(1 to 10, min = 2, max = 5) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4), Seq(1, 2, 3, 4, 5),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5), Seq(2, 3, 4, 5, 6),
+ * Seq(3, 4), Seq(3, 4, 5), Seq(3, 4, 5, 6), Seq(3, 4, 5, 6, 7),
+ * Seq(4, 5), Seq(4, 5, 6), Seq(4, 5, 6, 7), Seq(4, 5, 6, 7, 8),
+ * Seq(5, 6), Seq(5, 6, 7), Seq(5, 6, 7, 8), Seq(5, 6, 7, 8, 9),
+ * Seq(6, 7), Seq(6, 7, 8), Seq(6, 7, 8, 9), Seq(6, 7, 8, 9, 10),
+ * Seq(7, 8), Seq(7, 8, 9), Seq(7, 8, 9, 10),
+ * Seq(8, 9), Seq(8, 9, 10),
+ * Seq(9, 10)
+ * )
+ * }}}
+ */
+ @Since("From which version?")
--- End diff --
We probably don't want these APIs to be public unless there is a good reason to have them be. Maybe make them private?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on the issue:
https://github.com/apache/spark/pull/19659
@holdenk : Thanks for the review.
Fixed/addressed your comments, can you please have another look?
Also I'd like some help with regards to the `@Since` attributes; how should I fill them in? Should I fill them in at all?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r148930753
--- Diff: mllib/src/main/scala/org/apache/spark/ml/extensions/seq/package.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.extensions
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+
+import org.apache.spark.annotation.Since
+
+package object seq {
+
+ /**
+ * Calculates sliding windows over multiple length parameters simultaneously.
+ * @param x the input sequence over which sliding windows will be collected
+ * @param min the inclusive minimal length of the window to be collected
+ * @param max the inclusive maximal length of the window to be collected
+ * @return the collected windows
+ *
+ * @example {{{
+ * multiSliding(1 to 5, min = 2, max = 4) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5),
+ * Seq(3, 4), Seq(3, 4, 5),
+ * Seq(4, 5)
+ * )
+ *
+ * multiSliding(1 to 10, min = 2, max = 5) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4), Seq(1, 2, 3, 4, 5),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5), Seq(2, 3, 4, 5, 6),
+ * Seq(3, 4), Seq(3, 4, 5), Seq(3, 4, 5, 6), Seq(3, 4, 5, 6, 7),
+ * Seq(4, 5), Seq(4, 5, 6), Seq(4, 5, 6, 7), Seq(4, 5, 6, 7, 8),
+ * Seq(5, 6), Seq(5, 6, 7), Seq(5, 6, 7, 8), Seq(5, 6, 7, 8, 9),
+ * Seq(6, 7), Seq(6, 7, 8), Seq(6, 7, 8, 9), Seq(6, 7, 8, 9, 10),
+ * Seq(7, 8), Seq(7, 8, 9), Seq(7, 8, 9, 10),
+ * Seq(8, 9), Seq(8, 9, 10),
+ * Seq(9, 10)
+ * )
+ * }}}
+ */
+ @Since("From which version?")
+ def multiSliding[A](x: Seq[A], min: Int, max: Int): Seq[Seq[A]] = {
+ type B = Seq[A]
+
+ def addWindowsFromBuffer(acc: List[B], buffer: Queue[A]): List[B] = {
+ buffer.drop(min - 1).foldLeft((acc, buffer.take(min - 1))) {
+ case ((a, b), current) =>
+ val newB = b.enqueue(current)
+ (newB :: a, newB)
+ }._1
+ }
+
+ @tailrec
+ def addWindowsFromFinalBuffer(acc: List[B], buffer: Queue[A]): List[B] = {
+ buffer.dequeueOption match {
+ case Some((_, tail)) => addWindowsFromFinalBuffer(addWindowsFromBuffer(acc, tail), tail)
+ case None => acc
+ }
+ }
+
+ def calculateMultiSliding(): List[B] = {
+ val (accumulated, finalBuffer) = x.foldLeft((List.empty[B], Queue.empty[A])) {
+
+ case ((acc, buffer), current) if buffer.length < min - 1 =>
+ (acc, buffer.enqueue(current))
+
+ case ((acc, buffer), current) if buffer.length == min - 1 =>
+ val newBuffer = buffer.enqueue(current)
+ (newBuffer :: acc, newBuffer)
+
+ case ((acc, buffer), current) if buffer.length >= min && buffer.length < max =>
+ val newBuffer = buffer.enqueue(current)
+ (newBuffer :: acc, newBuffer)
+
+ case ((acc, buffer), current) if buffer.length == max =>
+ val (_, newBuffer) = buffer.enqueue(current).dequeue
+ (addWindowsFromBuffer(acc, newBuffer), newBuffer)
+
+ case ((acc, buffer), _) if buffer.length > max =>
+ (acc, buffer)
+
+ }
+
+ addWindowsFromFinalBuffer(accumulated, finalBuffer).reverse
+ }
+
+ (1 <= min) && (min <= max) match {
+ case true => calculateMultiSliding()
+ case false => Seq.empty
+ }
+ }
+
+ @Since("From which version?")
+ implicit class SeqOps[A](val x: Seq[A]) extends AnyVal {
+
+ /**
+ * Calculates sliding windows over multiple length parameters
+ * simultaneously.
+ * @see `org.apache.spark.ml.extensions.seq.multiSliding`
+ */
+ @Since("From which version?")
--- End diff --
Note to self: please don't forget to fill this in.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r149600715
--- Diff: mllib/src/main/scala/org/apache/spark/ml/extensions/seq/package.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.extensions
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+
+import org.apache.spark.annotation.Since
+
+package object seq {
+
+ /**
+ * Calculates sliding windows over multiple length parameters simultaneously.
+ * @param x the input sequence over which sliding windows will be collected
+ * @param min the inclusive minimal length of the window to be collected
+ * @param max the inclusive maximal length of the window to be collected
+ * @return the collected windows
+ *
+ * @example {{{
+ * multiSliding(1 to 5, min = 2, max = 4) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5),
+ * Seq(3, 4), Seq(3, 4, 5),
+ * Seq(4, 5)
+ * )
+ *
+ * multiSliding(1 to 10, min = 2, max = 5) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4), Seq(1, 2, 3, 4, 5),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5), Seq(2, 3, 4, 5, 6),
+ * Seq(3, 4), Seq(3, 4, 5), Seq(3, 4, 5, 6), Seq(3, 4, 5, 6, 7),
+ * Seq(4, 5), Seq(4, 5, 6), Seq(4, 5, 6, 7), Seq(4, 5, 6, 7, 8),
+ * Seq(5, 6), Seq(5, 6, 7), Seq(5, 6, 7, 8), Seq(5, 6, 7, 8, 9),
+ * Seq(6, 7), Seq(6, 7, 8), Seq(6, 7, 8, 9), Seq(6, 7, 8, 9, 10),
+ * Seq(7, 8), Seq(7, 8, 9), Seq(7, 8, 9, 10),
+ * Seq(8, 9), Seq(8, 9, 10),
+ * Seq(9, 10)
+ * )
+ * }}}
+ */
+ @Since("From which version?")
--- End diff --
Right, agreed, will move the implementation into `NGram`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r148930766
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala ---
@@ -57,10 +68,22 @@ class NGram @Since("1.5.0") (@Since("1.5.0") override val uid: String)
@Since("1.5.0")
def getN: Int = $(n)
+ /** @group setParam */
+ @Since("From which version?")
+ def setMaxN(value: Int): this.type = set(maxN, value)
+
+ /** @group getParam */
+ @Since("From which version?")
--- End diff --
Note to self: please fill this in.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mpetruska <gi...@git.apache.org>.
Github user mpetruska commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r148930742
--- Diff: mllib/src/main/scala/org/apache/spark/ml/extensions/seq/package.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.extensions
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+
+import org.apache.spark.annotation.Since
+
+package object seq {
+
+ /**
+ * Calculates sliding windows over multiple length parameters simultaneously.
+ * @param x the input sequence over which sliding windows will be collected
+ * @param min the inclusive minimal length of the window to be collected
+ * @param max the inclusive maximal length of the window to be collected
+ * @return the collected windows
+ *
+ * @example {{{
+ * multiSliding(1 to 5, min = 2, max = 4) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5),
+ * Seq(3, 4), Seq(3, 4, 5),
+ * Seq(4, 5)
+ * )
+ *
+ * multiSliding(1 to 10, min = 2, max = 5) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4), Seq(1, 2, 3, 4, 5),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5), Seq(2, 3, 4, 5, 6),
+ * Seq(3, 4), Seq(3, 4, 5), Seq(3, 4, 5, 6), Seq(3, 4, 5, 6, 7),
+ * Seq(4, 5), Seq(4, 5, 6), Seq(4, 5, 6, 7), Seq(4, 5, 6, 7, 8),
+ * Seq(5, 6), Seq(5, 6, 7), Seq(5, 6, 7, 8), Seq(5, 6, 7, 8, 9),
+ * Seq(6, 7), Seq(6, 7, 8), Seq(6, 7, 8, 9), Seq(6, 7, 8, 9, 10),
+ * Seq(7, 8), Seq(7, 8, 9), Seq(7, 8, 9, 10),
+ * Seq(8, 9), Seq(8, 9, 10),
+ * Seq(9, 10)
+ * )
+ * }}}
+ */
+ @Since("From which version?")
--- End diff --
Please help me fill this in.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/19659
@holdenk I can take this up if this is needed. Let me know. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/19659
So SparkR will have to be separate, I'm going to cc @felixcheung , I don't know if we currently expose this functionality in the Spark R side.
As for the @Since attributes you choose them based on the version that your code is going to first appear in. At the point it might be 2.3.0 but it might also not. I'll try and do a more thorough review next week.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19659: [SPARK-19668][ML] Multiple NGram sizes
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19659#discussion_r149480876
--- Diff: mllib/src/main/scala/org/apache/spark/ml/extensions/seq/package.scala ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.extensions
+
+import scala.annotation.tailrec
+import scala.collection.immutable.Queue
+
+import org.apache.spark.annotation.Since
+
+package object seq {
+
+ /**
+ * Calculates sliding windows over multiple length parameters simultaneously.
+ * @param x the input sequence over which sliding windows will be collected
+ * @param min the inclusive minimal length of the window to be collected
+ * @param max the inclusive maximal length of the window to be collected
+ * @return the collected windows
+ *
+ * @example {{{
+ * multiSliding(1 to 5, min = 2, max = 4) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5),
+ * Seq(3, 4), Seq(3, 4, 5),
+ * Seq(4, 5)
+ * )
+ *
+ * multiSliding(1 to 10, min = 2, max = 5) == Seq(
+ * Seq(1, 2), Seq(1, 2, 3), Seq(1, 2, 3, 4), Seq(1, 2, 3, 4, 5),
+ * Seq(2, 3), Seq(2, 3, 4), Seq(2, 3, 4, 5), Seq(2, 3, 4, 5, 6),
+ * Seq(3, 4), Seq(3, 4, 5), Seq(3, 4, 5, 6), Seq(3, 4, 5, 6, 7),
+ * Seq(4, 5), Seq(4, 5, 6), Seq(4, 5, 6, 7), Seq(4, 5, 6, 7, 8),
+ * Seq(5, 6), Seq(5, 6, 7), Seq(5, 6, 7, 8), Seq(5, 6, 7, 8, 9),
+ * Seq(6, 7), Seq(6, 7, 8), Seq(6, 7, 8, 9), Seq(6, 7, 8, 9, 10),
+ * Seq(7, 8), Seq(7, 8, 9), Seq(7, 8, 9, 10),
+ * Seq(8, 9), Seq(8, 9, 10),
+ * Seq(9, 10)
+ * )
+ * }}}
+ */
+ @Since("From which version?")
+ def multiSliding[A](x: Seq[A], min: Int, max: Int): Seq[Seq[A]] = {
+ type B = Seq[A]
+
+ def addWindowsFromBuffer(acc: List[B], buffer: Queue[A]): List[B] = {
+ buffer.drop(min - 1).foldLeft((acc, buffer.take(min - 1))) {
+ case ((a, b), current) =>
+ val newB = b.enqueue(current)
+ (newB :: a, newB)
+ }._1
+ }
+
+ @tailrec
+ def addWindowsFromFinalBuffer(acc: List[B], buffer: Queue[A]): List[B] = {
+ buffer.dequeueOption match {
+ case Some((_, tail)) => addWindowsFromFinalBuffer(addWindowsFromBuffer(acc, tail), tail)
+ case None => acc
+ }
+ }
+
+ def calculateMultiSliding(): List[B] = {
--- End diff --
Does this perform as well as the current code does?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org