You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by huaxingao <gi...@git.apache.org> on 2018/07/04 03:23:10 UTC
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
GitHub user huaxingao opened a pull request:
https://github.com/apache/spark/pull/21710
[SPARK-24207][R]add R API for PrefixSpan
## What changes were proposed in this pull request?
add R API for PrefixSpan
## How was this patch tested?
add test in test_mllib_fpm.R
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/huaxingao/spark spark-24207
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21710.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21710
----
commit 5ed11e67703dc7dfb23fb7ff68acffde33c13a30
Author: Huaxin Gao <hu...@...>
Date: 2018-07-04T03:18:08Z
[SPARK-24207][R]add R API for PrefixSpan
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203245010
--- Diff: R/pkg/R/generics.R ---
@@ -1415,6 +1415,13 @@ setGeneric("spark.freqItemsets", function(object) { standardGeneric("spark.freqI
#' @rdname spark.fpGrowth
setGeneric("spark.associationRules", function(object) { standardGeneric("spark.associationRules") })
+#' @rdname spark.prefixSpan
+setGeneric("spark.prefixSpan", function(...) { standardGeneric("spark.prefixSpan") })
+
+#' @rdname spark.prefixSpan
--- End diff --
i think he meant the rdname - yes ok to have both in one rd
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #93245 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93245/testReport)** for PR 21710 at commit [`ec88d38`](https://github.com/apache/spark/commit/ec88d38f3bafb9740c7c9a3856ddc3a64cb16fe7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #92592 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92592/testReport)** for PR 21710 at commit [`5ed11e6`](https://github.com/apache/spark/commit/5ed11e67703dc7dfb23fb7ff68acffde33c13a30).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #97125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97125/testReport)** for PR 21710 at commit [`0393076`](https://github.com/apache/spark/commit/039307693a34f8b58575a9df70a107d0d2e52c74).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3608/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #93245 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93245/testReport)** for PR 21710 at commit [`ec88d38`](https://github.com/apache/spark/commit/ec88d38f3bafb9740c7c9a3856ddc3a64cb16fe7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97125/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203486740
--- Diff: mllib/src/main/scala/org/apache/spark/ml/r/PrefixSpanWrapper.scala ---
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.r
+
+import org.apache.spark.ml.fpm.PrefixSpan
+
+private[r] object PrefixSpanWrapper {
+ def getPrefixSpan(
+ minSupport: Double,
+ maxPatternLength: Int,
+ maxLocalProjDBSize: Double,
--- End diff --
Why `maxLocalProjDBSize` is `Double`, I see you do `toLong` later?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #97317 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97317/testReport)** for PR 21710 at commit [`0bab5ac`](https://github.com/apache/spark/commit/0bab5aca283bacdfe36ba1c669521df9d7ff81f3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r223574019
--- Diff: R/pkg/R/mllib_fpm.R ---
@@ -154,3 +160,74 @@ setMethod("write.ml", signature(object = "FPGrowthModel", path = "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PrefixSpan
+#'
+#' A parallel PrefixSpan algorithm to mine frequent sequential patterns.
+#' \code{spark.prefixSpan} returns an instance of PrefixSpan.
+#' \code{spark.findFrequentSequentialPatterns} returns a complete set of frequent sequential
+#' patterns.
+#' For more details, see
+#' \href{https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html#prefixspan}{
+#' PrefixSpan}.
+#'
+#' @param minSupport Minimal support level.
+#' @param maxPatternLength Maximal pattern length.
+#' @param maxLocalProjDBSize Maximum number of items (including delimiters used in the internal
+#' storage format) allowed in a projected database before local
+#' processing.
+#' @param sequenceCol name of the sequence column in dataset.
+#' @param ... additional argument(s) passed to the method.
+#' @return \code{spark.prefixSpan} returns an instance of PrefixSpan
+#' @rdname spark.prefixSpan
+#' @name spark.prefixSpan
+#' @aliases spark.prefixSpan,ANY-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
+#' list(list(list(1L), list(3L, 2L), list(1L, 2L))),
+#' list(list(list(1L, 2L), list(5L))),
+#' list(list(list(6L)))), schema = c("sequence"))
+#' prefix_Span <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L,
+#' maxLocalProjDBSize = 32000000L)
+#' frequency <- spark.findFrequentSequentialPatterns(prefix_Span, df)
+#' showDF(frequency)
+#' }
+#' @note spark.prefixSpan since 3.0.0
+setMethod("spark.prefixSpan", signature(),
+ function(minSupport=0.1, maxPatternLength=10L,
+ maxLocalProjDBSize=32000000L, sequenceCol="sequence") {
+ if (!is.numeric(minSupport) || minSupport < 0) {
+ stop("minSupport should be a number with value >= 0.")
+ }
+ if (!is.integer(maxPatternLength) || maxPatternLength <= 0) {
+ stop("maxPatternLength should be a number with value > 0.")
+ }
+ if (!is.numeric(maxLocalProjDBSize) || maxLocalProjDBSize <= 0) {
+ stop("maxLocalProjDBSize should be a number with value > 0.")
+ }
+
+ jobj <- callJStatic("org.apache.spark.ml.r.PrefixSpanWrapper", "getPrefixSpan",
+ as.numeric(minSupport), as.integer(maxPatternLength),
+ as.numeric(maxLocalProjDBSize), as.character(sequenceCol))
+ new("PrefixSpan", jobj = jobj)
+ })
+
+# Find frequent sequential patterns.
+
+#' @param object a PrefixSpan object.
+#' @param data A SparkDataFrame.
+#' @return A complete set of frequent sequential patterns in the input sequences of itemsets.
+#' The returned \code{SparkDataFrame} contains columns of sequence and corresponding
+#' frequency. The schema of it will be:
+#' \code{sequence: ArrayType(ArrayType(T))} (T is the item type)
+#' \code{freq: Long}
+#' @rdname spark.prefixSpan
+#' @aliases findFrequentSequentialPatterns,PrefixSpan,SparkDataFrame-method
+#' @note spark.findFrequentSequentialPatterns(PrefixSpan, SparkDataFrame) since 3.0.0
+
+setMethod("spark.findFrequentSequentialPatterns",
+ signature(object = "PrefixSpan", data = "SparkDataFrame"),
--- End diff --
hmm, another one - why isn't `data` the first param? should also be called `x` (the convention in R API)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203526021
--- Diff: mllib/src/main/scala/org/apache/spark/ml/r/PrefixSpanWrapper.scala ---
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.r
+
+import org.apache.spark.ml.fpm.PrefixSpan
+
+private[r] object PrefixSpanWrapper {
+ def getPrefixSpan(
+ minSupport: Double,
+ maxPatternLength: Int,
+ maxLocalProjDBSize: Double,
--- End diff --
```maxLocalProjDBSize``` is a ```LongParam``` in scala, but it seems that r doesn't have ```long```.
See ```help(integer):```
```
Note that on almost all implementations of R the range of
representable integers is restricted to about +/-2*10^9: ‘double’s
can hold much larger integers exactly.
```
so I used ```double``` in r, and do ```toLong``` later
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203229733
--- Diff: R/pkg/R/generics.R ---
@@ -1415,6 +1415,13 @@ setGeneric("spark.freqItemsets", function(object) { standardGeneric("spark.freqI
#' @rdname spark.fpGrowth
setGeneric("spark.associationRules", function(object) { standardGeneric("spark.associationRules") })
+#' @rdname spark.prefixSpan
+setGeneric("spark.prefixSpan", function(...) { standardGeneric("spark.prefixSpan") })
+
+#' @rdname spark.prefixSpan
--- End diff --
@viirya Thanks for your review.
Do you mean to use ```findFrequentSequentialPatterns``` instead of ```spark.findFrequentSequentialPatterns```? I have a question about when we need to add ```spark``` in front of the method name. In FPGrowth, some methods have ```spark.```, e.g. ```spark.freqItemsets```, ```spark.associationRules```. Some methods don't have ```spark.```, e.g. ```predict``` and ```write.ml``` .
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #96829 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96829/testReport)** for PR 21710 at commit [`93591ac`](https://github.com/apache/spark/commit/93591acefa405dcb9d8597659106496cd42d069e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r223568243
--- Diff: R/pkg/R/mllib_fpm.R ---
@@ -154,3 +160,74 @@ setMethod("write.ml", signature(object = "FPGrowthModel", path = "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PrefixSpan
+#'
+#' A parallel PrefixSpan algorithm to mine frequent sequential patterns.
+#' \code{spark.prefixSpan} returns an instance of PrefixSpan.
+#' \code{spark.findFrequentSequentialPatterns} returns a complete set of frequent sequential
+#' patterns.
+#' For more details, see
+#' \href{https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html#prefixspan}{
+#' PrefixSpan}.
+#'
+#' @param minSupport Minimal support level.
+#' @param maxPatternLength Maximal pattern length.
+#' @param maxLocalProjDBSize Maximum number of items (including delimiters used in the internal
+#' storage format) allowed in a projected database before local
+#' processing.
+#' @param sequenceCol name of the sequence column in dataset.
+#' @param ... additional argument(s) passed to the method.
+#' @return \code{spark.prefixSpan} returns an instance of PrefixSpan
+#' @rdname spark.prefixSpan
+#' @name spark.prefixSpan
+#' @aliases spark.prefixSpan,ANY-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
+#' list(list(list(1L), list(3L, 2L), list(1L, 2L))),
+#' list(list(list(1L, 2L), list(5L))),
+#' list(list(list(6L)))), schema = c("sequence"))
+#' prefix_Span <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L,
+#' maxLocalProjDBSize = 32000000L)
+#' frequency <- spark.findFrequentSequentialPatterns(prefix_Span, df)
+#' showDF(frequency)
+#' }
+#' @note spark.prefixSpan since 3.0.0
+setMethod("spark.prefixSpan", signature(),
+ function(minSupport=0.1, maxPatternLength=10L,
+ maxLocalProjDBSize=32000000L, sequenceCol="sequence") {
+ if (!is.numeric(minSupport) || minSupport < 0) {
+ stop("minSupport should be a number with value >= 0.")
+ }
+ if (!is.integer(maxPatternLength) || maxPatternLength <= 0) {
+ stop("maxPatternLength should be a number with value > 0.")
+ }
+ if (!is.numeric(maxLocalProjDBSize) || maxLocalProjDBSize <= 0) {
+ stop("maxLocalProjDBSize should be a number with value > 0.")
+ }
+
+ jobj <- callJStatic("org.apache.spark.ml.r.PrefixSpanWrapper", "getPrefixSpan",
+ as.numeric(minSupport), as.integer(maxPatternLength),
+ as.numeric(maxLocalProjDBSize), as.character(sequenceCol))
+ new("PrefixSpan", jobj = jobj)
+ })
+
+# Find frequent sequential patterns.
+
+#' @param object a PrefixSpan object.
+#' @param data A SparkDataFrame.
+#' @return A complete set of frequent sequential patterns in the input sequences of itemsets.
+#' The returned \code{SparkDataFrame} contains columns of sequence and corresponding
+#' frequency. The schema of it will be:
+#' \code{sequence: ArrayType(ArrayType(T))} (T is the item type)
+#' \code{freq: Long}
+#' @rdname spark.prefixSpan
+#' @aliases findFrequentSequentialPatterns,PrefixSpan,SparkDataFrame-method
+#' @note spark.findFrequentSequentialPatterns(PrefixSpan, SparkDataFrame) since 3.0.0
+
--- End diff --
remove extra line - note this might break doc generation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97317/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21710
we could, but we tend to forget about them until the release, so I'd prefer we don't and have everything in one PR.
not ideal, we could leave out code example, or maybe even vigenette, but we really should have
- SparkR programming guide
- docs/ml-*.md for the "R" tab
which should only take a line or two.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96829/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203148676
--- Diff: R/pkg/R/mllib_fpm.R ---
@@ -154,3 +160,74 @@ setMethod("write.ml", signature(object = "FPGrowthModel", path = "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PrefixSpan
+#'
+#' A parallel PrefixSpan algorithm to mine frequent sequential patterns.
+#' \code{spark.prefixSpan} returns an instance of PrefixSpan.
+#' \code{spark.findFrequentSequentialPatterns} returns a complete set of frequent sequential
+#' patterns.
+#' For more details, see
+#' \href{https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html#prefixspan}{
+#' PrefixSpan}.
+#'
+#' @param minSupport Minimal support level.
+#' @param maxPatternLength Maximal pattern length.
+#' @param maxLocalProjDBSize Maximum number of items (including delimiters used in the internal
+#' storage format) allowed in a projected database before local
+#' processing.
+#' @param sequenceCol name of the sequence column in dataset.
+#' @param ... additional argument(s) passed to the method.
+#' @return \code{spark.prefixSpan} returns an instance of PrefixSpan
+#' @rdname spark.prefixSpan
+#' @name spark.prefixSpan
+#' @aliases spark.prefixSpan,ANY-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
+#' list(list(list(1L), list(3L, 2L), list(1L, 2L))),
+#' list(list(list(1L, 2L), list(5L))),
+#' list(list(list(6L)))), schema = c("sequence"))
+#' prefix_Span <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L,
+#' maxLocalProjDBSize = 32000000L)
+#' frequency <- spark.findFrequentSequentialPatterns(prefix_Span, df)
+#' showDF(frequency)
+#' }
+#' @note spark.prefixSpan since 2.4.0
+setMethod("spark.prefixSpan", signature(),
+ function(minSupport=0.1, maxPatternLength=10L,
+ maxLocalProjDBSize=32000000L, sequenceCol="sequence") {
+ if (!is.numeric(minSupport) || minSupport < 0) {
+ stop("minSupport should be a number with value >= 0.")
+ }
+ if (!is.integer(maxPatternLength) || maxPatternLength <= 0) {
+ stop("maxPatternLength should be a number with value > 0.")
+ }
+ if (!is.numeric(maxLocalProjDBSize) || maxLocalProjDBSize <= 0) {
+ stop("maxLocalProjDBSize should be a number with value > 0.")
+ }
+
+ jobj <- callJStatic("org.apache.spark.ml.r.PrefixSpanWrapper", "getPrefixSpan",
+ as.numeric(minSupport), as.integer(maxPatternLength),
+ as.numeric(maxLocalProjDBSize), as.character(sequenceCol))
+ new("PrefixSpan", jobj = jobj)
+ })
+
+# Find frequent sequential patterns.
+
+#' @param object a prefixSpan object.
+#' @param data A SparkDataFrame.
+#' @return A complete set of frequent sequential patterns in the input sequences of itemsets.
+#' The returned \code{SparkDataFrame} contains columns of sequence and corresponding
+#' frequency. The schema of it will be:
+#' \code{sequence: ArrayType(ArrayType(T))} (T is the item type)
+#' \code{freq: Long}
+#' @rdname spark.prefixSpan
--- End diff --
`findFrequentSequentialPatterns`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203147694
--- Diff: R/pkg/R/generics.R ---
@@ -1415,6 +1415,13 @@ setGeneric("spark.freqItemsets", function(object) { standardGeneric("spark.freqI
#' @rdname spark.fpGrowth
setGeneric("spark.associationRules", function(object) { standardGeneric("spark.associationRules") })
+#' @rdname spark.prefixSpan
+setGeneric("spark.prefixSpan", function(...) { standardGeneric("spark.prefixSpan") })
+
+#' @rdname spark.prefixSpan
--- End diff --
`findFrequentSequentialPatterns`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203538149
--- Diff: mllib/src/main/scala/org/apache/spark/ml/r/PrefixSpanWrapper.scala ---
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.r
+
+import org.apache.spark.ml.fpm.PrefixSpan
+
+private[r] object PrefixSpanWrapper {
+ def getPrefixSpan(
+ minSupport: Double,
+ maxPatternLength: Int,
+ maxLocalProjDBSize: Double,
--- End diff --
Ah, I see. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #92658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92658/testReport)** for PR 21710 at commit [`d2e85f5`](https://github.com/apache/spark/commit/d2e85f5f30a89dbd12d5bcd4f4c5187aac956b87).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on the issue:
https://github.com/apache/spark/pull/21710
@felixcheung
Are there any other things I need to change? If not, could this PR be merged in 2.4? Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on the issue:
https://github.com/apache/spark/pull/21710
@felixcheung Can I open a new jira for code example and documentation?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21710
Sure
________________________________
From: Huaxin Gao <no...@github.com>
Sent: Friday, October 26, 2018 1:16:38 PM
To: apache/spark
Cc: Felix Cheung; Mention
Subject: Re: [apache/spark] [SPARK-24207][R]add R API for PrefixSpan (#21710)
@felixcheung<https://github.com/felixcheung> I am terribly sorry that I missed your comment for the ml doc and example for 2.4. Is it still time to merge in 2.4? I saw one of my PR got merged in 2.4 last night. I can submit a PR for 2.4 doc and example now. Please let me know. Thanks a lot!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/apache/spark/pull/21710#issuecomment-433530451>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AIjc-0WUC8M3qb08DT-yHbc1UhI7Osv4ks5uo22mgaJpZM4VB5oc>.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3802/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #96829 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96829/testReport)** for PR 21710 at commit [`93591ac`](https://github.com/apache/spark/commit/93591acefa405dcb9d8597659106496cd42d069e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r223760003
--- Diff: examples/src/main/python/ml/prefixspan_example.py ---
@@ -0,0 +1,48 @@
+#
--- End diff --
@felixcheung I don't think the doc, java/scala/python example are in 2.4. Shall I open a separate jira to merge in the fix?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #93145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93145/testReport)** for PR 21710 at commit [`61f0de8`](https://github.com/apache/spark/commit/61f0de8c8a4544a6baabc85ab9abb8cd396528a5).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203159650
--- Diff: R/pkg/tests/fulltests/test_mllib_fpm.R ---
@@ -82,4 +82,26 @@ test_that("spark.fpGrowth", {
})
+test_that("spark.prefixSpan", {
+ df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
+ list(list(list(1L), list(3L, 2L), list(1L, 2L))),
+ list(list(list(1L, 2L), list(5L))),
+ list(list(list(6L)))), schema = c("sequence"))
+ prefix_Span1 <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L,
+ maxLocalProjDBSize = 32000000L)
+ result1 <- spark.findFrequentSequentialPatterns(prefix_Span1, df)
+
+ expected_result <- createDataFrame(list(list(list(list(1L)), 3L),
+ list(list(list(3L)), 2L),
+ list(list(list(2L)), 3L),
+ list(list(list(1L, 2L)), 3L),
+ list(list(list(1L), list(3L)), 2L)),
+ schema = c("sequence", "freq"))
+ expect_equivalent(expected_result, result1)
+
+ prefix_Span2 <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L)
+ result2 <- spark.findFrequentSequentialPatterns(prefix_Span2, df)
+ expect_equivalent(expected_result, result2)
--- End diff --
Is any difference between this test and above one? They are both equal to `expected_result`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #97125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97125/testReport)** for PR 21710 at commit [`0393076`](https://github.com/apache/spark/commit/039307693a34f8b58575a9df70a107d0d2e52c74).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93145/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on the issue:
https://github.com/apache/spark/pull/21710
@felixcheung I am terribly sorry that I missed your comment for the ml doc and example for 2.4. Is it still time to merge in 2.4? I saw one of my PR got merged in 2.4 last night. I can submit a PR for 2.4 doc and example now. Please let me know. Thanks a lot!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/712/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r223568592
--- Diff: docs/ml-frequent-pattern-mining.md ---
@@ -85,3 +85,56 @@ Refer to the [R API docs](api/R/spark.fpGrowth.html) for more details.
</div>
</div>
+
+## PrefixSpan
--- End diff --
is this not in the doc for branch-2.4? we might need a quick fix on the branch then before the next RC, I think
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r223574367
--- Diff: R/pkg/R/mllib_fpm.R ---
@@ -154,3 +160,74 @@ setMethod("write.ml", signature(object = "FPGrowthModel", path = "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PrefixSpan
+#'
+#' A parallel PrefixSpan algorithm to mine frequent sequential patterns.
+#' \code{spark.prefixSpan} returns an instance of PrefixSpan.
+#' \code{spark.findFrequentSequentialPatterns} returns a complete set of frequent sequential
+#' patterns.
+#' For more details, see
+#' \href{https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html#prefixspan}{
+#' PrefixSpan}.
+#'
+#' @param minSupport Minimal support level.
+#' @param maxPatternLength Maximal pattern length.
+#' @param maxLocalProjDBSize Maximum number of items (including delimiters used in the internal
+#' storage format) allowed in a projected database before local
+#' processing.
+#' @param sequenceCol name of the sequence column in dataset.
+#' @param ... additional argument(s) passed to the method.
+#' @return \code{spark.prefixSpan} returns an instance of PrefixSpan
+#' @rdname spark.prefixSpan
+#' @name spark.prefixSpan
+#' @aliases spark.prefixSpan,ANY-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
+#' list(list(list(1L), list(3L, 2L), list(1L, 2L))),
+#' list(list(list(1L, 2L), list(5L))),
+#' list(list(list(6L)))), schema = c("sequence"))
+#' prefix_Span <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L,
+#' maxLocalProjDBSize = 32000000L)
+#' frequency <- spark.findFrequentSequentialPatterns(prefix_Span, df)
+#' showDF(frequency)
+#' }
+#' @note spark.prefixSpan since 3.0.0
+setMethod("spark.prefixSpan", signature(),
+ function(minSupport=0.1, maxPatternLength=10L,
+ maxLocalProjDBSize=32000000L, sequenceCol="sequence") {
+ if (!is.numeric(minSupport) || minSupport < 0) {
+ stop("minSupport should be a number with value >= 0.")
+ }
+ if (!is.integer(maxPatternLength) || maxPatternLength <= 0) {
+ stop("maxPatternLength should be a number with value > 0.")
+ }
+ if (!is.numeric(maxLocalProjDBSize) || maxLocalProjDBSize <= 0) {
+ stop("maxLocalProjDBSize should be a number with value > 0.")
+ }
+
+ jobj <- callJStatic("org.apache.spark.ml.r.PrefixSpanWrapper", "getPrefixSpan",
+ as.numeric(minSupport), as.integer(maxPatternLength),
+ as.numeric(maxLocalProjDBSize), as.character(sequenceCol))
+ new("PrefixSpan", jobj = jobj)
+ })
+
+# Find frequent sequential patterns.
+
+#' @param object a PrefixSpan object.
+#' @param data A SparkDataFrame.
+#' @return A complete set of frequent sequential patterns in the input sequences of itemsets.
+#' The returned \code{SparkDataFrame} contains columns of sequence and corresponding
+#' frequency. The schema of it will be:
+#' \code{sequence: ArrayType(ArrayType(T))} (T is the item type)
+#' \code{freq: Long}
+#' @rdname spark.prefixSpan
+#' @aliases findFrequentSequentialPatterns,PrefixSpan,SparkDataFrame-method
+#' @note spark.findFrequentSequentialPatterns(PrefixSpan, SparkDataFrame) since 3.0.0
+
+setMethod("spark.findFrequentSequentialPatterns",
+ signature(object = "PrefixSpan", data = "SparkDataFrame"),
--- End diff --
nm, see the other comment instead
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203163321
--- Diff: mllib/src/main/scala/org/apache/spark/ml/r/PrefixSpanWrapper.scala ---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.r
+
+import org.apache.spark.ml.fpm.PrefixSpan
+import org.apache.spark.sql.{DataFrame, Dataset}
+
+private[r] class PrefixSpanWrapper private (val prefixSpan: PrefixSpan) {
--- End diff --
Is this wrapper needed? Can't we directly call `findFrequentSequentialPatterns`on `PrefixSpan` object?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #93184 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93184/testReport)** for PR 21710 at commit [`950241a`](https://github.com/apache/spark/commit/950241aa441b22330536e329b10dc386523fcdc3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #92592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92592/testReport)** for PR 21710 at commit [`5ed11e6`](https://github.com/apache/spark/commit/5ed11e67703dc7dfb23fb7ff68acffde33c13a30).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/657/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93184/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1032/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #93184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93184/testReport)** for PR 21710 at commit [`950241a`](https://github.com/apache/spark/commit/950241aa441b22330536e329b10dc386523fcdc3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203245118
--- Diff: R/pkg/R/mllib_fpm.R ---
@@ -154,3 +160,74 @@ setMethod("write.ml", signature(object = "FPGrowthModel", path = "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PrefixSpan
+#'
+#' A parallel PrefixSpan algorithm to mine frequent sequential patterns.
+#' \code{spark.prefixSpan} returns an instance of PrefixSpan.
+#' \code{spark.findFrequentSequentialPatterns} returns a complete set of frequent sequential
+#' patterns.
+#' For more details, see
+#' \href{https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html#prefixspan}{
+#' PrefixSpan}.
+#'
+#' @param minSupport Minimal support level.
+#' @param maxPatternLength Maximal pattern length.
+#' @param maxLocalProjDBSize Maximum number of items (including delimiters used in the internal
+#' storage format) allowed in a projected database before local
+#' processing.
+#' @param sequenceCol name of the sequence column in dataset.
+#' @param ... additional argument(s) passed to the method.
+#' @return \code{spark.prefixSpan} returns an instance of PrefixSpan
+#' @rdname spark.prefixSpan
+#' @name spark.prefixSpan
+#' @aliases spark.prefixSpan,ANY-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
+#' list(list(list(1L), list(3L, 2L), list(1L, 2L))),
+#' list(list(list(1L, 2L), list(5L))),
+#' list(list(list(6L)))), schema = c("sequence"))
+#' prefix_Span <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L,
+#' maxLocalProjDBSize = 32000000L)
+#' frequency <- spark.findFrequentSequentialPatterns(prefix_Span, df)
+#' showDF(frequency)
+#' }
+#' @note spark.prefixSpan since 2.4.0
+setMethod("spark.prefixSpan", signature(),
+ function(minSupport=0.1, maxPatternLength=10L,
+ maxLocalProjDBSize=32000000L, sequenceCol="sequence") {
+ if (!is.numeric(minSupport) || minSupport < 0) {
+ stop("minSupport should be a number with value >= 0.")
+ }
+ if (!is.integer(maxPatternLength) || maxPatternLength <= 0) {
+ stop("maxPatternLength should be a number with value > 0.")
+ }
+ if (!is.numeric(maxLocalProjDBSize) || maxLocalProjDBSize <= 0) {
+ stop("maxLocalProjDBSize should be a number with value > 0.")
+ }
+
+ jobj <- callJStatic("org.apache.spark.ml.r.PrefixSpanWrapper", "getPrefixSpan",
+ as.numeric(minSupport), as.integer(maxPatternLength),
+ as.numeric(maxLocalProjDBSize), as.character(sequenceCol))
+ new("PrefixSpan", jobj = jobj)
+ })
+
+# Find frequent sequential patterns.
+
+#' @param object a prefixSpan object.
+#' @param data A SparkDataFrame.
+#' @return A complete set of frequent sequential patterns in the input sequences of itemsets.
+#' The returned \code{SparkDataFrame} contains columns of sequence and corresponding
+#' frequency. The schema of it will be:
+#' \code{sequence: ArrayType(ArrayType(T))} (T is the item type)
+#' \code{freq: Long}
+#' @rdname spark.prefixSpan
--- End diff --
ditto here
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r223573856
--- Diff: R/pkg/R/mllib_fpm.R ---
@@ -154,3 +160,74 @@ setMethod("write.ml", signature(object = "FPGrowthModel", path = "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PrefixSpan
+#'
+#' A parallel PrefixSpan algorithm to mine frequent sequential patterns.
+#' \code{spark.prefixSpan} returns an instance of PrefixSpan.
+#' \code{spark.findFrequentSequentialPatterns} returns a complete set of frequent sequential
+#' patterns.
+#' For more details, see
+#' \href{https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html#prefixspan}{
+#' PrefixSpan}.
+#'
+#' @param minSupport Minimal support level.
+#' @param maxPatternLength Maximal pattern length.
+#' @param maxLocalProjDBSize Maximum number of items (including delimiters used in the internal
+#' storage format) allowed in a projected database before local
+#' processing.
+#' @param sequenceCol name of the sequence column in dataset.
+#' @param ... additional argument(s) passed to the method.
+#' @return \code{spark.prefixSpan} returns an instance of PrefixSpan
+#' @rdname spark.prefixSpan
+#' @name spark.prefixSpan
+#' @aliases spark.prefixSpan,ANY-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
+#' list(list(list(1L), list(3L, 2L), list(1L, 2L))),
+#' list(list(list(1L, 2L), list(5L))),
+#' list(list(list(6L)))), schema = c("sequence"))
+#' prefix_Span <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L,
+#' maxLocalProjDBSize = 32000000L)
+#' frequency <- spark.findFrequentSequentialPatterns(prefix_Span, df)
+#' showDF(frequency)
+#' }
+#' @note spark.prefixSpan since 3.0.0
+setMethod("spark.prefixSpan", signature(),
--- End diff --
sorry, I didn't see this before - an empty `signature()` is a bit usual
should we either
1) put one of these param as required (if they are required?) and its type into the signature, eg. `signature(minSupport = "numeric")`
2) instead of having `spark.prefixSpan`, take all of these param into `spark.findFrequentSequentialPatterns`
ie.
```
setMethod("spark.findFrequentSequentialPatterns",
signature(object = "PrefixSpan", data = "SparkDataFrame"),
function(object, data, minSupport = 0.1, maxPatternLength = 10L,
maxLocalProjDBSize = 32000000L, sequenceCol = "sequence") {
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r223572974
--- Diff: R/pkg/R/mllib_fpm.R ---
@@ -154,3 +160,74 @@ setMethod("write.ml", signature(object = "FPGrowthModel", path = "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PrefixSpan
+#'
+#' A parallel PrefixSpan algorithm to mine frequent sequential patterns.
+#' \code{spark.prefixSpan} returns an instance of PrefixSpan.
+#' \code{spark.findFrequentSequentialPatterns} returns a complete set of frequent sequential
+#' patterns.
+#' For more details, see
+#' \href{https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html#prefixspan}{
+#' PrefixSpan}.
+#'
+#' @param minSupport Minimal support level.
+#' @param maxPatternLength Maximal pattern length.
+#' @param maxLocalProjDBSize Maximum number of items (including delimiters used in the internal
+#' storage format) allowed in a projected database before local
+#' processing.
+#' @param sequenceCol name of the sequence column in dataset.
+#' @param ... additional argument(s) passed to the method.
+#' @return \code{spark.prefixSpan} returns an instance of PrefixSpan
+#' @rdname spark.prefixSpan
+#' @name spark.prefixSpan
+#' @aliases spark.prefixSpan,ANY-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
+#' list(list(list(1L), list(3L, 2L), list(1L, 2L))),
+#' list(list(list(1L, 2L), list(5L))),
+#' list(list(list(6L)))), schema = c("sequence"))
+#' prefix_Span <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L,
+#' maxLocalProjDBSize = 32000000L)
+#' frequency <- spark.findFrequentSequentialPatterns(prefix_Span, df)
+#' showDF(frequency)
+#' }
+#' @note spark.prefixSpan since 3.0.0
+setMethod("spark.prefixSpan", signature(),
+ function(minSupport=0.1, maxPatternLength=10L,
+ maxLocalProjDBSize=32000000L, sequenceCol="sequence") {
--- End diff --
nit: I think R style we put space around `=`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21710
actually, as of 2 hrs ago seems like we are skipping 2.5, to 3.0. we could wait a bit until that is merged
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r223572694
--- Diff: mllib/src/main/scala/org/apache/spark/ml/r/PrefixSpanWrapper.scala ---
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.r
+
+import org.apache.spark.ml.fpm.PrefixSpan
+
+private[r] object PrefixSpanWrapper {
+ def getPrefixSpan(
+ minSupport: Double,
+ maxPatternLength: Int,
+ maxLocalProjDBSize: Double,
--- End diff --
no native 64 bit int support in R. double is ok
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203229835
--- Diff: mllib/src/main/scala/org/apache/spark/ml/r/PrefixSpanWrapper.scala ---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.r
+
+import org.apache.spark.ml.fpm.PrefixSpan
+import org.apache.spark.sql.{DataFrame, Dataset}
+
+private[r] class PrefixSpanWrapper private (val prefixSpan: PrefixSpan) {
--- End diff --
You are right. I will remove
```
private[r] class PrefixSpanWrapper private (val prefixSpan: PrefixSpan) {
def findFrequentSequentialPatterns(dataset: Dataset[_]): DataFrame = {
prefixSpan.findFrequentSequentialPatterns(dataset)
}
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21710
@huaxingao do you want to bring this up to 2.5.0?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #97317 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97317/testReport)** for PR 21710 at commit [`0bab5ac`](https://github.com/apache/spark/commit/0bab5aca283bacdfe36ba1c669521df9d7ff81f3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92592/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92658/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203229794
--- Diff: R/pkg/tests/fulltests/test_mllib_fpm.R ---
@@ -82,4 +82,26 @@ test_that("spark.fpGrowth", {
})
+test_that("spark.prefixSpan", {
+ df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
+ list(list(list(1L), list(3L, 2L), list(1L, 2L))),
+ list(list(list(1L, 2L), list(5L))),
+ list(list(list(6L)))), schema = c("sequence"))
+ prefix_Span1 <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L,
+ maxLocalProjDBSize = 32000000L)
+ result1 <- spark.findFrequentSequentialPatterns(prefix_Span1, df)
+
+ expected_result <- createDataFrame(list(list(list(list(1L)), 3L),
+ list(list(list(3L)), 2L),
+ list(list(list(2L)), 3L),
+ list(list(list(1L, 2L)), 3L),
+ list(list(list(1L), list(3L)), 2L)),
+ schema = c("sequence", "freq"))
+ expect_equivalent(expected_result, result1)
+
+ prefix_Span2 <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L)
+ result2 <- spark.findFrequentSequentialPatterns(prefix_Span2, df)
+ expect_equivalent(expected_result, result2)
--- End diff --
Just trying to test that the default value (```maxLocalProjDBSize = 32000000L```) will be used if the parameter isn't set explicitly.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21710
merged to master
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r223568708
--- Diff: examples/src/main/python/ml/prefixspan_example.py ---
@@ -0,0 +1,48 @@
+#
--- End diff --
ditto for java and scala and python example
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r226875038
--- Diff: examples/src/main/python/ml/prefixspan_example.py ---
@@ -0,0 +1,48 @@
+#
--- End diff --
yes, ping me on that and I can merge quickly
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1059/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203148904
--- Diff: R/pkg/R/mllib_fpm.R ---
@@ -154,3 +160,74 @@ setMethod("write.ml", signature(object = "FPGrowthModel", path = "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PrefixSpan
+#'
+#' A parallel PrefixSpan algorithm to mine frequent sequential patterns.
+#' \code{spark.prefixSpan} returns an instance of PrefixSpan.
+#' \code{spark.findFrequentSequentialPatterns} returns a complete set of frequent sequential
+#' patterns.
+#' For more details, see
+#' \href{https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html#prefixspan}{
+#' PrefixSpan}.
+#'
+#' @param minSupport Minimal support level.
+#' @param maxPatternLength Maximal pattern length.
+#' @param maxLocalProjDBSize Maximum number of items (including delimiters used in the internal
+#' storage format) allowed in a projected database before local
+#' processing.
+#' @param sequenceCol name of the sequence column in dataset.
+#' @param ... additional argument(s) passed to the method.
+#' @return \code{spark.prefixSpan} returns an instance of PrefixSpan
+#' @rdname spark.prefixSpan
+#' @name spark.prefixSpan
+#' @aliases spark.prefixSpan,ANY-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
+#' list(list(list(1L), list(3L, 2L), list(1L, 2L))),
+#' list(list(list(1L, 2L), list(5L))),
+#' list(list(list(6L)))), schema = c("sequence"))
+#' prefix_Span <- spark.prefixSpan(minSupport = 0.5, maxPatternLength = 5L,
+#' maxLocalProjDBSize = 32000000L)
+#' frequency <- spark.findFrequentSequentialPatterns(prefix_Span, df)
+#' showDF(frequency)
+#' }
+#' @note spark.prefixSpan since 2.4.0
+setMethod("spark.prefixSpan", signature(),
+ function(minSupport=0.1, maxPatternLength=10L,
+ maxLocalProjDBSize=32000000L, sequenceCol="sequence") {
+ if (!is.numeric(minSupport) || minSupport < 0) {
+ stop("minSupport should be a number with value >= 0.")
+ }
+ if (!is.integer(maxPatternLength) || maxPatternLength <= 0) {
+ stop("maxPatternLength should be a number with value > 0.")
+ }
+ if (!is.numeric(maxLocalProjDBSize) || maxLocalProjDBSize <= 0) {
+ stop("maxLocalProjDBSize should be a number with value > 0.")
+ }
+
+ jobj <- callJStatic("org.apache.spark.ml.r.PrefixSpanWrapper", "getPrefixSpan",
+ as.numeric(minSupport), as.integer(maxPatternLength),
+ as.numeric(maxLocalProjDBSize), as.character(sequenceCol))
+ new("PrefixSpan", jobj = jobj)
+ })
+
+# Find frequent sequential patterns.
+
+#' @param object a prefixSpan object.
--- End diff --
`PrefixSpan` object?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3928/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #93145 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93145/testReport)** for PR 21710 at commit [`61f0de8`](https://github.com/apache/spark/commit/61f0de8c8a4544a6baabc85ab9abb8cd396528a5).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class JavaPrefixSpanExample `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203485926
--- Diff: R/pkg/R/generics.R ---
@@ -1415,6 +1415,13 @@ setGeneric("spark.freqItemsets", function(object) { standardGeneric("spark.freqI
#' @rdname spark.fpGrowth
setGeneric("spark.associationRules", function(object) { standardGeneric("spark.associationRules") })
+#' @rdname spark.prefixSpan
+setGeneric("spark.prefixSpan", function(...) { standardGeneric("spark.prefixSpan") })
+
+#' @rdname spark.prefixSpan
--- End diff --
Thanks @felixcheung. That's what I meant. :-)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21710
**[Test build #92658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92658/testReport)** for PR 21710 at commit [`d2e85f5`](https://github.com/apache/spark/commit/d2e85f5f30a89dbd12d5bcd4f4c5187aac956b87).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21710#discussion_r203481597
--- Diff: R/pkg/R/generics.R ---
@@ -1415,6 +1415,13 @@ setGeneric("spark.freqItemsets", function(object) { standardGeneric("spark.freqI
#' @rdname spark.fpGrowth
setGeneric("spark.associationRules", function(object) { standardGeneric("spark.associationRules") })
+#' @rdname spark.prefixSpan
+setGeneric("spark.prefixSpan", function(...) { standardGeneric("spark.prefixSpan") })
+
+#' @rdname spark.prefixSpan
--- End diff --
OK thanks. I guess I will keep both spark.prefixSpan and spark.findFrequentSequentialPatterns in the same rd.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21710
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1103/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93245/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21710
I think we missed the windows before the branch, I'll review in a few days
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org