You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by hhbyyh <gi...@git.apache.org> on 2017/03/01 23:59:20 UTC

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

GitHub user hhbyyh opened a pull request:

    https://github.com/apache/spark/pull/17130

    [SPARK-19791] [ML] Add doc and example for fpgrowth

    ## What changes were proposed in this pull request?
    
    Add a new section for fpm
    Add Example for FPGrowth in scala and Java
    
    ## How was this patch tested?
    
    local doc generation. 


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/hhbyyh/spark fpmdoc

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/17130.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #17130
    
----
commit fdce2404688fee1b22154258de5d85f0cee8aa4b
Author: Yuhao Yang <yu...@intel.com>
Date:   2017-03-01T23:47:53Z

    fpm doc

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #74383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74383/testReport)** for PR 17130 at commit [`9ce0093`](https://github.com/apache/spark/commit/9ce00930ea18c7bb8fe0cc59b98f6ece34d20311).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/17130#discussion_r113129421

--- Diff: docs/ml-frequent-pattern-mining.md ---
@@ -0,0 +1,87 @@
+---
+layout: global
+title: Frequent Pattern Mining
+displayTitle: Frequent Pattern Mining
+---
+
+Mining frequent items, itemsets, subsequences, or other substructures is usually among the
+first steps to analyze a large-scale dataset, which has been an active research topic in
+data mining for years.
+We refer users to Wikipedia's [association rule learning](http://en.wikipedia.org/wiki/Association_rule_learning)
+for more information.
+
+**Table of Contents**
+
+* This will become a table of contents (this text will be scraped).
+{:toc}
+
+## FP-Growth
+
+The FP-growth algorithm is described in the paper
+[Han et al., Mining frequent patterns without candidate generation](http://dx.doi.org/10.1145/335191.335372),
+where "FP" stands for frequent pattern.
+Given a dataset of transactions, the first step of FP-growth is to calculate item frequencies and identify frequent items.
+Different from [Apriori-like](http://en.wikipedia.org/wiki/Apriori_algorithm) algorithms designed for the same purpose,
+the second step of FP-growth uses a suffix tree (FP-tree) structure to encode transactions without generating candidate sets
+explicitly, which are usually expensive to generate.
+After the second step, the frequent itemsets can be extracted from the FP-tree.
+In `spark.mllib`, we implemented a parallel version of FP-growth called PFP,
+as described in [Li et al., PFP: Parallel FP-growth for query recommendation](http://dx.doi.org/10.1145/1454008.1454027).
+PFP distributes the work of growing FP-trees based on the suffixes of transactions,
+and hence is more scalable than a single-machine implementation.
+We refer users to the papers for more details.
+
+`spark.ml`'s FP-growth implementation takes the following (hyper-)parameters:
+
+* `minSupport`: the minimum support for an itemset to be identified as frequent.
+ For example, if an item appears 3 out of 5 transactions, it has a support of 3/5=0.6.
+* `minConfidence`: minimum confidence for generating Association Rule. Confidence is an indication of how often an
+ association rule has been found to be true. For example, if in the transactions itemset `X` appears 4 times, `X`
+ and `Y` co-occur only 2 times, the confidence for the rule `X => Y` is then 2/4 = 0.5. The parameter will not
+ affect the mining for frequent itemsets, but specify the minimum confidence for generating association rules
+ from frequent itemsets.
+* `numPartitions`: the number of partitions used to distribute the work. By default the param is not set, and
+ number of partitions of the input dataset is used.
+
+The `FPGrowthModel` provides:
+
+* `freqItemsets`: frequent itemsets in the format of DataFrame("items"[Array], "freq"[Long])
+* `associationRules`: association rules generated with confidence above `minConfidence`, in the format of
+ DataFrame("antecedent"[Array], "consequent"[Array], "confidence"[Double]).
+* `transform`: For each transaction in itemsCol, the `transform` method will compare its items against the antecedents
--- End diff --

`itemsCol`?

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BryanCutler <gi...@git.apache.org>.

Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104006350
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala ---
    @@ -0,0 +1,71 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.examples.ml
    +
    +// scalastyle:off println
    +
    +// $example on$
    +import org.apache.spark.ml.fpm.FPGrowth
    +// $example off$
    +import org.apache.spark.sql.SparkSession
    +
    +/**
    + * An example demonstrating FP-Growth.
    + * Run with
    + * {{{
    + * bin/run-example ml.FPGrowthExample
    + * }}}
    + */
    +object FPGrowthExample {
    +
    +  def main(args: Array[String]): Unit = {
    +
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +    import spark.implicits._
    +
    +    // $example on$
    +    // Loads data.
    --- End diff --
    
    I think this comment is pretty obvious, you can probably remove it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    right, how are we on this? let's get this ready soon and merge?
    could you also add reference to the R example which is merged yesterday.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #74565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74565/testReport)** for PR 17130 at commit [`fa4c734`](https://github.com/apache/spark/commit/fa4c734b31cc5aaa7ded18f4649d71d7249902ec).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r113846530
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -268,12 +269,8 @@ class FPGrowthModel private[ml] (
         val predictUDF = udf((items: Seq[_]) => {
           if (items != null) {
             val itemset = items.toSet
    -        brRules.value.flatMap(rule =>
    -          if (items != null && rule._1.forall(item => itemset.contains(item))) {
    -            rule._2.filter(item => !itemset.contains(item))
    -          } else {
    -            Seq.empty
    -          }).distinct
    +        brRules.value.filter(_._1.forall(itemset.contains))
    +          .flatMap(_._2.filter(!itemset.contains(_))).distinct
    --- End diff --
    
    let's update the PR/JIRA if code change is required for the doc change.
    otherwise, let's leave code change as a separate PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74565/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BenFradet <gi...@git.apache.org>.

Github user BenFradet commented on a diff in the pull request:

https://github.com/apache/spark/pull/17130#discussion_r108450311

--- Diff: docs/ml-frequent-pattern-mining.md ---
@@ -0,0 +1,75 @@
+---
+layout: global
+title: Frequent Pattern Mining
+displayTitle: Frequent Pattern Mining
+---
+
+Mining frequent items, itemsets, subsequences, or other substructures is usually among the
+first steps to analyze a large-scale dataset, which has been an active research topic in
+data mining for years.
+We refer users to Wikipedia's [association rule learning](http://en.wikipedia.org/wiki/Association_rule_learning)
+for more information.
+
+**Table of Contents**
+
+* This will become a table of contents (this text will be scraped).
+{:toc}
+
+## FP-Growth
+
+The FP-growth algorithm is described in the paper
+[Han et al., Mining frequent patterns without candidate generation](http://dx.doi.org/10.1145/335191.335372),
+where "FP" stands for frequent pattern.
+Given a dataset of transactions, the first step of FP-growth is to calculate item frequencies and identify frequent items.
+Different from [Apriori-like](http://en.wikipedia.org/wiki/Apriori_algorithm) algorithms designed for the same purpose,
+the second step of FP-growth uses a suffix tree (FP-tree) structure to encode transactions without generating candidate sets
+explicitly, which are usually expensive to generate.
+After the second step, the frequent itemsets can be extracted from the FP-tree.
+In `spark.mllib`, we implemented a parallel version of FP-growth called PFP,
+as described in [Li et al., PFP: Parallel FP-growth for query recommendation](http://dx.doi.org/10.1145/1454008.1454027).
+PFP distributes the work of growing FP-trees based on the suffices of transactions,
+and hence more scalable than a single-machine implementation.
+We refer users to the papers for more details.
+
+`spark.ml`'s FP-growth implementation takes the following (hyper-)parameters:
+
+* `minSupport`: the minimum support for an itemset to be identified as frequent.
+ For example, if an item appears 3 out of 5 transactions, it has a support of 3/5=0.6.
+* `minConfidence`: minimum confidence for generating Association Rule. The parameter will not affect the mining
+ for frequent itemsets,, but specify the minimum confidence for generating association rules from frequent itemsets.
--- End diff --

also, there are two commas after itemsets

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Please hold on merging this until https://github.com/apache/spark/pull/17321 is resolved.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r113366111
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -268,12 +269,8 @@ class FPGrowthModel private[ml] (
         val predictUDF = udf((items: Seq[_]) => {
           if (items != null) {
             val itemset = items.toSet
    -        brRules.value.flatMap(rule =>
    -          if (items != null && rule._1.forall(item => itemset.contains(item))) {
    -            rule._2.filter(item => !itemset.contains(item))
    -          } else {
    -            Seq.empty
    -          }).distinct
    +        brRules.value.filter(_._1.forall(itemset.contains))
    +          .flatMap(_._2.filter(!itemset.contains(_))).distinct
    --- End diff --
    
    right, 2 things - first just calling out while the PR says doc changes there is this one code change here.
    second, before this code was checking `items != null` do we need not consider that now?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r114002784
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -268,12 +269,8 @@ class FPGrowthModel private[ml] (
         val predictUDF = udf((items: Seq[_]) => {
           if (items != null) {
             val itemset = items.toSet
    -        brRules.value.flatMap(rule =>
    -          if (items != null && rule._1.forall(item => itemset.contains(item))) {
    -            rule._2.filter(item => !itemset.contains(item))
    -          } else {
    -            Seq.empty
    -          }).distinct
    +        brRules.value.filter(_._1.forall(itemset.contains))
    +          .flatMap(_._2.filter(!itemset.contains(_))).distinct
    --- End diff --
    
    I guess that's the right way. I will revert the code change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BenFradet <gi...@git.apache.org>.

Github user BenFradet commented on a diff in the pull request:

https://github.com/apache/spark/pull/17130#discussion_r108449521

I don't think this really explains what `transform` does or maybe it's just me?

I would have said something like:

>The transform method will produce `predictionCol` containing all the consequents of the association rules containing the items in `itemsCol` as their antecedents. The prediction column...

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by jkbradley <gi...@git.apache.org>.

Github user jkbradley commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    LGTM, thanks for adding this!  @felixcheung OK with merging?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BryanCutler <gi...@git.apache.org>.

Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104009419
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala ---
    @@ -0,0 +1,71 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.examples.ml
    +
    +// scalastyle:off println
    +
    +// $example on$
    +import org.apache.spark.ml.fpm.FPGrowth
    +// $example off$
    +import org.apache.spark.sql.SparkSession
    +
    +/**
    + * An example demonstrating FP-Growth.
    + * Run with
    + * {{{
    + * bin/run-example ml.FPGrowthExample
    + * }}}
    + */
    +object FPGrowthExample {
    +
    +  def main(args: Array[String]): Unit = {
    +
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +    import spark.implicits._
    +
    +    // $example on$
    +    // Loads data.
    +    val dataset = spark.createDataset(Seq(
    +      "1 2 5",
    +      "1 2 3 5",
    +      "1 2")
    +    ).map(t => t.split(" ")).toDF("features")
    +
    +    // Trains a FPGrowth model.
    --- End diff --
    
    nit: technically it is just the line below calling `fit` that trains the model, I would move this comment down or just take it out


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r113130099
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -268,12 +269,8 @@ class FPGrowthModel private[ml] (
         val predictUDF = udf((items: Seq[_]) => {
           if (items != null) {
             val itemset = items.toSet
    -        brRules.value.flatMap(rule =>
    -          if (items != null && rule._1.forall(item => itemset.contains(item))) {
    -            rule._2.filter(item => !itemset.contains(item))
    -          } else {
    -            Seq.empty
    -          }).distinct
    +        brRules.value.filter(_._1.forall(itemset.contains))
    +          .flatMap(_._2.filter(!itemset.contains(_))).distinct
    --- End diff --
    
    so we don't need to handle null item?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BryanCutler <gi...@git.apache.org>.

Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104006222
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala ---
    @@ -0,0 +1,71 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.examples.ml
    +
    +// scalastyle:off println
    +
    +// $example on$
    +import org.apache.spark.ml.fpm.FPGrowth
    +// $example off$
    +import org.apache.spark.sql.SparkSession
    +
    +/**
    + * An example demonstrating FP-Growth.
    + * Run with
    + * {{{
    + * bin/run-example ml.FPGrowthExample
    + * }}}
    + */
    +object FPGrowthExample {
    +
    +  def main(args: Array[String]): Unit = {
    +
    --- End diff --
    
    nit: remove blank line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #75947 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75947/testReport)** for PR 17130 at commit [`2b1efb3`](https://github.com/apache/spark/commit/2b1efb34493272abf46ec5142aa26117d83ed63b).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #75273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75273/testReport)** for PR 17130 at commit [`c957ba5`](https://github.com/apache/spark/commit/c957ba5821d2bb6004c41a2e55c651d8c156de09).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

https://github.com/apache/spark/pull/17130#discussion_r113782786

Please refer to https://issues.apache.org/jira/browse/SPARK-19899

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #75112 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75112/testReport)** for PR 17130 at commit [`9e908d0`](https://github.com/apache/spark/commit/9e908d0d88ce15781b7de87f4e24c6b8ba9dc2a4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76287/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r113220960
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -268,12 +269,8 @@ class FPGrowthModel private[ml] (
         val predictUDF = udf((items: Seq[_]) => {
           if (items != null) {
             val itemset = items.toSet
    -        brRules.value.flatMap(rule =>
    -          if (items != null && rule._1.forall(item => itemset.contains(item))) {
    -            rule._2.filter(item => !itemset.contains(item))
    -          } else {
    -            Seq.empty
    -          }).distinct
    +        brRules.value.filter(_._1.forall(itemset.contains))
    +          .flatMap(_._2.filter(!itemset.contains(_))).distinct
    --- End diff --
    
    Hi @felixcheung , can you be more specific about the null item that concerns you? Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #76287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76287/testReport)** for PR 17130 at commit [`ea3b973`](https://github.com/apache/spark/commit/ea3b9733329b7f50985d99001c179b040f802875).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r113783993
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -268,12 +269,8 @@ class FPGrowthModel private[ml] (
         val predictUDF = udf((items: Seq[_]) => {
           if (items != null) {
             val itemset = items.toSet
    -        brRules.value.flatMap(rule =>
    -          if (items != null && rule._1.forall(item => itemset.contains(item))) {
    -            rule._2.filter(item => !itemset.contains(item))
    -          } else {
    -            Seq.empty
    -          }).distinct
    +        brRules.value.filter(_._1.forall(itemset.contains))
    +          .flatMap(_._2.filter(!itemset.contains(_))).distinct
    --- End diff --
    
    `items != null` is already checked at two lines above. 
    Please refer to the comments in the PR for the history of the code change. I can update title to include the code change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #75947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75947/testReport)** for PR 17130 at commit [`2b1efb3`](https://github.com/apache/spark/commit/2b1efb34493272abf46ec5142aa26117d83ed63b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

https://github.com/apache/spark/pull/17130#discussion_r114002856

sure.

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75273/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73719/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by jkbradley <gi...@git.apache.org>.

Github user jkbradley commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    I'll be happy to help get this merged now that the column renaming is done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Thanks for the review. I'll wait for https://github.com/apache/spark/pull/17283 to be merged first.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by jkbradley <gi...@git.apache.org>.

Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r105702725
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -56,8 +56,8 @@ private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with HasPre
       def getMinSupport: Double = $(minSupport)
     
       /**
    -   * Number of partitions (>=1) used by parallel FP-growth. By default the param is not set, and
    -   * partition number of the input dataset is used.
    +   * Number of partitions (positive) used by parallel FP-growth. By default the param is not set,
    --- End diff --
    
    Let's just use ">=1" but figure out how to escape the characters for javadoc.  We'll want to do that long-term.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #75714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75714/testReport)** for PR 17130 at commit [`0fb5a87`](https://github.com/apache/spark/commit/0fb5a8757f351649a1d6648e74f126a720b37b21).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BryanCutler <gi...@git.apache.org>.

Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104008090
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala ---
    @@ -0,0 +1,71 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.examples.ml
    +
    +// scalastyle:off println
    +
    +// $example on$
    +import org.apache.spark.ml.fpm.FPGrowth
    +// $example off$
    +import org.apache.spark.sql.SparkSession
    +
    +/**
    + * An example demonstrating FP-Growth.
    + * Run with
    + * {{{
    + * bin/run-example ml.FPGrowthExample
    + * }}}
    + */
    +object FPGrowthExample {
    +
    +  def main(args: Array[String]): Unit = {
    +
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +    import spark.implicits._
    +
    +    // $example on$
    +    // Loads data.
    +    val dataset = spark.createDataset(Seq(
    +      "1 2 5",
    +      "1 2 3 5",
    +      "1 2")
    +    ).map(t => t.split(" ")).toDF("features")
    +
    +    // Trains a FPGrowth model.
    +    val fpgrowth = new FPGrowth().setMinSupport(0.5).setMinConfidence(0.6)
    +    val model = fpgrowth.fit(dataset)
    +
    +    // get frequent itemsets.
    +    model.freqItemsets.show()
    +
    +    // get generated association rules.
    +    model.associationRules.show()
    +
    +    // transform examines the input items against all the association rules and summarize the
    +    // consequents as prediction
    +    model.transform(dataset).show()
    +
    --- End diff --
    
    nit: remove blank line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #75409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75409/testReport)** for PR 17130 at commit [`e9b090a`](https://github.com/apache/spark/commit/e9b090ac07291de9e09c81a0bb371fffb3384a4f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75409/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by jkbradley <gi...@git.apache.org>.

Github user jkbradley commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    The updated transform looks good; thanks for pinging!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #73719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73719/testReport)** for PR 17130 at commit [`fdce240`](https://github.com/apache/spark/commit/fdce2404688fee1b22154258de5d85f0cee8aa4b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74994/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #75409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75409/testReport)** for PR 17130 at commit [`e9b090a`](https://github.com/apache/spark/commit/e9b090ac07291de9e09c81a0bb371fffb3384a4f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r113129720
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -82,8 +81,8 @@ private[fpm] trait FPGrowthParams extends Params with HasPredictionCol {
       def getNumPartitions: Int = $(numPartitions)
     
       /**
    -   * Minimal confidence for generating Association Rule.
    -   * Note that minConfidence has no effect during fitting.
    +   * Minimal confidence for generating Association Rule. MinConfidence will not affect the mining
    --- End diff --
    
    lower case `minConfidence`? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    @felixcheung, reverted the code change of `transform` as requested. Please check the update. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #75273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75273/testReport)** for PR 17130 at commit [`c957ba5`](https://github.com/apache/spark/commit/c957ba5821d2bb6004c41a2e55c651d8c156de09).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r112252772
  
    --- Diff: docs/ml-frequent-pattern-mining.md ---
    @@ -0,0 +1,80 @@
    +---
    +layout: global
    +title: Frequent Pattern Mining
    +displayTitle: Frequent Pattern Mining
    +---
    +
    +Mining frequent items, itemsets, subsequences, or other substructures is usually among the
    +first steps to analyze a large-scale dataset, which has been an active research topic in
    +data mining for years.
    +We refer users to Wikipedia's [association rule learning](http://en.wikipedia.org/wiki/Association_rule_learning)
    +for more information.
    +
    +**Table of Contents**
    +
    +* This will become a table of contents (this text will be scraped).
    +{:toc}
    +
    +## FP-Growth
    +
    +The FP-growth algorithm is described in the paper
    +[Han et al., Mining frequent patterns without candidate generation](http://dx.doi.org/10.1145/335191.335372),
    +where "FP" stands for frequent pattern.
    +Given a dataset of transactions, the first step of FP-growth is to calculate item frequencies and identify frequent items.
    +Different from [Apriori-like](http://en.wikipedia.org/wiki/Apriori_algorithm) algorithms designed for the same purpose,
    +the second step of FP-growth uses a suffix tree (FP-tree) structure to encode transactions without generating candidate sets
    +explicitly, which are usually expensive to generate.
    +After the second step, the frequent itemsets can be extracted from the FP-tree.
    +In `spark.mllib`, we implemented a parallel version of FP-growth called PFP,
    +as described in [Li et al., PFP: Parallel FP-growth for query recommendation](http://dx.doi.org/10.1145/1454008.1454027).
    +PFP distributes the work of growing FP-trees based on the suffixes of transactions,
    +and hence is more scalable than a single-machine implementation.
    +We refer users to the papers for more details.
    +
    +`spark.ml`'s FP-growth implementation takes the following (hyper-)parameters:
    +
    +* `minSupport`: the minimum support for an itemset to be identified as frequent.
    +  For example, if an item appears 3 out of 5 transactions, it has a support of 3/5=0.6.
    +* `minConfidence`: minimum confidence for generating Association Rule. Confidence is an indication of how often an
    +  association rule has been found to be true. For example, if in the transactions itemset `X` appears 4 times, `X`
    +  and `Y` co-occur only 2 times, the confidence for the rule `X => Y` is then 2/4 = 0.5. The parameter will not
    +  affect the mining for frequent itemsets, but specify the minimum confidence for generating association rules
    +  from frequent itemsets.
    +* `numPartitions`: the number of partitions used to distribute the work. By default the param is not set, and
    +  number of partitions of the input dataset is used.
    +
    +The `FPGrowthModel` provides:
    +
    +* `freqItemsets`: frequent itemsets in the format of DataFrame("items"[Array], "freq"[Long])
    +* `associationRules`: association rules generated with confidence above `minConfidence`, in the format of 
    +  DataFrame("antecedent"[Array], "consequent"[Array], "confidence"[Double]).
    +* `transform`: For each transaction in itemsCol, the `transform` method will compare its items against the antecedents
    +  of each association rule. If the record contains all the antecedents of a specific association rule, the rule
    +  will be considered as applicable and its consequents will be added to the prediction result. The transform
    +  method will summarize the consequents from all the applicable rules as prediction. The prediction column has
    +  the same data type as `itemsCol` and does not contain existing items in the `itemsCol`.
    +
    +
    +**Examples**
    +
    +<div class="codetabs">
    +
    +<div data-lang="scala" markdown="1">
    +Refer to the [Scala API docs](api/scala/index.html#org.apache.spark.ml.fpm.FPGrowth) for more details.
    +
    +{% include_example scala/org/apache/spark/examples/ml/FPGrowthExample.scala %}
    +</div>
    +
    +<div data-lang="java" markdown="1">
    +Refer to the [Java API docs](api/java/org/apache/spark/ml/fpm/FPGrowth.html) for more details.
    +
    +{% include_example java/org/apache/spark/examples/ml/JavaFPGrowthExample.java %}
    +</div>
    +
    +<div data-lang="python" markdown="1">
    +Refer to the [Python API docs](api/python/pyspark.ml.html#pyspark.ml.fpm.FPGrowth) for more details.
    +
    +{% include_example python/ml/fpgrowth_example.py %}
    +</div>
    --- End diff --
    
    add R please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    merged to master/2.2


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #3599 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3599/testReport)** for PR 17130 at commit [`9ce0093`](https://github.com/apache/spark/commit/9ce00930ea18c7bb8fe0cc59b98f6ece34d20311).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #76287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76287/testReport)** for PR 17130 at commit [`ea3b973`](https://github.com/apache/spark/commit/ea3b9733329b7f50985d99001c179b040f802875).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #74636 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74636/testReport)** for PR 17130 at commit [`de1bfc8`](https://github.com/apache/spark/commit/de1bfc8eb48015ea629dea5bdc72ba913b76d234).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BryanCutler <gi...@git.apache.org>.

Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104007438
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala ---
    @@ -0,0 +1,71 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.examples.ml
    +
    +// scalastyle:off println
    +
    +// $example on$
    +import org.apache.spark.ml.fpm.FPGrowth
    +// $example off$
    +import org.apache.spark.sql.SparkSession
    +
    +/**
    + * An example demonstrating FP-Growth.
    + * Run with
    + * {{{
    + * bin/run-example ml.FPGrowthExample
    + * }}}
    + */
    +object FPGrowthExample {
    +
    +  def main(args: Array[String]): Unit = {
    +
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +    import spark.implicits._
    +
    +    // $example on$
    +    // Loads data.
    +    val dataset = spark.createDataset(Seq(
    +      "1 2 5",
    +      "1 2 3 5",
    +      "1 2")
    +    ).map(t => t.split(" ")).toDF("features")
    +
    +    // Trains a FPGrowth model.
    +    val fpgrowth = new FPGrowth().setMinSupport(0.5).setMinConfidence(0.6)
    +    val model = fpgrowth.fit(dataset)
    +
    +    // get frequent itemsets.
    +    model.freqItemsets.show()
    +
    +    // get generated association rules.
    --- End diff --
    
    same as comment above


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75112/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #74636 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74636/testReport)** for PR 17130 at commit [`de1bfc8`](https://github.com/apache/spark/commit/de1bfc8eb48015ea629dea5bdc72ba913b76d234).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #75112 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75112/testReport)** for PR 17130 at commit [`9e908d0`](https://github.com/apache/spark/commit/9e908d0d88ce15781b7de87f4e24c6b8ba9dc2a4).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104009914
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -56,8 +56,8 @@ private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with HasPre
       def getMinSupport: Double = $(minSupport)
     
       /**
    -   * Number of partitions (>=1) used by parallel FP-growth. By default the param is not set, and
    -   * partition number of the input dataset is used.
    +   * Number of partitions (positive) used by parallel FP-growth. By default the param is not set,
    --- End diff --
    
    I presume to fix a javadoc error because angle brackets are read as opening an HTML tag


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #73790 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73790/testReport)** for PR 17130 at commit [`ca12877`](https://github.com/apache/spark/commit/ca12877c7f7e224268e145c3e8c4c37413596d66).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BryanCutler <gi...@git.apache.org>.

Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104008468
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -56,8 +56,8 @@ private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with HasPre
       def getMinSupport: Double = $(minSupport)
     
       /**
    -   * Number of partitions (>=1) used by parallel FP-growth. By default the param is not set, and
    -   * partition number of the input dataset is used.
    +   * Number of partitions (positive) used by parallel FP-growth. By default the param is not set,
    --- End diff --
    
    why change to "positive", I think it was clearer before


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BenFradet <gi...@git.apache.org>.

Github user BenFradet commented on a diff in the pull request:

https://github.com/apache/spark/pull/17130#discussion_r108442407

It might be good to give an example for confidence as well since one has been given for support

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #74565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74565/testReport)** for PR 17130 at commit [`fa4c734`](https://github.com/apache/spark/commit/fa4c734b31cc5aaa7ded18f4649d71d7249902ec).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r110990099
  
    --- Diff: examples/src/main/python/ml/fpgrowth_example.py ---
    @@ -0,0 +1,48 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +# $example on$
    +from pyspark.ml.fpm import FPGrowth
    +# $example off$
    +from pyspark.sql import SparkSession
    +
    +"""
    +An example demonstrating FPGrowth.
    +Run with:
    +  bin/spark-submit examples/src/main/python/ml/fpgrowth_example.py
    +"""
    +
    +if __name__ == "__main__":
    +    spark = SparkSession\
    +        .builder\
    +        .appName("FPGrowthExample")\
    +        .getOrCreate()
    +
    +    # $example on$
    +    df = spark.createDataFrame([
    +        (0, [1, 2, 5]),
    +        (1, [1, 2, 3, 5]),
    +        (2, [1, 2])
    +    ], ["id", "items"])
    +
    +    fpGrowth = FPGrowth(itemsCol="items", minSupport=0.5, minConfidence=0.6)
    +    fpGrowthModel = fpGrowth.fit(df)
    +
    --- End diff --
    
    definitely. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r114002811
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -82,8 +81,8 @@ private[fpm] trait FPGrowthParams extends Params with HasPredictionCol {
       def getNumPartitions: Int = $(numPartitions)
     
       /**
    -   * Minimal confidence for generating Association Rule.
    -   * Note that minConfidence has no effect during fitting.
    +   * Minimal confidence for generating Association Rule. MinConfidence will not affect the mining
    --- End diff --
    
    got it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BryanCutler <gi...@git.apache.org>.

Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104006027
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala ---
    @@ -0,0 +1,71 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.examples.ml
    +
    +// scalastyle:off println
    +
    +// $example on$
    +import org.apache.spark.ml.fpm.FPGrowth
    +// $example off$
    +import org.apache.spark.sql.SparkSession
    +
    +/**
    + * An example demonstrating FP-Growth.
    + * Run with
    + * {{{
    + * bin/run-example ml.FPGrowthExample
    + * }}}
    + */
    +object FPGrowthExample {
    +
    --- End diff --
    
    nit: remove blank line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BenFradet <gi...@git.apache.org>.

Github user BenFradet commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r108441411
  
    --- Diff: docs/ml-frequent-pattern-mining.md ---
    @@ -0,0 +1,75 @@
    +---
    +layout: global
    +title: Frequent Pattern Mining
    +displayTitle: Frequent Pattern Mining
    +---
    +
    +Mining frequent items, itemsets, subsequences, or other substructures is usually among the
    +first steps to analyze a large-scale dataset, which has been an active research topic in
    +data mining for years.
    +We refer users to Wikipedia's [association rule learning](http://en.wikipedia.org/wiki/Association_rule_learning)
    +for more information.
    +
    +**Table of Contents**
    +
    +* This will become a table of contents (this text will be scraped).
    +{:toc}
    +
    +## FP-Growth
    +
    +The FP-growth algorithm is described in the paper
    +[Han et al., Mining frequent patterns without candidate generation](http://dx.doi.org/10.1145/335191.335372),
    +where "FP" stands for frequent pattern.
    +Given a dataset of transactions, the first step of FP-growth is to calculate item frequencies and identify frequent items.
    +Different from [Apriori-like](http://en.wikipedia.org/wiki/Apriori_algorithm) algorithms designed for the same purpose,
    +the second step of FP-growth uses a suffix tree (FP-tree) structure to encode transactions without generating candidate sets
    +explicitly, which are usually expensive to generate.
    +After the second step, the frequent itemsets can be extracted from the FP-tree.
    +In `spark.mllib`, we implemented a parallel version of FP-growth called PFP,
    +as described in [Li et al., PFP: Parallel FP-growth for query recommendation](http://dx.doi.org/10.1145/1454008.1454027).
    +PFP distributes the work of growing FP-trees based on the suffices of transactions,
    +and hence more scalable than a single-machine implementation.
    --- End diff --
    
    is more scalable


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BryanCutler <gi...@git.apache.org>.

Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104006906
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala ---
    @@ -0,0 +1,71 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.examples.ml
    +
    +// scalastyle:off println
    +
    +// $example on$
    +import org.apache.spark.ml.fpm.FPGrowth
    +// $example off$
    +import org.apache.spark.sql.SparkSession
    +
    +/**
    + * An example demonstrating FP-Growth.
    + * Run with
    + * {{{
    + * bin/run-example ml.FPGrowthExample
    + * }}}
    + */
    +object FPGrowthExample {
    +
    +  def main(args: Array[String]): Unit = {
    +
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +    import spark.implicits._
    +
    +    // $example on$
    +    // Loads data.
    +    val dataset = spark.createDataset(Seq(
    +      "1 2 5",
    +      "1 2 3 5",
    +      "1 2")
    +    ).map(t => t.split(" ")).toDF("features")
    --- End diff --
    
    I think it is better to explicitly declare the data instead of manipulating strings, that way it is very clear what the input data is for the example.  like this
    ```
    val dataset = spark.createDataset(Seq(
          Array(1, 2, 5),
          Array(1, 2, 3, 5),
          Array(1, 2))
        ).toDF("features")
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BenFradet <gi...@git.apache.org>.

Github user BenFradet commented on a diff in the pull request:

https://github.com/apache/spark/pull/17130#discussion_r108613380

even better :+1:

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104039923
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala ---
    @@ -0,0 +1,71 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.examples.ml
    +
    +// scalastyle:off println
    +
    +// $example on$
    +import org.apache.spark.ml.fpm.FPGrowth
    +// $example off$
    +import org.apache.spark.sql.SparkSession
    +
    +/**
    + * An example demonstrating FP-Growth.
    + * Run with
    + * {{{
    + * bin/run-example ml.FPGrowthExample
    + * }}}
    + */
    +object FPGrowthExample {
    +
    +  def main(args: Array[String]): Unit = {
    +
    --- End diff --
    
    Thanks, I'll remove this line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74636/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r113846602
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -82,8 +81,8 @@ private[fpm] trait FPGrowthParams extends Params with HasPredictionCol {
       def getNumPartitions: Int = $(numPartitions)
     
       /**
    -   * Minimal confidence for generating Association Rule.
    -   * Note that minConfidence has no effect during fitting.
    +   * Minimal confidence for generating Association Rule. MinConfidence will not affect the mining
    --- End diff --
    
    ping


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BryanCutler <gi...@git.apache.org>.

Github user BryanCutler commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104007323
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala ---
    @@ -0,0 +1,71 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.examples.ml
    +
    +// scalastyle:off println
    +
    +// $example on$
    +import org.apache.spark.ml.fpm.FPGrowth
    +// $example off$
    +import org.apache.spark.sql.SparkSession
    +
    +/**
    + * An example demonstrating FP-Growth.
    + * Run with
    + * {{{
    + * bin/run-example ml.FPGrowthExample
    + * }}}
    + */
    +object FPGrowthExample {
    +
    +  def main(args: Array[String]): Unit = {
    +
    +    val spark = SparkSession
    +      .builder
    +      .appName(s"${this.getClass.getSimpleName}")
    +      .getOrCreate()
    +    import spark.implicits._
    +
    +    // $example on$
    +    // Loads data.
    +    val dataset = spark.createDataset(Seq(
    +      "1 2 5",
    +      "1 2 3 5",
    +      "1 2")
    +    ).map(t => t.split(" ")).toDF("features")
    +
    +    // Trains a FPGrowth model.
    +    val fpgrowth = new FPGrowth().setMinSupport(0.5).setMinConfidence(0.6)
    +    val model = fpgrowth.fit(dataset)
    +
    +    // get frequent itemsets.
    --- End diff --
    
    should say "Display frequent itemsets."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Updated with Python example.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104036137
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -56,8 +56,8 @@ private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with HasPre
       def getMinSupport: Double = $(minSupport)
     
       /**
    -   * Number of partitions (>=1) used by parallel FP-growth. By default the param is not set, and
    -   * partition number of the input dataset is used.
    +   * Number of partitions (positive) used by parallel FP-growth. By default the param is not set,
    --- End diff --
    
    Yes. That's the reason. But still I'm getting some java doc error after merging code. Looking into it..



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #74994 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74994/testReport)** for PR 17130 at commit [`9fef280`](https://github.com/apache/spark/commit/9fef280751378dbeaa843c673fd962192320a5b1).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #73719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73719/testReport)** for PR 17130 at commit [`fdce240`](https://github.com/apache/spark/commit/fdce2404688fee1b22154258de5d85f0cee8aa4b).
     * This patch **fails MiMa tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `public class JavaFPGrowthExample `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73790/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r105518166
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -56,8 +56,8 @@ private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with HasPre
       def getMinSupport: Double = $(minSupport)
     
       /**
    -   * Number of partitions (>=1) used by parallel FP-growth. By default the param is not set, and
    -   * partition number of the input dataset is used.
    +   * Number of partitions (positive) used by parallel FP-growth. By default the param is not set,
    --- End diff --
    
    I don't think the error is related to this change. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by zero323 <gi...@git.apache.org>.

Github user zero323 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r110540400
  
    --- Diff: examples/src/main/python/ml/fpgrowth_example.py ---
    @@ -0,0 +1,48 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +# $example on$
    +from pyspark.ml.fpm import FPGrowth
    +# $example off$
    +from pyspark.sql import SparkSession
    +
    +"""
    +An example demonstrating FPGrowth.
    +Run with:
    +  bin/spark-submit examples/src/main/python/ml/fpgrowth_example.py
    +"""
    +
    +if __name__ == "__main__":
    +    spark = SparkSession\
    +        .builder\
    +        .appName("FPGrowthExample")\
    +        .getOrCreate()
    +
    +    # $example on$
    +    df = spark.createDataFrame([
    +        (0, [1, 2, 5]),
    +        (1, [1, 2, 3, 5]),
    +        (2, [1, 2])
    +    ], ["id", "items"])
    +
    +    fpGrowth = FPGrowth(itemsCol="items", minSupport=0.5, minConfidence=0.6)
    +    fpGrowthModel = fpGrowth.fit(df)
    +
    --- End diff --
    
    Can we  `associationRules` example? After all this is the biggest advantage for the Python user.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r104010510
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -240,12 +240,13 @@ class FPGrowthModel private[ml] (
         val predictUDF = udf((items: Seq[_]) => {
           if (items != null) {
             val itemset = items.toSet
    -        brRules.value.flatMap(rule =>
    -          if (items != null && rule._1.forall(item => itemset.contains(item))) {
    +        brRules.value.flatMap { rule =>
    --- End diff --
    
    The change was about a style comment from the orginal PR that I missed. But it's great to see your suggestion. I'll run some test to confirm the performance.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r112278365
  
    --- Diff: docs/ml-frequent-pattern-mining.md ---
    @@ -0,0 +1,80 @@
    +---
    +layout: global
    +title: Frequent Pattern Mining
    +displayTitle: Frequent Pattern Mining
    +---
    +
    +Mining frequent items, itemsets, subsequences, or other substructures is usually among the
    +first steps to analyze a large-scale dataset, which has been an active research topic in
    +data mining for years.
    +We refer users to Wikipedia's [association rule learning](http://en.wikipedia.org/wiki/Association_rule_learning)
    +for more information.
    +
    +**Table of Contents**
    +
    +* This will become a table of contents (this text will be scraped).
    +{:toc}
    +
    +## FP-Growth
    +
    +The FP-growth algorithm is described in the paper
    +[Han et al., Mining frequent patterns without candidate generation](http://dx.doi.org/10.1145/335191.335372),
    +where "FP" stands for frequent pattern.
    +Given a dataset of transactions, the first step of FP-growth is to calculate item frequencies and identify frequent items.
    +Different from [Apriori-like](http://en.wikipedia.org/wiki/Apriori_algorithm) algorithms designed for the same purpose,
    +the second step of FP-growth uses a suffix tree (FP-tree) structure to encode transactions without generating candidate sets
    +explicitly, which are usually expensive to generate.
    +After the second step, the frequent itemsets can be extracted from the FP-tree.
    +In `spark.mllib`, we implemented a parallel version of FP-growth called PFP,
    +as described in [Li et al., PFP: Parallel FP-growth for query recommendation](http://dx.doi.org/10.1145/1454008.1454027).
    +PFP distributes the work of growing FP-trees based on the suffixes of transactions,
    +and hence is more scalable than a single-machine implementation.
    +We refer users to the papers for more details.
    +
    +`spark.ml`'s FP-growth implementation takes the following (hyper-)parameters:
    +
    +* `minSupport`: the minimum support for an itemset to be identified as frequent.
    +  For example, if an item appears 3 out of 5 transactions, it has a support of 3/5=0.6.
    +* `minConfidence`: minimum confidence for generating Association Rule. Confidence is an indication of how often an
    +  association rule has been found to be true. For example, if in the transactions itemset `X` appears 4 times, `X`
    +  and `Y` co-occur only 2 times, the confidence for the rule `X => Y` is then 2/4 = 0.5. The parameter will not
    +  affect the mining for frequent itemsets, but specify the minimum confidence for generating association rules
    +  from frequent itemsets.
    +* `numPartitions`: the number of partitions used to distribute the work. By default the param is not set, and
    +  number of partitions of the input dataset is used.
    +
    +The `FPGrowthModel` provides:
    +
    +* `freqItemsets`: frequent itemsets in the format of DataFrame("items"[Array], "freq"[Long])
    +* `associationRules`: association rules generated with confidence above `minConfidence`, in the format of 
    +  DataFrame("antecedent"[Array], "consequent"[Array], "confidence"[Double]).
    +* `transform`: For each transaction in itemsCol, the `transform` method will compare its items against the antecedents
    +  of each association rule. If the record contains all the antecedents of a specific association rule, the rule
    +  will be considered as applicable and its consequents will be added to the prediction result. The transform
    +  method will summarize the consequents from all the applicable rules as prediction. The prediction column has
    +  the same data type as `itemsCol` and does not contain existing items in the `itemsCol`.
    +
    +
    +**Examples**
    +
    +<div class="codetabs">
    +
    +<div data-lang="scala" markdown="1">
    +Refer to the [Scala API docs](api/scala/index.html#org.apache.spark.ml.fpm.FPGrowth) for more details.
    +
    +{% include_example scala/org/apache/spark/examples/ml/FPGrowthExample.scala %}
    +</div>
    +
    +<div data-lang="java" markdown="1">
    +Refer to the [Java API docs](api/java/org/apache/spark/ml/fpm/FPGrowth.html) for more details.
    +
    +{% include_example java/org/apache/spark/examples/ml/JavaFPGrowthExample.java %}
    +</div>
    +
    +<div data-lang="python" markdown="1">
    +Refer to the [Python API docs](api/python/pyspark.ml.html#pyspark.ml.fpm.FPGrowth) for more details.
    +
    +{% include_example python/ml/fpgrowth_example.py %}
    +</div>
    --- End diff --
    
    Sure. Added reference to R example. Manually checked on generated doc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74383/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75714/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #73790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73790/testReport)** for PR 17130 at commit [`ca12877`](https://github.com/apache/spark/commit/ca12877c7f7e224268e145c3e8c4c37413596d66).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BenFradet <gi...@git.apache.org>.

Github user BenFradet commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r108441341
  
    --- Diff: docs/ml-frequent-pattern-mining.md ---
    @@ -0,0 +1,75 @@
    +---
    +layout: global
    +title: Frequent Pattern Mining
    +displayTitle: Frequent Pattern Mining
    +---
    +
    +Mining frequent items, itemsets, subsequences, or other substructures is usually among the
    +first steps to analyze a large-scale dataset, which has been an active research topic in
    +data mining for years.
    +We refer users to Wikipedia's [association rule learning](http://en.wikipedia.org/wiki/Association_rule_learning)
    +for more information.
    +
    +**Table of Contents**
    +
    +* This will become a table of contents (this text will be scraped).
    +{:toc}
    +
    +## FP-Growth
    +
    +The FP-growth algorithm is described in the paper
    +[Han et al., Mining frequent patterns without candidate generation](http://dx.doi.org/10.1145/335191.335372),
    +where "FP" stands for frequent pattern.
    +Given a dataset of transactions, the first step of FP-growth is to calculate item frequencies and identify frequent items.
    +Different from [Apriori-like](http://en.wikipedia.org/wiki/Apriori_algorithm) algorithms designed for the same purpose,
    +the second step of FP-growth uses a suffix tree (FP-tree) structure to encode transactions without generating candidate sets
    +explicitly, which are usually expensive to generate.
    +After the second step, the frequent itemsets can be extracted from the FP-tree.
    +In `spark.mllib`, we implemented a parallel version of FP-growth called PFP,
    +as described in [Li et al., PFP: Parallel FP-growth for query recommendation](http://dx.doi.org/10.1145/1454008.1454027).
    +PFP distributes the work of growing FP-trees based on the suffices of transactions,
    --- End diff --
    
    suffixes


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75947/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #74994 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74994/testReport)** for PR 17130 at commit [`9fef280`](https://github.com/apache/spark/commit/9fef280751378dbeaa843c673fd962192320a5b1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r113129906
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -268,12 +269,8 @@ class FPGrowthModel private[ml] (
         val predictUDF = udf((items: Seq[_]) => {
           if (items != null) {
             val itemset = items.toSet
    -        brRules.value.flatMap(rule =>
    -          if (items != null && rule._1.forall(item => itemset.contains(item))) {
    -            rule._2.filter(item => !itemset.contains(item))
    -          } else {
    -            Seq.empty
    -          }).distinct
    +        brRules.value.filter(_._1.forall(itemset.contains))
    +          .flatMap(_._2.filter(!itemset.contains(_))).distinct
    --- End diff --
    
    are we aware of code changes here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/17130#discussion_r113846563

I mean style it as code with backtick

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    ping @jkbradley since we're changing the FPGrowth `transform`.
    Sean made a great suggestion to simplify `transform` code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on a diff in the pull request:

https://github.com/apache/spark/pull/17130#discussion_r108600468

Thanks for the suggestion. I do wish to have a better illustration here. But the two containing in your version make it not that straightforward, and actually it should be items in `itemsCol ` contains the antecedents for association rules.

I extend it to a longer version,

For each record in `itemsCol`, the `transform` method will compare its items against the antecedents of each association rule. If the record contains all the antecedents of a specific association rule, the rule will be considered as applicable and its consequents will be added to the prediction result. The `transform` method will summarize the consequents from all the applicable rules as prediction.

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #75714 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75714/testReport)** for PR 17130 at commit [`0fb5a87`](https://github.com/apache/spark/commit/0fb5a8757f351649a1d6648e74f126a720b37b21).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17130#discussion_r103822448
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
    @@ -240,12 +240,13 @@ class FPGrowthModel private[ml] (
         val predictUDF = udf((items: Seq[_]) => {
           if (items != null) {
             val itemset = items.toSet
    -        brRules.value.flatMap(rule =>
    -          if (items != null && rule._1.forall(item => itemset.contains(item))) {
    +        brRules.value.flatMap { rule =>
    --- End diff --
    
    Nit, while we're here -- why change this bit?
    Or if simplifying, what about
    
    ```
    brRules.value.filter(_._1_forall(itemset.contains)).flatMap(_._2.filter(!itemset.contains(_)))
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by hhbyyh <gi...@git.apache.org>.

Github user hhbyyh commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Refined some comments and minor things. This should be ready for review. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/17130


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by jkbradley <gi...@git.apache.org>.

Github user jkbradley commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Noting here: Please check out the "Issue this PR brought up" here: https://github.com/apache/spark/pull/17218
    
    It may affect this PR.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17130: [SPARK-19791] [ML] Add doc and example for fpgrow...

Posted by BenFradet <gi...@git.apache.org>.

Github user BenFradet commented on a diff in the pull request:

https://github.com/apache/spark/pull/17130#discussion_r108442588

the number of partitions of the input dataset

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    **[Test build #3599 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3599/testReport)** for PR 17130 at commit [`9ce0093`](https://github.com/apache/spark/commit/9ce00930ea18c7bb8fe0cc59b98f6ece34d20311).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17130: [SPARK-19791] [ML] Add doc and example for fpgrowth

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17130
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org