You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Joseph K. Bradley (JIRA)" <ji...@apache.org> on 2017/06/21 20:54:00 UTC
[jira] [Updated] (SPARK-20114) spark.ml parity for sequential
pattern mining - PrefixSpan
[ https://issues.apache.org/jira/browse/SPARK-20114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joseph K. Bradley updated SPARK-20114:
--------------------------------------
Issue Type: Sub-task (was: New Feature)
Parent: SPARK-14501
> spark.ml parity for sequential pattern mining - PrefixSpan
> ----------------------------------------------------------
>
> Key: SPARK-20114
> URL: https://issues.apache.org/jira/browse/SPARK-20114
> Project: Spark
> Issue Type: Sub-task
> Components: ML
> Affects Versions: 2.2.0
> Reporter: yuhao yang
>
> Creating this jira to track the feature parity for PrefixSpan and sequential pattern mining in Spark ml with DataFrame API.
> First list a few design issues to be discussed, then subtasks like Scala, Python and R API will be created.
> # Wrapping the MLlib PrefixSpan and provide a generic fit() should be straightforward. Yet PrefixSpan only extracts frequent sequential patterns, which is not good to be used directly for predicting on new records. Please read http://data-mining.philippe-fournier-viger.com/introduction-to-sequential-rule-mining/ for some background knowledge. Thanks Philippe Fournier-Viger for providing insights. If we want to keep using the Estimator/Transformer pattern, options are:
> #* Implement a dummy transform for PrefixSpanModel, which will not add new column to the input DataSet. The PrefixSpanModel is only used to provide access for frequent sequential patterns.
> #* Adding the feature to extract sequential rules from sequential patterns. Then use the sequential rules in the transform as FPGrowthModel. The rules extracted are of the form X–> Y where X and Y are sequential patterns. But in practice, these rules are not very good as they are too precise and thus not noise tolerant.
> # Different from association rules and frequent itemsets, sequential rules can be extracted from the original dataset more efficiently using algorithms like RuleGrowth, ERMiner. The rules are X–> Y where X is unordered and Y is unordered, but X must appear before Y, which is more general and can work better in practice for prediction.
> I'd like to hear more from the users to see which kind of Sequential rules are more practical.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org