You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2014/06/05 19:20:02 UTC
[jira] [Commented] (MAHOUT-1541) Create CLI Driver for Spark
Cooccurrence Analysis
[ https://issues.apache.org/jira/browse/MAHOUT-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018986#comment-14018986 ]
ASF GitHub Bot commented on MAHOUT-1541:
----------------------------------------
GitHub user pferrel opened a pull request:
https://github.com/apache/mahout/pull/11
Mahout 1541
MAHOUT-1541 WIP, need access to an RDD, which used to be in a DrmLike[Int]. Still need the DrmLike to pass in to cooccurrence but need the RDD to write a text file. CheckpointedDrmSpark doesn't have the DrmLike?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pferrel/mahout mahout-1541
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/mahout/pull/11.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11
----
commit 107a0ba9605241653a85b113661a8fa5c055529f
Author: pferrel <pa...@occamsmachete.com>
Date: 2014-06-04T19:54:22Z
added Sebastian's CooccurrenceAnalysis patch updated it to use current Mahout-DSL
commit 74b9921c4c9bd8903585bbd74d9e66298ea8b7a0
Author: pferrel <pa...@occamsmachete.com>
Date: 2014-06-04T20:09:07Z
Adding stuff for itemsimilarity driver for Spark
commit a59265931ed3a51ba81e1a0a7171ebb102be4fa4
Author: pferrel <pa...@occamsmachete.com>
Date: 2014-06-04T20:13:13Z
added scopt to pom deps
commit 16c03f7fa73c156859d1dba3a333ef9e8bf922b0
Author: pferrel <pa...@occamsmachete.com>
Date: 2014-06-04T21:32:18Z
added Sebastian's MurmurHash changes
Signed-off-by: pferrel <pa...@occamsmachete.com>
commit 8a4b4347ddb7b9ac97590aa20189d89d8a07a80a
Author: pferrel <pa...@occamsmachete.com>
Date: 2014-06-04T21:33:11Z
Merge branch 'mahout-1464' into mahout-1541
commit 2f87f5433f90fa2c49ef386ca245943e1fc73beb
Author: pferrel <pa...@occamsmachete.com>
Date: 2014-06-05T01:44:16Z
MAHOUT-1541 still working on this, some refactoring in the DSL for abstracting away Spark has moved access to rddsno Jira is closed yet
commit c6adaa44c80bba99d41600e260bbb1ad5c972e69
Author: pferrel <pa...@occamsmachete.com>
Date: 2014-06-05T16:52:23Z
MAHOUT-1464 import cleanup, minor changes to examples for running on Spark Cluster
commit 2caceab31703ed214c1e66d5fc63b8bdb05d37a3
Author: pferrel <pa...@occamsmachete.com>
Date: 2014-06-05T16:55:09Z
Merge branch 'mahout-1464' into mahout-1541
----
> Create CLI Driver for Spark Cooccurrence Analysis
> -------------------------------------------------
>
> Key: MAHOUT-1541
> URL: https://issues.apache.org/jira/browse/MAHOUT-1541
> Project: Mahout
> Issue Type: Bug
> Components: CLI
> Reporter: Pat Ferrel
> Assignee: Pat Ferrel
>
> Create a CLI driver to import data in a flexible manner, create an IndexedDataset with BiMap ID translation dictionaries, call the Spark CooccurrenceAnalysis with the appropriate params, then write output with external IDs optionally reattached.
> Ultimately it should be able to read input as the legacy mr does but will support reading externally defined IDs and flexible formats. Output will be of the legacy format or text files of the user's specification with reattached Item IDs.
> Support for legacy formats is a question, users can always use the legacy code if they want this. Internal to the IndexedDataset is a Spark DRM so pipelining can be accomplished without any writing to an actual file so the legacy sequence file output may not be needed.
> Opinions?
--
This message was sent by Atlassian JIRA
(v6.2#6252)