You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "RJ Nowling (JIRA)" <ji...@apache.org> on 2014/11/21 21:55:34 UTC
[jira] [Commented] (BIGTOP-1414) Add Apache Spark implementation to
BigPetStore
[ https://issues.apache.org/jira/browse/BIGTOP-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221414#comment-14221414 ]
RJ Nowling commented on BIGTOP-1414:
------------------------------------
[~jayunit100] I think BIGTOP-1366, BIGTOP-1535, and BIGTOP-1537 solve this JIRA but are broken down into separate steps. Should we close this JIRA or make the other JIRAs subtasks of this one?
> Add Apache Spark implementation to BigPetStore
> ----------------------------------------------
>
> Key: BIGTOP-1414
> URL: https://issues.apache.org/jira/browse/BIGTOP-1414
> Project: Bigtop
> Issue Type: Improvement
> Components: blueprints
> Affects Versions: backlog
> Reporter: jay vyas
> Fix For: 0.9.0
>
> Attachments: chart.png
>
>
> Currently we only process data with hadoop. Now its time to add spark to the bigpetstore application. This will basically demonstrate the difference between a mapreduce based hadoop implementation of a big data app, versus a Spark one.
> *We will need to*
> - update graphviz arch.dot to diagram spark as a new path.
> - Adding a spark job to the existing code, in a new package., which uses existing scala based generator, however, we will use it inside a spark job, rather than in a hadoop inputsplit.
> - The job should output to an RDD, which can then be serialized to disk, or else, fed into the next spark job...
> *So, the next spark job should*
> - group the data and write product summaries to a local file
> - run a product recommender against the input data set.
> We want the jobs to be runnable as modular, or as a single job, to leverage the RDD paradigm.
> So it will be interesting to see how the code is architected. Lets start the planning in this JIRA. I have some stuff ive informally hacked together, maybe i can attach an initial patch just to start a dialog.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)