You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Alec Taylor <al...@gmail.com> on 2015/01/04 01:43:02 UTC
Spark for core business-logic? - Replacing: MongoDB?
In the middle of doing the architecture for a new project, which has
various machine learning and related components, including:
recommender systems, search engines and sequence [common intersection]
matching.
Usually I use: MongoDB (as db), Redis (as cache) and celery (as queue,
backed by Redis).
Though I don't have experience with Hadoop, I was thinking of using
Hadoop for the machine-learning (as this will become a Big Data
problem quite quickly). To push the data into Hadoop, I would use a
connector of some description, or push the MongoDB backups into HDFS
at set intervals.
However I was thinking that it might be better to put the whole thing
in Hadoop, store all persistent data in Hadoop, and maybe do all the
layers in Apache Spark (with caching remaining in Redis).
Is that a viable option? - Most of what I see discusses Spark (and
Hadoop in general) for analytics only. Apache Phoenix exposes a nice
interface for read/write over HBase, so I might use that if Spark ends
up being the wrong solution.
Thanks for all suggestions,
Alec Taylor
PS: I need this for both "Big" and "Small" data. Note that I am using
the Cloudera definition of "Big Data" referring to processing/storage
across more than 1 machine.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark for core business-logic? - Replacing: MongoDB?
Posted by Simon Chan <si...@gmail.com>.
PredictionIO comes with a event server that handles data collection:
http://docs.prediction.io/datacollection/overview/
It's based on HBase, which works fine with Spark as the data store of the
event/training data.
You probably need a separate CRUD-supported database for your application.
Your application will then communicate with PredictionIO after
authentication is done, for example.
I hope it helps.
Regards,
Simon
On Mon, Jan 5, 2015 at 6:22 PM, Alec Taylor <al...@gmail.com> wrote:
> Thanks Simon, that's a good way to train on incoming events (and
> related problems / and result computations).
>
> However, does it handle the actual data storage? - E.g.: CRUD documents
>
> On Tue, Jan 6, 2015 at 1:18 PM, Simon Chan <si...@gmail.com> wrote:
> > Alec,
> >
> > If you are looking for a Machine Learning stack that supports
> > business-logics, you may take a look at PredictionIO:
> > http://prediction.io/
> >
> > It's based on Spark and HBase.
> >
> > Simon
> >
> >
> > On Mon, Jan 5, 2015 at 6:14 PM, Alec Taylor <al...@gmail.com>
> wrote:
> >>
> >> Thanks all. To answer your clarification questions:
> >>
> >> - I'm writing this in Python
> >> - A similar problem to my actual one is to find common 30 minute slots
> >> (over the next 12 months) [r] that k users have in common. Total
> >> users: n. Given n=10000 and r=17472 then the [naïve] time-complexity
> >> is $\mathcal{O}(nr)$. n*r=17,472,000. I may be able to get
> >> $\mathcal{O}(n \log r)$ if not $\log \log$ from reading the literature
> >> on sequence matching, however this is uncertain.
> >>
> >> So assuming all the other business-logic which needs to be built in,
> >> such as authentication and various other CRUD operations, as well as
> >> this more intensive sequence searching operation, what stack would be
> >> best for me?
> >>
> >> Thanks for all suggestions
> >>
> >> On Mon, Jan 5, 2015 at 4:24 PM, Jörn Franke <jo...@gmail.com>
> wrote:
> >> > Hallo,
> >> >
> >> > It really depends on your requirements, what kind of machine learning
> >> > algorithm your budget, if you do currently something really new or
> >> > integrate
> >> > it with an existing application, etc.. You can run MongoDB as well as
> a
> >> > cluster. I don't think this question can be answered generally, but
> >> > depends
> >> > on details of your case.
> >> >
> >> > Best regards
> >> >
> >> > Le 4 janv. 2015 01:44, "Alec Taylor" <al...@gmail.com> a
> écrit :
> >> >>
> >> >> In the middle of doing the architecture for a new project, which has
> >> >> various machine learning and related components, including:
> >> >> recommender systems, search engines and sequence [common
> intersection]
> >> >> matching.
> >> >>
> >> >> Usually I use: MongoDB (as db), Redis (as cache) and celery (as
> queue,
> >> >> backed by Redis).
> >> >>
> >> >> Though I don't have experience with Hadoop, I was thinking of using
> >> >> Hadoop for the machine-learning (as this will become a Big Data
> >> >> problem quite quickly). To push the data into Hadoop, I would use a
> >> >> connector of some description, or push the MongoDB backups into HDFS
> >> >> at set intervals.
> >> >>
> >> >> However I was thinking that it might be better to put the whole thing
> >> >> in Hadoop, store all persistent data in Hadoop, and maybe do all the
> >> >> layers in Apache Spark (with caching remaining in Redis).
> >> >>
> >> >> Is that a viable option? - Most of what I see discusses Spark (and
> >> >> Hadoop in general) for analytics only. Apache Phoenix exposes a nice
> >> >> interface for read/write over HBase, so I might use that if Spark
> ends
> >> >> up being the wrong solution.
> >> >>
> >> >> Thanks for all suggestions,
> >> >>
> >> >> Alec Taylor
> >> >>
> >> >> PS: I need this for both "Big" and "Small" data. Note that I am using
> >> >> the Cloudera definition of "Big Data" referring to processing/storage
> >> >> across more than 1 machine.
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> >> For additional commands, e-mail: user-help@spark.apache.org
> >> >>
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: user-help@spark.apache.org
> >>
> >
>
Re: Spark for core business-logic? - Replacing: MongoDB?
Posted by Alec Taylor <al...@gmail.com>.
Thanks Simon, that's a good way to train on incoming events (and
related problems / and result computations).
However, does it handle the actual data storage? - E.g.: CRUD documents
On Tue, Jan 6, 2015 at 1:18 PM, Simon Chan <si...@gmail.com> wrote:
> Alec,
>
> If you are looking for a Machine Learning stack that supports
> business-logics, you may take a look at PredictionIO:
> http://prediction.io/
>
> It's based on Spark and HBase.
>
> Simon
>
>
> On Mon, Jan 5, 2015 at 6:14 PM, Alec Taylor <al...@gmail.com> wrote:
>>
>> Thanks all. To answer your clarification questions:
>>
>> - I'm writing this in Python
>> - A similar problem to my actual one is to find common 30 minute slots
>> (over the next 12 months) [r] that k users have in common. Total
>> users: n. Given n=10000 and r=17472 then the [naïve] time-complexity
>> is $\mathcal{O}(nr)$. n*r=17,472,000. I may be able to get
>> $\mathcal{O}(n \log r)$ if not $\log \log$ from reading the literature
>> on sequence matching, however this is uncertain.
>>
>> So assuming all the other business-logic which needs to be built in,
>> such as authentication and various other CRUD operations, as well as
>> this more intensive sequence searching operation, what stack would be
>> best for me?
>>
>> Thanks for all suggestions
>>
>> On Mon, Jan 5, 2015 at 4:24 PM, Jörn Franke <jo...@gmail.com> wrote:
>> > Hallo,
>> >
>> > It really depends on your requirements, what kind of machine learning
>> > algorithm your budget, if you do currently something really new or
>> > integrate
>> > it with an existing application, etc.. You can run MongoDB as well as a
>> > cluster. I don't think this question can be answered generally, but
>> > depends
>> > on details of your case.
>> >
>> > Best regards
>> >
>> > Le 4 janv. 2015 01:44, "Alec Taylor" <al...@gmail.com> a écrit :
>> >>
>> >> In the middle of doing the architecture for a new project, which has
>> >> various machine learning and related components, including:
>> >> recommender systems, search engines and sequence [common intersection]
>> >> matching.
>> >>
>> >> Usually I use: MongoDB (as db), Redis (as cache) and celery (as queue,
>> >> backed by Redis).
>> >>
>> >> Though I don't have experience with Hadoop, I was thinking of using
>> >> Hadoop for the machine-learning (as this will become a Big Data
>> >> problem quite quickly). To push the data into Hadoop, I would use a
>> >> connector of some description, or push the MongoDB backups into HDFS
>> >> at set intervals.
>> >>
>> >> However I was thinking that it might be better to put the whole thing
>> >> in Hadoop, store all persistent data in Hadoop, and maybe do all the
>> >> layers in Apache Spark (with caching remaining in Redis).
>> >>
>> >> Is that a viable option? - Most of what I see discusses Spark (and
>> >> Hadoop in general) for analytics only. Apache Phoenix exposes a nice
>> >> interface for read/write over HBase, so I might use that if Spark ends
>> >> up being the wrong solution.
>> >>
>> >> Thanks for all suggestions,
>> >>
>> >> Alec Taylor
>> >>
>> >> PS: I need this for both "Big" and "Small" data. Note that I am using
>> >> the Cloudera definition of "Big Data" referring to processing/storage
>> >> across more than 1 machine.
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> >> For additional commands, e-mail: user-help@spark.apache.org
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark for core business-logic? - Replacing: MongoDB?
Posted by Simon Chan <si...@gmail.com>.
Alec,
If you are looking for a Machine Learning stack that supports
business-logics, you may take a look at PredictionIO:
http://prediction.io/
It's based on Spark and HBase.
Simon
On Mon, Jan 5, 2015 at 6:14 PM, Alec Taylor <al...@gmail.com> wrote:
> Thanks all. To answer your clarification questions:
>
> - I'm writing this in Python
> - A similar problem to my actual one is to find common 30 minute slots
> (over the next 12 months) [r] that k users have in common. Total
> users: n. Given n=10000 and r=17472 then the [naïve] time-complexity
> is $\mathcal{O}(nr)$. n*r=17,472,000. I may be able to get
> $\mathcal{O}(n \log r)$ if not $\log \log$ from reading the literature
> on sequence matching, however this is uncertain.
>
> So assuming all the other business-logic which needs to be built in,
> such as authentication and various other CRUD operations, as well as
> this more intensive sequence searching operation, what stack would be
> best for me?
>
> Thanks for all suggestions
>
> On Mon, Jan 5, 2015 at 4:24 PM, Jörn Franke <jo...@gmail.com> wrote:
> > Hallo,
> >
> > It really depends on your requirements, what kind of machine learning
> > algorithm your budget, if you do currently something really new or
> integrate
> > it with an existing application, etc.. You can run MongoDB as well as a
> > cluster. I don't think this question can be answered generally, but
> depends
> > on details of your case.
> >
> > Best regards
> >
> > Le 4 janv. 2015 01:44, "Alec Taylor" <al...@gmail.com> a écrit :
> >>
> >> In the middle of doing the architecture for a new project, which has
> >> various machine learning and related components, including:
> >> recommender systems, search engines and sequence [common intersection]
> >> matching.
> >>
> >> Usually I use: MongoDB (as db), Redis (as cache) and celery (as queue,
> >> backed by Redis).
> >>
> >> Though I don't have experience with Hadoop, I was thinking of using
> >> Hadoop for the machine-learning (as this will become a Big Data
> >> problem quite quickly). To push the data into Hadoop, I would use a
> >> connector of some description, or push the MongoDB backups into HDFS
> >> at set intervals.
> >>
> >> However I was thinking that it might be better to put the whole thing
> >> in Hadoop, store all persistent data in Hadoop, and maybe do all the
> >> layers in Apache Spark (with caching remaining in Redis).
> >>
> >> Is that a viable option? - Most of what I see discusses Spark (and
> >> Hadoop in general) for analytics only. Apache Phoenix exposes a nice
> >> interface for read/write over HBase, so I might use that if Spark ends
> >> up being the wrong solution.
> >>
> >> Thanks for all suggestions,
> >>
> >> Alec Taylor
> >>
> >> PS: I need this for both "Big" and "Small" data. Note that I am using
> >> the Cloudera definition of "Big Data" referring to processing/storage
> >> across more than 1 machine.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: user-help@spark.apache.org
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Spark for core business-logic? - Replacing: MongoDB?
Posted by Alec Taylor <al...@gmail.com>.
Thanks all. To answer your clarification questions:
- I'm writing this in Python
- A similar problem to my actual one is to find common 30 minute slots
(over the next 12 months) [r] that k users have in common. Total
users: n. Given n=10000 and r=17472 then the [naïve] time-complexity
is $\mathcal{O}(nr)$. n*r=17,472,000. I may be able to get
$\mathcal{O}(n \log r)$ if not $\log \log$ from reading the literature
on sequence matching, however this is uncertain.
So assuming all the other business-logic which needs to be built in,
such as authentication and various other CRUD operations, as well as
this more intensive sequence searching operation, what stack would be
best for me?
Thanks for all suggestions
On Mon, Jan 5, 2015 at 4:24 PM, Jörn Franke <jo...@gmail.com> wrote:
> Hallo,
>
> It really depends on your requirements, what kind of machine learning
> algorithm your budget, if you do currently something really new or integrate
> it with an existing application, etc.. You can run MongoDB as well as a
> cluster. I don't think this question can be answered generally, but depends
> on details of your case.
>
> Best regards
>
> Le 4 janv. 2015 01:44, "Alec Taylor" <al...@gmail.com> a écrit :
>>
>> In the middle of doing the architecture for a new project, which has
>> various machine learning and related components, including:
>> recommender systems, search engines and sequence [common intersection]
>> matching.
>>
>> Usually I use: MongoDB (as db), Redis (as cache) and celery (as queue,
>> backed by Redis).
>>
>> Though I don't have experience with Hadoop, I was thinking of using
>> Hadoop for the machine-learning (as this will become a Big Data
>> problem quite quickly). To push the data into Hadoop, I would use a
>> connector of some description, or push the MongoDB backups into HDFS
>> at set intervals.
>>
>> However I was thinking that it might be better to put the whole thing
>> in Hadoop, store all persistent data in Hadoop, and maybe do all the
>> layers in Apache Spark (with caching remaining in Redis).
>>
>> Is that a viable option? - Most of what I see discusses Spark (and
>> Hadoop in general) for analytics only. Apache Phoenix exposes a nice
>> interface for read/write over HBase, so I might use that if Spark ends
>> up being the wrong solution.
>>
>> Thanks for all suggestions,
>>
>> Alec Taylor
>>
>> PS: I need this for both "Big" and "Small" data. Note that I am using
>> the Cloudera definition of "Big Data" referring to processing/storage
>> across more than 1 machine.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark for core business-logic? - Replacing: MongoDB?
Posted by Jörn Franke <jo...@gmail.com>.
Hallo,
It really depends on your requirements, what kind of machine learning
algorithm your budget, if you do currently something really new or
integrate it with an existing application, etc.. You can run MongoDB as
well as a cluster. I don't think this question can be answered generally,
but depends on details of your case.
Best regards
Le 4 janv. 2015 01:44, "Alec Taylor" <al...@gmail.com> a écrit :
> In the middle of doing the architecture for a new project, which has
> various machine learning and related components, including:
> recommender systems, search engines and sequence [common intersection]
> matching.
>
> Usually I use: MongoDB (as db), Redis (as cache) and celery (as queue,
> backed by Redis).
>
> Though I don't have experience with Hadoop, I was thinking of using
> Hadoop for the machine-learning (as this will become a Big Data
> problem quite quickly). To push the data into Hadoop, I would use a
> connector of some description, or push the MongoDB backups into HDFS
> at set intervals.
>
> However I was thinking that it might be better to put the whole thing
> in Hadoop, store all persistent data in Hadoop, and maybe do all the
> layers in Apache Spark (with caching remaining in Redis).
>
> Is that a viable option? - Most of what I see discusses Spark (and
> Hadoop in general) for analytics only. Apache Phoenix exposes a nice
> interface for read/write over HBase, so I might use that if Spark ends
> up being the wrong solution.
>
> Thanks for all suggestions,
>
> Alec Taylor
>
> PS: I need this for both "Big" and "Small" data. Note that I am using
> the Cloudera definition of "Big Data" referring to processing/storage
> across more than 1 machine.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Spark for core business-logic? - Replacing: MongoDB?
Posted by Krishna Sankar <ks...@gmail.com>.
Alec,
Good questions. Suggestions:
1. Refactor the problem into layers viz. DFS, Data Store, DB, SQL Layer,
Cache, Queue, App Server, App (Interface), App (backend ML) et al.
2. Then slot-in the appropriate technologies - may be even multiple
technologies for the same layer and then work thru the pros and cons.
3. Looking at the layers (moving from the easy to difficult, the mundane
to the esoteric ;o)):
- Cache & Queue - stick with what you are comfortable with ie Redis
et al. Also take a look at Kafka
- App Server - Tomcat et al
- App (Interface) - JavaScript et al
- DB, SQL Layer - Better off with with MongoDB. You can explore
HBase, but it is not the same.
- The same way as MongoDB != mySQL, HBase != MongoDB
- Machine Learning Server/Layer - Spark would fit very well here.
- Machine Learning DFS, Data Store - HDFS
- The idea of pushing the data to Hadoop for ML is good
- But you need to think thru things like incremental data load,
semantics like at least once, at most once et al.
4. You could architect all with the Hadoop eco system. It might work,
depending on the system.
- But I would use caution. Most probably many of the elements would
rather be implemented in appropriate technologies.
5. Doubleclick couple more times on the design, think thru the
functionality, scaling requirements et al
- Draw 3 or 4 alternatives and jot down the top 5 requirements, pros
and cons, the knowns and the unknowns
- The optimum design will fall thru
Cheers
<k/>
On Sat, Jan 3, 2015 at 4:43 PM, Alec Taylor <al...@gmail.com> wrote:
> In the middle of doing the architecture for a new project, which has
> various machine learning and related components, including:
> recommender systems, search engines and sequence [common intersection]
> matching.
>
> Usually I use: MongoDB (as db), Redis (as cache) and celery (as queue,
> backed by Redis).
>
> Though I don't have experience with Hadoop, I was thinking of using
> Hadoop for the machine-learning (as this will become a Big Data
> problem quite quickly). To push the data into Hadoop, I would use a
> connector of some description, or push the MongoDB backups into HDFS
> at set intervals.
>
> However I was thinking that it might be better to put the whole thing
> in Hadoop, store all persistent data in Hadoop, and maybe do all the
> layers in Apache Spark (with caching remaining in Redis).
>
> Is that a viable option? - Most of what I see discusses Spark (and
> Hadoop in general) for analytics only. Apache Phoenix exposes a nice
> interface for read/write over HBase, so I might use that if Spark ends
> up being the wrong solution.
>
> Thanks for all suggestions,
>
> Alec Taylor
>
> PS: I need this for both "Big" and "Small" data. Note that I am using
> the Cloudera definition of "Big Data" referring to processing/storage
> across more than 1 machine.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>