You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jason Rutherglen <ja...@gmail.com> on 2008/09/04 00:50:34 UTC
Re: Realtime Search for Social Networks Collaboration
Hi Yonik,
The SOLR 2 list looks good. The question is, who is going to do the
work? I tried to simplify the scope of Ocean as much as possible to
make it possible (and slowly at that over time) for me to eventually
finish what is mentioned on the wiki. I think SOLR is very cool and
was major step forward when it came out. I also think it's got a
lot of things now which makes integration difficult to do properly. I
did try to integrate and received a lukewarm response and so decided
to just move ahead separately until folks have time to collaborate.
We probably should try to integrate SOLR and Ocean somehow however we
may want to simply reduce the scope a bit and figure what is needed
most, with the main use case being social networks.
I think the problem with integration with SOLR is it was designed with
a different problem set in mind than Ocean, originally the CNET
shopping application. Facets were important, realtime was not needed
because pricing doesn't change very often. I designed Ocean for
social networks and actually further into the future realtime
messaging based mobile applications.
SOLR needs to be backward compatible and support it's existing user
base. How do you plan on doing this for a SOLR 2 if the architecture
is changed dramatically? SOLR solves a problem set that is very
common making SOLR very useful in many situations. However I wanted
Ocean to be like GData. So I wanted the scalability of Google which
SOLR doesn't quite have yet, and the realtime, and then I figured the
other stuff could be added later, stuff people seem to spend a lot of
time on in the SOLR community currently (spellchecker, db imports,
many others). I did use some of the SOLR terminology in building
Ocean, like snapshots! But most of it is a digression. I tried to
use schemas, but they just make the system harder to use. For
distributed search I prefer serialized objects as this enables things
like SpanQueries and payloads without writing request handlers and
such. Also there is no need to write new request handlers and deploy
(an expensive operation for systems that are in the 100s of servers)
them as any new classes are simply dynamically loaded by the server
from the client.
A lot is now outlined on the wiki site
http://wiki.apache.org/lucene-java/OceanRealtimeSearch now and there
will be a lot more javadocs in the forthcoming patch. The latest code
is also available all the time at
http://oceansearch.googlecode.com/svn/trunk/trunk/oceanlucene
I do welcome more discussion and if there are Solr developers who wish
to work on Ocean feel free to drop me a line. Most of all though I
think it would be useful for social networks interested in realtime
search to get involved as it may be something that is difficult for
one company to have enough resources to implement to a production
level. I think this is where open source collaboration is
particularly useful.
Cheers,
Jason Rutherglen
jason.rutherglen@gmail.com
On Wed, Sep 3, 2008 at 4:56 PM, Yonik Seeley <yo...@apache.org> wrote:
> On Wed, Sep 3, 2008 at 3:20 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
>> I am wondering
>> if there are social networks (or anyone else) out there who would be
>> interested in collaborating with Apache on realtime search to get it
>> to the point it can be used in production.
>
> Good timing Jason, I think you'll find some other people right here
> at Apache (solr-dev) that want to collaborate in this area:
>
> http://www.nabble.com/solr2%3A-Onward-and-Upward-td19224805.html
>
> I've looked at your wiki briefly, and all the high level goals/features seem
> to really be synergistic with where we are going with Solr2.
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Realtime Search for Social Networks Collaboration
Posted by Jason Rutherglen <ja...@gmail.com>.
Hi Yonik,
I found the basic integration with SOLR and Ocean to be fairly
straightforward, the https://issues.apache.org/jira/browse/SOLR-567
patch is key to that. SOLR just needs an optimistic concurrency
update handler and most of the functionality would work. I guess the
problem would be, removing the ability to do things that in realtime
are unnecessary like commit, optimize, and others. It's doable, let me
know if you need some help making some decisions about things.
Take care,
Jason
On Thu, Sep 4, 2008 at 10:13 AM, Yonik Seeley <yo...@apache.org> wrote:
> On Wed, Sep 3, 2008 at 6:50 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
>> I also think it's got a
>> lot of things now which makes integration difficult to do properly.
>
> I agree, and that's why the major bump in version number rather than
> minor - we recognize that some features will need some amount of
> rearchitecture.
>
>> I think the problem with integration with SOLR is it was designed with
>> a different problem set in mind than Ocean, originally the CNET
>> shopping application.
>
> That was the first use of Solr, but it actually existed before that
> w/o any defined use other than to be a "plan B" alternative to MySQL
> based search servers (that's actually where some of the parameter
> names come from... the default /select URL instead of /search, the
> "rows" parameter, etc).
>
> But you're right... some things like the replication strategy were
> designed (well, borrowed from Doug to be exact) with the idea that it
> would be OK to have slightly "stale" views of the data in the range of
> minutes. It just made things easier/possible at the time. But tons
> of Solr and Lucene users want almost instantaneous visibility of added
> documents, if they can get it. It's hardly restricted to social
> network applications.
>
> Bottom line is that Solr aims to be a general enterprise search
> platform, and getting as real-time as we can get, and as scalable as
> we can get are some of the top priorities going forward.
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Realtime Search for Social Networks Collaboration
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Sep 3, 2008 at 6:50 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> I also think it's got a
> lot of things now which makes integration difficult to do properly.
I agree, and that's why the major bump in version number rather than
minor - we recognize that some features will need some amount of
rearchitecture.
> I think the problem with integration with SOLR is it was designed with
> a different problem set in mind than Ocean, originally the CNET
> shopping application.
That was the first use of Solr, but it actually existed before that
w/o any defined use other than to be a "plan B" alternative to MySQL
based search servers (that's actually where some of the parameter
names come from... the default /select URL instead of /search, the
"rows" parameter, etc).
But you're right... some things like the replication strategy were
designed (well, borrowed from Doug to be exact) with the idea that it
would be OK to have slightly "stale" views of the data in the range of
minutes. It just made things easier/possible at the time. But tons
of Solr and Lucene users want almost instantaneous visibility of added
documents, if they can get it. It's hardly restricted to social
network applications.
Bottom line is that Solr aims to be a general enterprise search
platform, and getting as real-time as we can get, and as scalable as
we can get are some of the top priorities going forward.
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org