You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Mark Hahn <ma...@boutiquing.com> on 2011/03/21 18:24:38 UTC

blocking during view generation

I have noticed that all view queries are blocked for some time when I just
change one view.  This seems like it is going to be a serious problem when
my db grows.  Is there someway to avoid this in production?  I've never seen
mysql or any other db block for any reason.

Re: blocking during view generation

Posted by Robert Newson <ro...@gmail.com>.
This is documented behavior.

View queries block by default to catch up with all changes made since
the last view query.

http://guide.couchdb.org/editions/1/en/views.html

B.

On 21 March 2011 17:24, Mark Hahn <ma...@boutiquing.com> wrote:
> I have noticed that all view queries are blocked for some time when I just
> change one view.  This seems like it is going to be a serious problem when
> my db grows.  Is there someway to avoid this in production?  I've never seen
> mysql or any other db block for any reason.
>

Re: blocking during view generation

Posted by Paul Davis <pa...@gmail.com>.
On Mon, Mar 21, 2011 at 4:49 PM, Mark Hahn <ma...@elleh.com> wrote:
> What is the disadvantage to having a different design doc for each view?
>
> On Mon, Mar 21, 2011 at 11:05 AM, Paul Davis
> <pa...@gmail.com> wrote:
>> On Mon, Mar 21, 2011 at 1:24 PM, Mark Hahn <ma...@boutiquing.com> wrote:
>>> I have noticed that all view queries are blocked for some time when I just
>>> change one view.  This seems like it is going to be a serious problem when
>>> my db grows.  Is there someway to avoid this in production?  I've never seen
>>> mysql or any other db block for any reason.
>>>
>>
>> Also of note is that the updates are grouped at the _design/doc level,
>> so if you want to update some subset of views individually you just
>> need to split them amongst a set of _design/docs. There's also a
>> scheme that lets you create a temporary _design/doc that can be moved
>> over top of the existing version to have quickly updated views, but
>> this approach requires you to have the disk capacity to have both
>> versions built on disk simultaneously.
>>
>

Slightly more disk space used and longer total view build times due to
overhead of passing each doc to the view engine multiple times. Though
you can build multiple view groups in parallel if you have the CPU
resources.

Its hard to say exactly what the difference for each bit would be as
it depends directly on your specific data and usage patterns.

Re: blocking during view generation

Posted by Mark Hahn <ma...@elleh.com>.
What is the disadvantage to having a different design doc for each view?

On Mon, Mar 21, 2011 at 11:05 AM, Paul Davis
<pa...@gmail.com> wrote:
> On Mon, Mar 21, 2011 at 1:24 PM, Mark Hahn <ma...@boutiquing.com> wrote:
>> I have noticed that all view queries are blocked for some time when I just
>> change one view.  This seems like it is going to be a serious problem when
>> my db grows.  Is there someway to avoid this in production?  I've never seen
>> mysql or any other db block for any reason.
>>
>
> Also of note is that the updates are grouped at the _design/doc level,
> so if you want to update some subset of views individually you just
> need to split them amongst a set of _design/docs. There's also a
> scheme that lets you create a temporary _design/doc that can be moved
> over top of the existing version to have quickly updated views, but
> this approach requires you to have the disk capacity to have both
> versions built on disk simultaneously.
>

Re: blocking during view generation

Posted by David Rose <do...@gmail.com>.
janl described this idea to me last week at SXSW. You create a new
design/views, and wait for it to finish indexing, then use HTTP COPY
to replace the original design with the newly-indexed design.

David

On Mon, Mar 21, 2011 at 5:15 PM, Paul Davis <pa...@gmail.com> wrote:
> On Mon, Mar 21, 2011 at 5:36 PM, Dirkjan Ochtman <di...@ochtman.nl> wrote:
>> On Mon, Mar 21, 2011 at 19:05, Paul Davis <pa...@gmail.com> wrote:
>>> Also of note is that the updates are grouped at the _design/doc level,
>>> so if you want to update some subset of views individually you just
>>> need to split them amongst a set of _design/docs. There's also a
>>> scheme that lets you create a temporary _design/doc that can be moved
>>> over top of the existing version to have quickly updated views, but
>>> this approach requires you to have the disk capacity to have both
>>> versions built on disk simultaneously.
>>
>> Is that scheme documented anywhere? It sounds useful, but I'm not
>> exactly sure how I'd go about doing that.
>>
>> Cheers,
>>
>> Dirkjan
>>
>
> I don't know if its been written up anywhere, but the basic idea is
> that you just create a copy of the design doc, update it with new
> code, build the indexes and move it back to the original docid.
>

Re: blocking during view generation

Posted by Paul Davis <pa...@gmail.com>.
On Mon, Mar 21, 2011 at 5:36 PM, Dirkjan Ochtman <di...@ochtman.nl> wrote:
> On Mon, Mar 21, 2011 at 19:05, Paul Davis <pa...@gmail.com> wrote:
>> Also of note is that the updates are grouped at the _design/doc level,
>> so if you want to update some subset of views individually you just
>> need to split them amongst a set of _design/docs. There's also a
>> scheme that lets you create a temporary _design/doc that can be moved
>> over top of the existing version to have quickly updated views, but
>> this approach requires you to have the disk capacity to have both
>> versions built on disk simultaneously.
>
> Is that scheme documented anywhere? It sounds useful, but I'm not
> exactly sure how I'd go about doing that.
>
> Cheers,
>
> Dirkjan
>

I don't know if its been written up anywhere, but the basic idea is
that you just create a copy of the design doc, update it with new
code, build the indexes and move it back to the original docid.

Re: blocking during view generation

Posted by Dirkjan Ochtman <di...@ochtman.nl>.
On Mon, Mar 21, 2011 at 19:05, Paul Davis <pa...@gmail.com> wrote:
> Also of note is that the updates are grouped at the _design/doc level,
> so if you want to update some subset of views individually you just
> need to split them amongst a set of _design/docs. There's also a
> scheme that lets you create a temporary _design/doc that can be moved
> over top of the existing version to have quickly updated views, but
> this approach requires you to have the disk capacity to have both
> versions built on disk simultaneously.

Is that scheme documented anywhere? It sounds useful, but I'm not
exactly sure how I'd go about doing that.

Cheers,

Dirkjan

Re: blocking during view generation

Posted by Paul Davis <pa...@gmail.com>.
On Mon, Mar 21, 2011 at 1:24 PM, Mark Hahn <ma...@boutiquing.com> wrote:
> I have noticed that all view queries are blocked for some time when I just
> change one view.  This seems like it is going to be a serious problem when
> my db grows.  Is there someway to avoid this in production?  I've never seen
> mysql or any other db block for any reason.
>

Also of note is that the updates are grouped at the _design/doc level,
so if you want to update some subset of views individually you just
need to split them amongst a set of _design/docs. There's also a
scheme that lets you create a temporary _design/doc that can be moved
over top of the existing version to have quickly updated views, but
this approach requires you to have the disk capacity to have both
versions built on disk simultaneously.

Re: blocking during view generation

Posted by "Andrew Stuart (SuperCoders)" <an...@supercoders.com.au>.
Hmmm.  I'd have imagined this would be on the critical "get this fixed  
ASAP" task list for the developers.  Is it even on the roadmap?  I'd  
prefer to avoid building an app that has to choose between making  
requests for stale data and potentially waiting an unknown amount of  
time for views to update.

Odd that such an issue has remained for so long into the life of  
CouchDB.

as


On 23/03/2011, at 2:34 PM, kowsik wrote:

On Tue, Mar 22, 2011 at 8:28 PM, Andrew Stuart (SuperCoders)
<an...@supercoders.com.au> wrote:
> This one seems hard to believe - is it true that CouchDB blocks the  
> server
> whilst updating views?
>
> View updates can be alot of work for a server.
>
> So in reality, queries to the server pause whilst views are updated?
>
> This doesn't seem practical for any production usage.

See this blog: http://labs.mudynamics.com/2011/03/10/blitzio-couchdb-in-production/

As long as you have a worker of sorts that watches the _changes feed
to "catch up" on the index, things work great. Just like any other
piece of software, you have to have some understanding how CouchDB
works internally. There is a stale=ok view query which will get you
the older revisions so your UI doesn't block at all.

> Can someone confirm that this is true, that during production a  
> server will
> block whilst views are updated?

See above. :) Yes, couch doesn't do background view indexing, though I
tend to agree that if there was an Erlang task of sorts that did this
automatically, end users can relax even more.

> Does anyone else see this as a major issue or am I missing  
> something? I'm
> happy to say I have missed the point many times before :-)
>
> I'm with Mark - I can't think of any other type of modern server  
> that stops
> processing to get something else done.

K.
---
http://blitz.io
http://twitter.com/pcapr
-- 
Message  protected by MailGuard: e-mail anti-virus, anti-spam and  
content filtering.http://www.mailguard.com.au/mg
Click here to report this message as spam:
https://login.mailguard.com.au/report/1BXlqzc02P/15tn9vMQnEeSeRDmMoOocd/0.6

Re: blocking during view generation

Posted by Mark Hahn <ma...@elleh.com>.
After re-reading this thread, I see that possibly everyone is
referring to other accesses being blocked for a short period of time
after a doc update.  I should have noticed that was being referred to
instead of what I originally meant, which was the blocking when
changing to source code for a view.

I understand why one has to wait for an update to be reflected in a
view unless you choose to allow reading stale docs.  This is common
sense.  You choose to either wait for the new or accept the old.  I
can't imagine a third possibility.


On Tue, Mar 22, 2011 at 8:42 PM, Mark Hahn <ma...@boutiquing.com> wrote:
> Just to make it clear, I am only talking about the process that
> happens when I change the source code of a view.  I wasn't surprised
> to see accesses to that view being blocked, but seeing all access to
> all views being blocked was stupefying.  Why rebuild views that
> haven't changed?
>
>
> On Tue, Mar 22, 2011 at 8:34 PM, kowsik <ko...@gmail.com> wrote:
>> On Tue, Mar 22, 2011 at 8:28 PM, Andrew Stuart (SuperCoders)
>> <an...@supercoders.com.au> wrote:
>>> This one seems hard to believe - is it true that CouchDB blocks the server
>>> whilst updating views?
>>>
>>> View updates can be alot of work for a server.
>>>
>>> So in reality, queries to the server pause whilst views are updated?
>>>
>>> This doesn't seem practical for any production usage.
>>
>> See this blog: http://labs.mudynamics.com/2011/03/10/blitzio-couchdb-in-production/
>>
>> As long as you have a worker of sorts that watches the _changes feed
>> to "catch up" on the index, things work great. Just like any other
>> piece of software, you have to have some understanding how CouchDB
>> works internally. There is a stale=ok view query which will get you
>> the older revisions so your UI doesn't block at all.
>>
>>> Can someone confirm that this is true, that during production a server will
>>> block whilst views are updated?
>>
>> See above. :) Yes, couch doesn't do background view indexing, though I
>> tend to agree that if there was an Erlang task of sorts that did this
>> automatically, end users can relax even more.
>>
>>> Does anyone else see this as a major issue or am I missing something? I'm
>>> happy to say I have missed the point many times before :-)
>>>
>>> I'm with Mark - I can't think of any other type of modern server that stops
>>> processing to get something else done.
>>
>> K.
>> ---
>> http://blitz.io
>> http://twitter.com/pcapr
>>
>
>
>
> --
> Mark Hahn
> Website Manager
> mark@boutiquing.com
> 949-229-1012
>

Re: Full text search - is it coming? If yes, approx when.

Posted by Gabor Ratky <rg...@rgabostyle.com>.
look at couchdb-lucene. it's worked flawlessly for us and its a rock solid integration of two separate technologies: https://github.com/rnewson/couchdb-lucene

If you're on a mac dev box, you're one 'brew install couchdb-lucene' away, granted that you installed lucene with Homebrew as well (it will automatically configure both to work together immediately).

Gabor

On Mar 28, 2011, at 6:31 AM, Andrew Stuart (SuperCoders) wrote:

> It would be good to know if full text search is coming as a core feature and if yes, approximately when - does anyone know?
> 
> Even an approximate timeframe would be good.
> 
> thanks


Re: Full text search - is it coming? If yes, approx when.

Posted by Luciano Ramalho <lu...@ramalho.org>.
On Mon, Mar 28, 2011 at 8:16 AM, Benoit Chesneau <bc...@gmail.com> wrote:
> CouchBase != CouchDB. CouchBase is using couchdb burt couchdb itself
> is a full opensource project.

Thanks, Benoit, I do undestand that.

It's a bit confusing, though, because all the other CouchDB features
mentioned in the previous CouchIO (not CouchOne) home page were in
fact features of the full opensource CouchDB, if I remember correclty.
It's a pity I don't have a screenshot.

When many of the core developers in an Open Source project belong to
one company, it is easy to think that the company is driving the
project.

That is not a problem by itself, but it does require very careful
communication between all parties (core developers, contributors and
users), particularly when it comes to the product roadmap if the
company business plan involves having proprietary features, and not
just services build around the product.

That said, considering that there are already good Open Source
solutions to the full-text indexing problem, I'd say the CouchDB Open
Source community should focus on other  priorities.


-- 
Luciano Ramalho
programador repentista || stand-up programmer
Twitter: @luciano

Re: Full text search - is it coming? If yes, approx when.

Posted by Benoit Chesneau <bc...@gmail.com>.
On Mon, Mar 28, 2011 at 1:14 PM, Luciano Ramalho <lu...@ramalho.org> wrote:
> On Mon, Mar 28, 2011 at 6:24 AM, Robert Newson <ro...@gmail.com> wrote:
>> To answer the original post, there are no publicly known plans to
>> build a native full-text indexing feature for CouchDB.
>
> Was there a change of plans?
>
> Because before becoming Couchbase, the front page o CouchOne.com had a
> short list of major CouchDB features and the last one mentioned was
> "full-text search (soon)".
>
> Why would a company create expectations over a feature it never
> planned to deliver?
>
CouchBase != CouchDB. CouchBase is using couchdb burt couchdb itself
is a full opensource project.

- benoît

Re: Full text search - is it coming? If yes, approx when.

Posted by Luciano Ramalho <lu...@ramalho.org>.
On Mon, Mar 28, 2011 at 6:24 AM, Robert Newson <ro...@gmail.com> wrote:
> To answer the original post, there are no publicly known plans to
> build a native full-text indexing feature for CouchDB.

Was there a change of plans?

Because before becoming Couchbase, the front page o CouchOne.com had a
short list of major CouchDB features and the last one mentioned was
"full-text search (soon)".

Why would a company create expectations over a feature it never
planned to deliver?



-- 
Luciano Ramalho
programador repentista || stand-up programmer
Twitter: @luciano

Re: Full text search - is it coming? If yes, approx when.

Posted by Robert Dionne <di...@dionne-associates.com>.
Yes this would be a great feature. I've made some modest progress[1], based on the examples in the erlang book, on native FTI. I'm very keen on it as my
use case isn't handled by Lucene. I've used both bitcask[2] and couchdb for storage of the inverted indices but neither seem well suited for something this write heavy
without lots of compaction.

With all due respect to Lucene, I'm not at all interested in a port of it. I think a native solution ought to try and move forward and leverage the features of couchdb.

I also don't think there is any "official" position and/or roadmap, other than what's in JIRA.

YMMV,

Bob

[1] http://dionne.posterous.com/relaxed-searching-in-couchdb
[2] https://github.com/basho/bitcask




On Mar 28, 2011, at 5:51 AM, Andrew Stuart (SuperCoders) wrote:

> Although it may be a huge amount of work, it would still seem to be a necessary feature for the core developers to build?
> 
> It seems a fairly reasonable thing to expect a database could return records that contain "java OR C#" in a "subject line" field, without resorting to external software.  It's not ideal to require installation additional software to get basic functionality going.
> 
> My primary question is  - what is the "official" position on full text search in CouchDB - is it coming, sooner, later or never? Is there an up to date development roadmap for couchdb?  Maybe its on there.
> 
> as
> 
> On 28/03/2011, at 8:24 PM, Robert Newson wrote:
> 
> I have to dispute "There does not seem to be much understanding that
> this could be a killer feature."
> 
> Obviously full-text search is a killer feature, but it's trivially
> available now via couchdb-lucene or elasticsearch.
> 
> What people are asking for is native full-text search which, to me, is
> essentially asking for an Erlang port of Lucene. We'd love this, but
> it's a huge amount of work. Continually asking others to do
> significant amounts of work is also wearying.
> 
> To replace a Lucene-based solution and match its quality and breadth
> is a huge chunk of work and is only necessary to satisfy people who,
> for various reasons, don't want to use Java.
> 
> To answer the original post, there are no publicly known plans to
> build a native full-text indexing feature for CouchDB.
> 
> B.
> 
> On 28 March 2011 10:15, Olafur Arason <ol...@olafura.com> wrote:
>> There does not seem to be much understanding that this could be a killer
>> feature. People are now relying on Lucene which monitors the _changes
>> feed.
>> 
>> Cloudant has done it's own implementation which I gather through the
>> information they have published makes a view out of all your word,
>> they recommend java view because you can then reuse the lexer from
>> Lucene. Then I think they are reusing the reader of the view to make
>> their query. They have a similar syntax as Lucene for the query interface.
>> They are still working on this and I think they don't have that much
>> incentive to opensource it right away. But they have in past both
>> opensourced there technology like BigCouch so I think it's more a
>> matter of when rather then if.
>> 
>> I think this is a good solution for a fulltext search. But I don't think that
>> the java view does not have direct access to the data so it could be
>> slow. But cloudant does clustering on view generation so that helps.
>> 
>> But there is also general problem with the current view system where
>> search technology could be used.
>> 
>> The view are really good at sorting but people are using them to
>> do key matches which they are not designed for. They beginkey and
>> endkey are for sorting ranges and are not good for matching which
>> most resources online are pointing to.
>> 
>> For example when you do:
>> beginkey = ["key11", "key21"]
>> endkey = ["key19", "key21"]
>> 
>> You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
>> ["key12","key22"]...
>> which makes sense when looking up sorting ranges but not using it to
>> match keys. But you can have a range match lookup but only on the
>> last key and never on two keys. So this would work:
>> 
>> beginkey = ["key21", "key11"]
>> endkey = ["key21", "key19"]
>> 
>> The current view interface could be augmented to accept queries
>> and could make them much more powerful then they currently are
>> and just using the keys for sorting and selecting which values you
>> want shown which they are designed to do and do really well.
>> 
>> This would be a killer feature and could use the new infrastructure
>> from Cloudant search.
>> 
>> And don't tell me the Elastic or Lucene interface could do anything
>> close to this :)
>> 
>> Regards,
>> Olafur Arason
>> 
>> On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
>> <an...@supercoders.com.au> wrote:
>>> It would be good to know if full text search is coming as a core feature and
>>> if yes, approximately when - does anyone know?
>>> 
>>> Even an approximate timeframe would be good.
>>> 
>>> thanks
>>> 
>> 
> -- 
> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content filtering.http://www.mailguard.com.au/mg
> Click here to report this message as spam:
> https://login.mailguard.com.au/report/1BZveI1wri/4izG2DWUCf9OUvbAh9DkfT/0


Re: Full text search - is it coming? If yes, approx when.

Posted by Robert Dionne <di...@dionne-associates.com>.
Yes this would be a great feature. I've made some modest progress[1], based on the examples in the erlang book, on native FTI. I'm very keen on it as my
use case isn't handled by Lucene. I've used both bitcask[2] and couchdb for storage of the inverted indices but neither seem well suited for something this write heavy
without lots of compaction.

With all due respect to Lucene, I'm not at all interested in a port of it. I think a native solution ought to try and move forward and leverage the features of couchdb.

I also don't think there is any "official" position and/or roadmap, other than what's in JIRA.

YMMV,

Bob

[1] http://dionne.posterous.com/relaxed-searching-in-couchdb
[2] https://github.com/basho/bitcask




On Mar 28, 2011, at 5:51 AM, Andrew Stuart (SuperCoders) wrote:

> Although it may be a huge amount of work, it would still seem to be a necessary feature for the core developers to build?
> 
> It seems a fairly reasonable thing to expect a database could return records that contain "java OR C#" in a "subject line" field, without resorting to external software.  It's not ideal to require installation additional software to get basic functionality going.
> 
> My primary question is  - what is the "official" position on full text search in CouchDB - is it coming, sooner, later or never? Is there an up to date development roadmap for couchdb?  Maybe its on there.
> 
> as
> 
> On 28/03/2011, at 8:24 PM, Robert Newson wrote:
> 
> I have to dispute "There does not seem to be much understanding that
> this could be a killer feature."
> 
> Obviously full-text search is a killer feature, but it's trivially
> available now via couchdb-lucene or elasticsearch.
> 
> What people are asking for is native full-text search which, to me, is
> essentially asking for an Erlang port of Lucene. We'd love this, but
> it's a huge amount of work. Continually asking others to do
> significant amounts of work is also wearying.
> 
> To replace a Lucene-based solution and match its quality and breadth
> is a huge chunk of work and is only necessary to satisfy people who,
> for various reasons, don't want to use Java.
> 
> To answer the original post, there are no publicly known plans to
> build a native full-text indexing feature for CouchDB.
> 
> B.
> 
> On 28 March 2011 10:15, Olafur Arason <ol...@olafura.com> wrote:
>> There does not seem to be much understanding that this could be a killer
>> feature. People are now relying on Lucene which monitors the _changes
>> feed.
>> 
>> Cloudant has done it's own implementation which I gather through the
>> information they have published makes a view out of all your word,
>> they recommend java view because you can then reuse the lexer from
>> Lucene. Then I think they are reusing the reader of the view to make
>> their query. They have a similar syntax as Lucene for the query interface.
>> They are still working on this and I think they don't have that much
>> incentive to opensource it right away. But they have in past both
>> opensourced there technology like BigCouch so I think it's more a
>> matter of when rather then if.
>> 
>> I think this is a good solution for a fulltext search. But I don't think that
>> the java view does not have direct access to the data so it could be
>> slow. But cloudant does clustering on view generation so that helps.
>> 
>> But there is also general problem with the current view system where
>> search technology could be used.
>> 
>> The view are really good at sorting but people are using them to
>> do key matches which they are not designed for. They beginkey and
>> endkey are for sorting ranges and are not good for matching which
>> most resources online are pointing to.
>> 
>> For example when you do:
>> beginkey = ["key11", "key21"]
>> endkey = ["key19", "key21"]
>> 
>> You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
>> ["key12","key22"]...
>> which makes sense when looking up sorting ranges but not using it to
>> match keys. But you can have a range match lookup but only on the
>> last key and never on two keys. So this would work:
>> 
>> beginkey = ["key21", "key11"]
>> endkey = ["key21", "key19"]
>> 
>> The current view interface could be augmented to accept queries
>> and could make them much more powerful then they currently are
>> and just using the keys for sorting and selecting which values you
>> want shown which they are designed to do and do really well.
>> 
>> This would be a killer feature and could use the new infrastructure
>> from Cloudant search.
>> 
>> And don't tell me the Elastic or Lucene interface could do anything
>> close to this :)
>> 
>> Regards,
>> Olafur Arason
>> 
>> On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
>> <an...@supercoders.com.au> wrote:
>>> It would be good to know if full text search is coming as a core feature and
>>> if yes, approximately when - does anyone know?
>>> 
>>> Even an approximate timeframe would be good.
>>> 
>>> thanks
>>> 
>> 
> -- 
> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content filtering.http://www.mailguard.com.au/mg
> Click here to report this message as spam:
> https://login.mailguard.com.au/report/1BZveI1wri/4izG2DWUCf9OUvbAh9DkfT/0


Re: Full text search - is it coming? If yes, approx when.

Posted by "Andrew Stuart (SuperCoders)" <an...@supercoders.com.au>.
Although it may be a huge amount of work, it would still seem to be a  
necessary feature for the core developers to build?

It seems a fairly reasonable thing to expect a database could return  
records that contain "java OR C#" in a "subject line" field, without  
resorting to external software.  It's not ideal to require  
installation additional software to get basic functionality going.

My primary question is  - what is the "official" position on full text  
search in CouchDB - is it coming, sooner, later or never? Is there an  
up to date development roadmap for couchdb?  Maybe its on there.

as

On 28/03/2011, at 8:24 PM, Robert Newson wrote:

I have to dispute "There does not seem to be much understanding that
this could be a killer feature."

Obviously full-text search is a killer feature, but it's trivially
available now via couchdb-lucene or elasticsearch.

What people are asking for is native full-text search which, to me, is
essentially asking for an Erlang port of Lucene. We'd love this, but
it's a huge amount of work. Continually asking others to do
significant amounts of work is also wearying.

To replace a Lucene-based solution and match its quality and breadth
is a huge chunk of work and is only necessary to satisfy people who,
for various reasons, don't want to use Java.

To answer the original post, there are no publicly known plans to
build a native full-text indexing feature for CouchDB.

B.

On 28 March 2011 10:15, Olafur Arason <ol...@olafura.com> wrote:
> There does not seem to be much understanding that this could be a  
> killer
> feature. People are now relying on Lucene which monitors the _changes
> feed.
>
> Cloudant has done it's own implementation which I gather through the
> information they have published makes a view out of all your word,
> they recommend java view because you can then reuse the lexer from
> Lucene. Then I think they are reusing the reader of the view to make
> their query. They have a similar syntax as Lucene for the query  
> interface.
> They are still working on this and I think they don't have that much
> incentive to opensource it right away. But they have in past both
> opensourced there technology like BigCouch so I think it's more a
> matter of when rather then if.
>
> I think this is a good solution for a fulltext search. But I don't  
> think that
> the java view does not have direct access to the data so it could be
> slow. But cloudant does clustering on view generation so that helps.
>
> But there is also general problem with the current view system where
> search technology could be used.
>
> The view are really good at sorting but people are using them to
> do key matches which they are not designed for. They beginkey and
> endkey are for sorting ranges and are not good for matching which
> most resources online are pointing to.
>
> For example when you do:
> beginkey = ["key11", "key21"]
> endkey = ["key19", "key21"]
>
> You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
> ["key12","key22"]...
> which makes sense when looking up sorting ranges but not using it to
> match keys. But you can have a range match lookup but only on the
> last key and never on two keys. So this would work:
>
> beginkey = ["key21", "key11"]
> endkey = ["key21", "key19"]
>
> The current view interface could be augmented to accept queries
> and could make them much more powerful then they currently are
> and just using the keys for sorting and selecting which values you
> want shown which they are designed to do and do really well.
>
> This would be a killer feature and could use the new infrastructure
> from Cloudant search.
>
> And don't tell me the Elastic or Lucene interface could do anything
> close to this :)
>
> Regards,
> Olafur Arason
>
> On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
> <an...@supercoders.com.au> wrote:
>> It would be good to know if full text search is coming as a core  
>> feature and
>> if yes, approximately when - does anyone know?
>>
>> Even an approximate timeframe would be good.
>>
>> thanks
>>
>
-- 
Message  protected by MailGuard: e-mail anti-virus, anti-spam and  
content filtering.http://www.mailguard.com.au/mg
Click here to report this message as spam:
https://login.mailguard.com.au/report/1BZveI1wri/4izG2DWUCf9OUvbAh9DkfT/0

Re: Full text search - is it coming? If yes, approx when.

Posted by Robert Newson <ro...@gmail.com>.
Hi Martin,

That's very kind, thank you. :)

B.

On 28 March 2011 16:16, Martin Hewitt <ma...@thenoi.se> wrote:
> Bit off-topic Robert, but CouchDB-lucene really is superb, we use it extensively and it's just brilliant, so thanks for contributing it to the Couch community.
>
> Martin
>
> On 28 Mar 2011, at 15:30, Robert Newson wrote:
>
>> I am a CouchDB committer and author of couchdb-lucene. :)
>>
>> B.
>>
>> On 28 March 2011 10:44, Andrew Stuart (SuperCoders)
>> <an...@supercoders.com.au> wrote:
>>> Hi Robert
>>>
>>> "there are no publicly known plans to build a native full-text indexing
>>> feature for CouchDB."
>>>
>>> I don't know who is who around here as yet - are you commenting from inside
>>> knowledge or as an end user/developer?
>>>
>>> Thanks
>>>
>>>
>>> On 28/03/2011, at 8:24 PM, Robert Newson wrote:
>>>
>>> I have to dispute "There does not seem to be much understanding that
>>> this could be a killer feature."
>>>
>>> Obviously full-text search is a killer feature, but it's trivially
>>> available now via couchdb-lucene or elasticsearch.
>>>
>>> What people are asking for is native full-text search which, to me, is
>>> essentially asking for an Erlang port of Lucene. We'd love this, but
>>> it's a huge amount of work. Continually asking others to do
>>> significant amounts of work is also wearying.
>>>
>>> To replace a Lucene-based solution and match its quality and breadth
>>> is a huge chunk of work and is only necessary to satisfy people who,
>>> for various reasons, don't want to use Java.
>>>
>>> To answer the original post, there are no publicly known plans to
>>> build a native full-text indexing feature for CouchDB.
>>>
>>> B.
>>>
>>> On 28 March 2011 10:15, Olafur Arason <ol...@olafura.com> wrote:
>>>>
>>>> There does not seem to be much understanding that this could be a killer
>>>> feature. People are now relying on Lucene which monitors the _changes
>>>> feed.
>>>>
>>>> Cloudant has done it's own implementation which I gather through the
>>>> information they have published makes a view out of all your word,
>>>> they recommend java view because you can then reuse the lexer from
>>>> Lucene. Then I think they are reusing the reader of the view to make
>>>> their query. They have a similar syntax as Lucene for the query interface.
>>>> They are still working on this and I think they don't have that much
>>>> incentive to opensource it right away. But they have in past both
>>>> opensourced there technology like BigCouch so I think it's more a
>>>> matter of when rather then if.
>>>>
>>>> I think this is a good solution for a fulltext search. But I don't think
>>>> that
>>>> the java view does not have direct access to the data so it could be
>>>> slow. But cloudant does clustering on view generation so that helps.
>>>>
>>>> But there is also general problem with the current view system where
>>>> search technology could be used.
>>>>
>>>> The view are really good at sorting but people are using them to
>>>> do key matches which they are not designed for. They beginkey and
>>>> endkey are for sorting ranges and are not good for matching which
>>>> most resources online are pointing to.
>>>>
>>>> For example when you do:
>>>> beginkey = ["key11", "key21"]
>>>> endkey = ["key19", "key21"]
>>>>
>>>> You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
>>>> ["key12","key22"]...
>>>> which makes sense when looking up sorting ranges but not using it to
>>>> match keys. But you can have a range match lookup but only on the
>>>> last key and never on two keys. So this would work:
>>>>
>>>> beginkey = ["key21", "key11"]
>>>> endkey = ["key21", "key19"]
>>>>
>>>> The current view interface could be augmented to accept queries
>>>> and could make them much more powerful then they currently are
>>>> and just using the keys for sorting and selecting which values you
>>>> want shown which they are designed to do and do really well.
>>>>
>>>> This would be a killer feature and could use the new infrastructure
>>>> from Cloudant search.
>>>>
>>>> And don't tell me the Elastic or Lucene interface could do anything
>>>> close to this :)
>>>>
>>>> Regards,
>>>> Olafur Arason
>>>>
>>>> On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
>>>> <an...@supercoders.com.au> wrote:
>>>>>
>>>>> It would be good to know if full text search is coming as a core feature
>>>>> and
>>>>> if yes, approximately when - does anyone know?
>>>>>
>>>>> Even an approximate timeframe would be good.
>>>>>
>>>>> thanks
>>>>>
>>>>
>>> --
>>> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
>>> filtering.http://www.mailguard.com.au/mg
>>> Click here to report this message as spam:
>>> https://login.mailguard.com.au/report/1BZveI1wri/4izG2DWUCf9OUvbAh9DkfT/0
>>>
>
>

Re: Full text search - is it coming? If yes, approx when.

Posted by Martin Hewitt <ma...@thenoi.se>.
Bit off-topic Robert, but CouchDB-lucene really is superb, we use it extensively and it's just brilliant, so thanks for contributing it to the Couch community.

Martin 

On 28 Mar 2011, at 15:30, Robert Newson wrote:

> I am a CouchDB committer and author of couchdb-lucene. :)
> 
> B.
> 
> On 28 March 2011 10:44, Andrew Stuart (SuperCoders)
> <an...@supercoders.com.au> wrote:
>> Hi Robert
>> 
>> "there are no publicly known plans to build a native full-text indexing
>> feature for CouchDB."
>> 
>> I don't know who is who around here as yet - are you commenting from inside
>> knowledge or as an end user/developer?
>> 
>> Thanks
>> 
>> 
>> On 28/03/2011, at 8:24 PM, Robert Newson wrote:
>> 
>> I have to dispute "There does not seem to be much understanding that
>> this could be a killer feature."
>> 
>> Obviously full-text search is a killer feature, but it's trivially
>> available now via couchdb-lucene or elasticsearch.
>> 
>> What people are asking for is native full-text search which, to me, is
>> essentially asking for an Erlang port of Lucene. We'd love this, but
>> it's a huge amount of work. Continually asking others to do
>> significant amounts of work is also wearying.
>> 
>> To replace a Lucene-based solution and match its quality and breadth
>> is a huge chunk of work and is only necessary to satisfy people who,
>> for various reasons, don't want to use Java.
>> 
>> To answer the original post, there are no publicly known plans to
>> build a native full-text indexing feature for CouchDB.
>> 
>> B.
>> 
>> On 28 March 2011 10:15, Olafur Arason <ol...@olafura.com> wrote:
>>> 
>>> There does not seem to be much understanding that this could be a killer
>>> feature. People are now relying on Lucene which monitors the _changes
>>> feed.
>>> 
>>> Cloudant has done it's own implementation which I gather through the
>>> information they have published makes a view out of all your word,
>>> they recommend java view because you can then reuse the lexer from
>>> Lucene. Then I think they are reusing the reader of the view to make
>>> their query. They have a similar syntax as Lucene for the query interface.
>>> They are still working on this and I think they don't have that much
>>> incentive to opensource it right away. But they have in past both
>>> opensourced there technology like BigCouch so I think it's more a
>>> matter of when rather then if.
>>> 
>>> I think this is a good solution for a fulltext search. But I don't think
>>> that
>>> the java view does not have direct access to the data so it could be
>>> slow. But cloudant does clustering on view generation so that helps.
>>> 
>>> But there is also general problem with the current view system where
>>> search technology could be used.
>>> 
>>> The view are really good at sorting but people are using them to
>>> do key matches which they are not designed for. They beginkey and
>>> endkey are for sorting ranges and are not good for matching which
>>> most resources online are pointing to.
>>> 
>>> For example when you do:
>>> beginkey = ["key11", "key21"]
>>> endkey = ["key19", "key21"]
>>> 
>>> You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
>>> ["key12","key22"]...
>>> which makes sense when looking up sorting ranges but not using it to
>>> match keys. But you can have a range match lookup but only on the
>>> last key and never on two keys. So this would work:
>>> 
>>> beginkey = ["key21", "key11"]
>>> endkey = ["key21", "key19"]
>>> 
>>> The current view interface could be augmented to accept queries
>>> and could make them much more powerful then they currently are
>>> and just using the keys for sorting and selecting which values you
>>> want shown which they are designed to do and do really well.
>>> 
>>> This would be a killer feature and could use the new infrastructure
>>> from Cloudant search.
>>> 
>>> And don't tell me the Elastic or Lucene interface could do anything
>>> close to this :)
>>> 
>>> Regards,
>>> Olafur Arason
>>> 
>>> On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
>>> <an...@supercoders.com.au> wrote:
>>>> 
>>>> It would be good to know if full text search is coming as a core feature
>>>> and
>>>> if yes, approximately when - does anyone know?
>>>> 
>>>> Even an approximate timeframe would be good.
>>>> 
>>>> thanks
>>>> 
>>> 
>> --
>> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
>> filtering.http://www.mailguard.com.au/mg
>> Click here to report this message as spam:
>> https://login.mailguard.com.au/report/1BZveI1wri/4izG2DWUCf9OUvbAh9DkfT/0
>> 


Re: Full text search - is it coming? If yes, approx when.

Posted by Benoit Chesneau <bc...@gmail.com>.
Would be cool to have a nifs integration of Apache Lucy. It may solve
the problem.

- benoit

On Mon, Mar 28, 2011 at 5:17 PM, Olafur Arason <ol...@olafura.com> wrote:
> I love the power of Lucene but it's not needed for many usecases
> and can even be gutted like Cloudant is doing with their search
> using the lexer from Lucene.
>
> But most of the time people need quick and dirty search and
> even search integration with views. Then you would maybe have
> a really simple lexer. And have it built in. If people need more
> power they would use Lucene.
>
> It's like using a Ferrari to go to the store, it's cool but an overkill.
>
> Hope you keep up the good work, couchdb-lucene is really easy
> to use.
>
> Regards,
> Olafur Arason
>
> Ps I was talking to an NLP expert and I realize that there is so
> much to searching. Especially doing it right that I think nobody
> will be able to re-implement Lucene anytime soon.
>
> On Mon, Mar 28, 2011 at 14:30, Robert Newson <ro...@gmail.com> wrote:
>> I am a CouchDB committer and author of couchdb-lucene. :)
>>
>> B.
>>
>> On 28 March 2011 10:44, Andrew Stuart (SuperCoders)
>> <an...@supercoders.com.au> wrote:
>>> Hi Robert
>>>
>>> "there are no publicly known plans to build a native full-text indexing
>>> feature for CouchDB."
>>>
>>> I don't know who is who around here as yet - are you commenting from inside
>>> knowledge or as an end user/developer?
>>>
>>> Thanks
>>>
>>>
>>> On 28/03/2011, at 8:24 PM, Robert Newson wrote:
>>>
>>> I have to dispute "There does not seem to be much understanding that
>>> this could be a killer feature."
>>>
>>> Obviously full-text search is a killer feature, but it's trivially
>>> available now via couchdb-lucene or elasticsearch.
>>>
>>> What people are asking for is native full-text search which, to me, is
>>> essentially asking for an Erlang port of Lucene. We'd love this, but
>>> it's a huge amount of work. Continually asking others to do
>>> significant amounts of work is also wearying.
>>>
>>> To replace a Lucene-based solution and match its quality and breadth
>>> is a huge chunk of work and is only necessary to satisfy people who,
>>> for various reasons, don't want to use Java.
>>>
>>> To answer the original post, there are no publicly known plans to
>>> build a native full-text indexing feature for CouchDB.
>>>
>>> B.
>>>
>>> On 28 March 2011 10:15, Olafur Arason <ol...@olafura.com> wrote:
>>>>
>>>> There does not seem to be much understanding that this could be a killer
>>>> feature. People are now relying on Lucene which monitors the _changes
>>>> feed.
>>>>
>>>> Cloudant has done it's own implementation which I gather through the
>>>> information they have published makes a view out of all your word,
>>>> they recommend java view because you can then reuse the lexer from
>>>> Lucene. Then I think they are reusing the reader of the view to make
>>>> their query. They have a similar syntax as Lucene for the query interface.
>>>> They are still working on this and I think they don't have that much
>>>> incentive to opensource it right away. But they have in past both
>>>> opensourced there technology like BigCouch so I think it's more a
>>>> matter of when rather then if.
>>>>
>>>> I think this is a good solution for a fulltext search. But I don't think
>>>> that
>>>> the java view does not have direct access to the data so it could be
>>>> slow. But cloudant does clustering on view generation so that helps.
>>>>
>>>> But there is also general problem with the current view system where
>>>> search technology could be used.
>>>>
>>>> The view are really good at sorting but people are using them to
>>>> do key matches which they are not designed for. They beginkey and
>>>> endkey are for sorting ranges and are not good for matching which
>>>> most resources online are pointing to.
>>>>
>>>> For example when you do:
>>>> beginkey = ["key11", "key21"]
>>>> endkey = ["key19", "key21"]
>>>>
>>>> You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
>>>> ["key12","key22"]...
>>>> which makes sense when looking up sorting ranges but not using it to
>>>> match keys. But you can have a range match lookup but only on the
>>>> last key and never on two keys. So this would work:
>>>>
>>>> beginkey = ["key21", "key11"]
>>>> endkey = ["key21", "key19"]
>>>>
>>>> The current view interface could be augmented to accept queries
>>>> and could make them much more powerful then they currently are
>>>> and just using the keys for sorting and selecting which values you
>>>> want shown which they are designed to do and do really well.
>>>>
>>>> This would be a killer feature and could use the new infrastructure
>>>> from Cloudant search.
>>>>
>>>> And don't tell me the Elastic or Lucene interface could do anything
>>>> close to this :)
>>>>
>>>> Regards,
>>>> Olafur Arason
>>>>
>>>> On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
>>>> <an...@supercoders.com.au> wrote:
>>>>>
>>>>> It would be good to know if full text search is coming as a core feature
>>>>> and
>>>>> if yes, approximately when - does anyone know?
>>>>>
>>>>> Even an approximate timeframe would be good.
>>>>>
>>>>> thanks
>>>>>
>>>>
>>> --
>>> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
>>> filtering.http://www.mailguard.com.au/mg
>>> Click here to report this message as spam:
>>> https://login.mailguard.com.au/report/1BZveI1wri/4izG2DWUCf9OUvbAh9DkfT/0
>>>
>>
>

Re: Full text search - is it coming? If yes, approx when.

Posted by Robert Dionne <di...@dionne-associates.com>.
Yes, I agree wholeheartedly with this view. I would go further and speculate (as I haven't kept up with Lucene features) that Lucene
is still primarily focused on lexical techniques and any real NLP functionality is done in Lucene plugins.

The core idea in FTI is easy, an inverted index. Quick and dirty search[1] is precisely the use case I had in mind. I wanted something like the "jumpTo"
functionality in Futon that would work for more than just _ids and I wanted the capability to filter the results by various fields on the documents and
have it index incrementally as updates occur. This isn't that difficult and couchdb isn't so hard to build plugins for, though one does need to grok some
of the internals. It's also kind of surprising that a database admin GUI doesn't have this simple capability.

Doing this in a way that scales to large dbs, is distributed, has all the wonderful features Lucene has acquired and so forth is a different matter.

Anyway I did this as much as an erlang learning exercise and planning to revisit it, now that I've picked up some skills,

Cheers,

Bob

[1] https://github.com/bdionne/bitstore/tree/master/src/search


On Mar 28, 2011, at 11:17 AM, Olafur Arason wrote:

> I love the power of Lucene but it's not needed for many usecases
> and can even be gutted like Cloudant is doing with their search
> using the lexer from Lucene.
> 
> But most of the time people need quick and dirty search and
> even search integration with views. Then you would maybe have
> a really simple lexer. And have it built in. If people need more
> power they would use Lucene.
> 
> It's like using a Ferrari to go to the store, it's cool but an overkill.
> 
> Hope you keep up the good work, couchdb-lucene is really easy
> to use.
> 
> Regards,
> Olafur Arason
> 
> Ps I was talking to an NLP expert and I realize that there is so
> much to searching. Especially doing it right that I think nobody
> will be able to re-implement Lucene anytime soon.
> 
> On Mon, Mar 28, 2011 at 14:30, Robert Newson <ro...@gmail.com> wrote:
>> I am a CouchDB committer and author of couchdb-lucene. :)
>> 
>> B.
>> 
>> On 28 March 2011 10:44, Andrew Stuart (SuperCoders)
>> <an...@supercoders.com.au> wrote:
>>> Hi Robert
>>> 
>>> "there are no publicly known plans to build a native full-text indexing
>>> feature for CouchDB."
>>> 
>>> I don't know who is who around here as yet - are you commenting from inside
>>> knowledge or as an end user/developer?
>>> 
>>> Thanks
>>> 
>>> 
>>> On 28/03/2011, at 8:24 PM, Robert Newson wrote:
>>> 
>>> I have to dispute "There does not seem to be much understanding that
>>> this could be a killer feature."
>>> 
>>> Obviously full-text search is a killer feature, but it's trivially
>>> available now via couchdb-lucene or elasticsearch.
>>> 
>>> What people are asking for is native full-text search which, to me, is
>>> essentially asking for an Erlang port of Lucene. We'd love this, but
>>> it's a huge amount of work. Continually asking others to do
>>> significant amounts of work is also wearying.
>>> 
>>> To replace a Lucene-based solution and match its quality and breadth
>>> is a huge chunk of work and is only necessary to satisfy people who,
>>> for various reasons, don't want to use Java.
>>> 
>>> To answer the original post, there are no publicly known plans to
>>> build a native full-text indexing feature for CouchDB.
>>> 
>>> B.
>>> 
>>> On 28 March 2011 10:15, Olafur Arason <ol...@olafura.com> wrote:
>>>> 
>>>> There does not seem to be much understanding that this could be a killer
>>>> feature. People are now relying on Lucene which monitors the _changes
>>>> feed.
>>>> 
>>>> Cloudant has done it's own implementation which I gather through the
>>>> information they have published makes a view out of all your word,
>>>> they recommend java view because you can then reuse the lexer from
>>>> Lucene. Then I think they are reusing the reader of the view to make
>>>> their query. They have a similar syntax as Lucene for the query interface.
>>>> They are still working on this and I think they don't have that much
>>>> incentive to opensource it right away. But they have in past both
>>>> opensourced there technology like BigCouch so I think it's more a
>>>> matter of when rather then if.
>>>> 
>>>> I think this is a good solution for a fulltext search. But I don't think
>>>> that
>>>> the java view does not have direct access to the data so it could be
>>>> slow. But cloudant does clustering on view generation so that helps.
>>>> 
>>>> But there is also general problem with the current view system where
>>>> search technology could be used.
>>>> 
>>>> The view are really good at sorting but people are using them to
>>>> do key matches which they are not designed for. They beginkey and
>>>> endkey are for sorting ranges and are not good for matching which
>>>> most resources online are pointing to.
>>>> 
>>>> For example when you do:
>>>> beginkey = ["key11", "key21"]
>>>> endkey = ["key19", "key21"]
>>>> 
>>>> You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
>>>> ["key12","key22"]...
>>>> which makes sense when looking up sorting ranges but not using it to
>>>> match keys. But you can have a range match lookup but only on the
>>>> last key and never on two keys. So this would work:
>>>> 
>>>> beginkey = ["key21", "key11"]
>>>> endkey = ["key21", "key19"]
>>>> 
>>>> The current view interface could be augmented to accept queries
>>>> and could make them much more powerful then they currently are
>>>> and just using the keys for sorting and selecting which values you
>>>> want shown which they are designed to do and do really well.
>>>> 
>>>> This would be a killer feature and could use the new infrastructure
>>>> from Cloudant search.
>>>> 
>>>> And don't tell me the Elastic or Lucene interface could do anything
>>>> close to this :)
>>>> 
>>>> Regards,
>>>> Olafur Arason
>>>> 
>>>> On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
>>>> <an...@supercoders.com.au> wrote:
>>>>> 
>>>>> It would be good to know if full text search is coming as a core feature
>>>>> and
>>>>> if yes, approximately when - does anyone know?
>>>>> 
>>>>> Even an approximate timeframe would be good.
>>>>> 
>>>>> thanks
>>>>> 
>>>> 
>>> --
>>> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
>>> filtering.http://www.mailguard.com.au/mg
>>> Click here to report this message as spam:
>>> https://login.mailguard.com.au/report/1BZveI1wri/4izG2DWUCf9OUvbAh9DkfT/0
>>> 
>> 


Re: Full text search - is it coming? If yes, approx when.

Posted by Olafur Arason <ol...@olafura.com>.
I love the power of Lucene but it's not needed for many usecases
and can even be gutted like Cloudant is doing with their search
using the lexer from Lucene.

But most of the time people need quick and dirty search and
even search integration with views. Then you would maybe have
a really simple lexer. And have it built in. If people need more
power they would use Lucene.

It's like using a Ferrari to go to the store, it's cool but an overkill.

Hope you keep up the good work, couchdb-lucene is really easy
to use.

Regards,
Olafur Arason

Ps I was talking to an NLP expert and I realize that there is so
much to searching. Especially doing it right that I think nobody
will be able to re-implement Lucene anytime soon.

On Mon, Mar 28, 2011 at 14:30, Robert Newson <ro...@gmail.com> wrote:
> I am a CouchDB committer and author of couchdb-lucene. :)
>
> B.
>
> On 28 March 2011 10:44, Andrew Stuart (SuperCoders)
> <an...@supercoders.com.au> wrote:
>> Hi Robert
>>
>> "there are no publicly known plans to build a native full-text indexing
>> feature for CouchDB."
>>
>> I don't know who is who around here as yet - are you commenting from inside
>> knowledge or as an end user/developer?
>>
>> Thanks
>>
>>
>> On 28/03/2011, at 8:24 PM, Robert Newson wrote:
>>
>> I have to dispute "There does not seem to be much understanding that
>> this could be a killer feature."
>>
>> Obviously full-text search is a killer feature, but it's trivially
>> available now via couchdb-lucene or elasticsearch.
>>
>> What people are asking for is native full-text search which, to me, is
>> essentially asking for an Erlang port of Lucene. We'd love this, but
>> it's a huge amount of work. Continually asking others to do
>> significant amounts of work is also wearying.
>>
>> To replace a Lucene-based solution and match its quality and breadth
>> is a huge chunk of work and is only necessary to satisfy people who,
>> for various reasons, don't want to use Java.
>>
>> To answer the original post, there are no publicly known plans to
>> build a native full-text indexing feature for CouchDB.
>>
>> B.
>>
>> On 28 March 2011 10:15, Olafur Arason <ol...@olafura.com> wrote:
>>>
>>> There does not seem to be much understanding that this could be a killer
>>> feature. People are now relying on Lucene which monitors the _changes
>>> feed.
>>>
>>> Cloudant has done it's own implementation which I gather through the
>>> information they have published makes a view out of all your word,
>>> they recommend java view because you can then reuse the lexer from
>>> Lucene. Then I think they are reusing the reader of the view to make
>>> their query. They have a similar syntax as Lucene for the query interface.
>>> They are still working on this and I think they don't have that much
>>> incentive to opensource it right away. But they have in past both
>>> opensourced there technology like BigCouch so I think it's more a
>>> matter of when rather then if.
>>>
>>> I think this is a good solution for a fulltext search. But I don't think
>>> that
>>> the java view does not have direct access to the data so it could be
>>> slow. But cloudant does clustering on view generation so that helps.
>>>
>>> But there is also general problem with the current view system where
>>> search technology could be used.
>>>
>>> The view are really good at sorting but people are using them to
>>> do key matches which they are not designed for. They beginkey and
>>> endkey are for sorting ranges and are not good for matching which
>>> most resources online are pointing to.
>>>
>>> For example when you do:
>>> beginkey = ["key11", "key21"]
>>> endkey = ["key19", "key21"]
>>>
>>> You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
>>> ["key12","key22"]...
>>> which makes sense when looking up sorting ranges but not using it to
>>> match keys. But you can have a range match lookup but only on the
>>> last key and never on two keys. So this would work:
>>>
>>> beginkey = ["key21", "key11"]
>>> endkey = ["key21", "key19"]
>>>
>>> The current view interface could be augmented to accept queries
>>> and could make them much more powerful then they currently are
>>> and just using the keys for sorting and selecting which values you
>>> want shown which they are designed to do and do really well.
>>>
>>> This would be a killer feature and could use the new infrastructure
>>> from Cloudant search.
>>>
>>> And don't tell me the Elastic or Lucene interface could do anything
>>> close to this :)
>>>
>>> Regards,
>>> Olafur Arason
>>>
>>> On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
>>> <an...@supercoders.com.au> wrote:
>>>>
>>>> It would be good to know if full text search is coming as a core feature
>>>> and
>>>> if yes, approximately when - does anyone know?
>>>>
>>>> Even an approximate timeframe would be good.
>>>>
>>>> thanks
>>>>
>>>
>> --
>> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
>> filtering.http://www.mailguard.com.au/mg
>> Click here to report this message as spam:
>> https://login.mailguard.com.au/report/1BZveI1wri/4izG2DWUCf9OUvbAh9DkfT/0
>>
>

Re: Full text search - is it coming? If yes, approx when.

Posted by Robert Newson <ro...@gmail.com>.
I am a CouchDB committer and author of couchdb-lucene. :)

B.

On 28 March 2011 10:44, Andrew Stuart (SuperCoders)
<an...@supercoders.com.au> wrote:
> Hi Robert
>
> "there are no publicly known plans to build a native full-text indexing
> feature for CouchDB."
>
> I don't know who is who around here as yet - are you commenting from inside
> knowledge or as an end user/developer?
>
> Thanks
>
>
> On 28/03/2011, at 8:24 PM, Robert Newson wrote:
>
> I have to dispute "There does not seem to be much understanding that
> this could be a killer feature."
>
> Obviously full-text search is a killer feature, but it's trivially
> available now via couchdb-lucene or elasticsearch.
>
> What people are asking for is native full-text search which, to me, is
> essentially asking for an Erlang port of Lucene. We'd love this, but
> it's a huge amount of work. Continually asking others to do
> significant amounts of work is also wearying.
>
> To replace a Lucene-based solution and match its quality and breadth
> is a huge chunk of work and is only necessary to satisfy people who,
> for various reasons, don't want to use Java.
>
> To answer the original post, there are no publicly known plans to
> build a native full-text indexing feature for CouchDB.
>
> B.
>
> On 28 March 2011 10:15, Olafur Arason <ol...@olafura.com> wrote:
>>
>> There does not seem to be much understanding that this could be a killer
>> feature. People are now relying on Lucene which monitors the _changes
>> feed.
>>
>> Cloudant has done it's own implementation which I gather through the
>> information they have published makes a view out of all your word,
>> they recommend java view because you can then reuse the lexer from
>> Lucene. Then I think they are reusing the reader of the view to make
>> their query. They have a similar syntax as Lucene for the query interface.
>> They are still working on this and I think they don't have that much
>> incentive to opensource it right away. But they have in past both
>> opensourced there technology like BigCouch so I think it's more a
>> matter of when rather then if.
>>
>> I think this is a good solution for a fulltext search. But I don't think
>> that
>> the java view does not have direct access to the data so it could be
>> slow. But cloudant does clustering on view generation so that helps.
>>
>> But there is also general problem with the current view system where
>> search technology could be used.
>>
>> The view are really good at sorting but people are using them to
>> do key matches which they are not designed for. They beginkey and
>> endkey are for sorting ranges and are not good for matching which
>> most resources online are pointing to.
>>
>> For example when you do:
>> beginkey = ["key11", "key21"]
>> endkey = ["key19", "key21"]
>>
>> You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
>> ["key12","key22"]...
>> which makes sense when looking up sorting ranges but not using it to
>> match keys. But you can have a range match lookup but only on the
>> last key and never on two keys. So this would work:
>>
>> beginkey = ["key21", "key11"]
>> endkey = ["key21", "key19"]
>>
>> The current view interface could be augmented to accept queries
>> and could make them much more powerful then they currently are
>> and just using the keys for sorting and selecting which values you
>> want shown which they are designed to do and do really well.
>>
>> This would be a killer feature and could use the new infrastructure
>> from Cloudant search.
>>
>> And don't tell me the Elastic or Lucene interface could do anything
>> close to this :)
>>
>> Regards,
>> Olafur Arason
>>
>> On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
>> <an...@supercoders.com.au> wrote:
>>>
>>> It would be good to know if full text search is coming as a core feature
>>> and
>>> if yes, approximately when - does anyone know?
>>>
>>> Even an approximate timeframe would be good.
>>>
>>> thanks
>>>
>>
> --
> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
> filtering.http://www.mailguard.com.au/mg
> Click here to report this message as spam:
> https://login.mailguard.com.au/report/1BZveI1wri/4izG2DWUCf9OUvbAh9DkfT/0
>

Re: Full text search - is it coming? If yes, approx when.

Posted by "Andrew Stuart (SuperCoders)" <an...@supercoders.com.au>.
Hi Robert

"there are no publicly known plans to build a native full-text  
indexing feature for CouchDB."

I don't know who is who around here as yet - are you commenting from  
inside knowledge or as an end user/developer?

Thanks


On 28/03/2011, at 8:24 PM, Robert Newson wrote:

I have to dispute "There does not seem to be much understanding that
this could be a killer feature."

Obviously full-text search is a killer feature, but it's trivially
available now via couchdb-lucene or elasticsearch.

What people are asking for is native full-text search which, to me, is
essentially asking for an Erlang port of Lucene. We'd love this, but
it's a huge amount of work. Continually asking others to do
significant amounts of work is also wearying.

To replace a Lucene-based solution and match its quality and breadth
is a huge chunk of work and is only necessary to satisfy people who,
for various reasons, don't want to use Java.

To answer the original post, there are no publicly known plans to
build a native full-text indexing feature for CouchDB.

B.

On 28 March 2011 10:15, Olafur Arason <ol...@olafura.com> wrote:
> There does not seem to be much understanding that this could be a  
> killer
> feature. People are now relying on Lucene which monitors the _changes
> feed.
>
> Cloudant has done it's own implementation which I gather through the
> information they have published makes a view out of all your word,
> they recommend java view because you can then reuse the lexer from
> Lucene. Then I think they are reusing the reader of the view to make
> their query. They have a similar syntax as Lucene for the query  
> interface.
> They are still working on this and I think they don't have that much
> incentive to opensource it right away. But they have in past both
> opensourced there technology like BigCouch so I think it's more a
> matter of when rather then if.
>
> I think this is a good solution for a fulltext search. But I don't  
> think that
> the java view does not have direct access to the data so it could be
> slow. But cloudant does clustering on view generation so that helps.
>
> But there is also general problem with the current view system where
> search technology could be used.
>
> The view are really good at sorting but people are using them to
> do key matches which they are not designed for. They beginkey and
> endkey are for sorting ranges and are not good for matching which
> most resources online are pointing to.
>
> For example when you do:
> beginkey = ["key11", "key21"]
> endkey = ["key19", "key21"]
>
> You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
> ["key12","key22"]...
> which makes sense when looking up sorting ranges but not using it to
> match keys. But you can have a range match lookup but only on the
> last key and never on two keys. So this would work:
>
> beginkey = ["key21", "key11"]
> endkey = ["key21", "key19"]
>
> The current view interface could be augmented to accept queries
> and could make them much more powerful then they currently are
> and just using the keys for sorting and selecting which values you
> want shown which they are designed to do and do really well.
>
> This would be a killer feature and could use the new infrastructure
> from Cloudant search.
>
> And don't tell me the Elastic or Lucene interface could do anything
> close to this :)
>
> Regards,
> Olafur Arason
>
> On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
> <an...@supercoders.com.au> wrote:
>> It would be good to know if full text search is coming as a core  
>> feature and
>> if yes, approximately when - does anyone know?
>>
>> Even an approximate timeframe would be good.
>>
>> thanks
>>
>
-- 
Message  protected by MailGuard: e-mail anti-virus, anti-spam and  
content filtering.http://www.mailguard.com.au/mg
Click here to report this message as spam:
https://login.mailguard.com.au/report/1BZveI1wri/4izG2DWUCf9OUvbAh9DkfT/0

Re: Full text search - is it coming? If yes, approx when.

Posted by Robert Newson <ro...@gmail.com>.
I have to dispute "There does not seem to be much understanding that
this could be a killer feature."

Obviously full-text search is a killer feature, but it's trivially
available now via couchdb-lucene or elasticsearch.

What people are asking for is native full-text search which, to me, is
essentially asking for an Erlang port of Lucene. We'd love this, but
it's a huge amount of work. Continually asking others to do
significant amounts of work is also wearying.

To replace a Lucene-based solution and match its quality and breadth
is a huge chunk of work and is only necessary to satisfy people who,
for various reasons, don't want to use Java.

To answer the original post, there are no publicly known plans to
build a native full-text indexing feature for CouchDB.

B.

On 28 March 2011 10:15, Olafur Arason <ol...@olafura.com> wrote:
> There does not seem to be much understanding that this could be a killer
> feature. People are now relying on Lucene which monitors the _changes
> feed.
>
> Cloudant has done it's own implementation which I gather through the
> information they have published makes a view out of all your word,
> they recommend java view because you can then reuse the lexer from
> Lucene. Then I think they are reusing the reader of the view to make
> their query. They have a similar syntax as Lucene for the query interface.
> They are still working on this and I think they don't have that much
> incentive to opensource it right away. But they have in past both
> opensourced there technology like BigCouch so I think it's more a
> matter of when rather then if.
>
> I think this is a good solution for a fulltext search. But I don't think that
> the java view does not have direct access to the data so it could be
> slow. But cloudant does clustering on view generation so that helps.
>
> But there is also general problem with the current view system where
> search technology could be used.
>
> The view are really good at sorting but people are using them to
> do key matches which they are not designed for. They beginkey and
> endkey are for sorting ranges and are not good for matching which
> most resources online are pointing to.
>
> For example when you do:
> beginkey = ["key11", "key21"]
> endkey = ["key19", "key21"]
>
> You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
> ["key12","key22"]...
> which makes sense when looking up sorting ranges but not using it to
> match keys. But you can have a range match lookup but only on the
> last key and never on two keys. So this would work:
>
> beginkey = ["key21", "key11"]
> endkey = ["key21", "key19"]
>
> The current view interface could be augmented to accept queries
> and could make them much more powerful then they currently are
> and just using the keys for sorting and selecting which values you
> want shown which they are designed to do and do really well.
>
> This would be a killer feature and could use the new infrastructure
> from Cloudant search.
>
> And don't tell me the Elastic or Lucene interface could do anything
> close to this :)
>
> Regards,
> Olafur Arason
>
> On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
> <an...@supercoders.com.au> wrote:
>> It would be good to know if full text search is coming as a core feature and
>> if yes, approximately when - does anyone know?
>>
>> Even an approximate timeframe would be good.
>>
>> thanks
>>
>

Re: Full text search - is it coming? If yes, approx when.

Posted by Benoit Chesneau <bc...@gmail.com>.
On Mon, Mar 28, 2011 at 6:31 AM, Andrew Stuart (SuperCoders)
<an...@supercoders.com.au> wrote:
> It would be good to know if full text search is coming as a core feature and
> if yes, approximately when - does anyone know?
>
> Even an approximate timeframe would be good.
>
> thanks
>

As far as I know there is nothing planned about that feature
currently. Can you eventually open a ticket describing the feature you
would like.

There are some work from basho porting some analyzer in full erlang in
riak search that could be interresting. Having a rest api based on
couch would be really interresting. ALso for me such implementation
should support distruibution across process natively/.

- benoît

Re: Full text search - is it coming? If yes, approx when.

Posted by Gerardo Di Iorio <ar...@gmail.com>.
hi,
this is an impletation of fts in erlang with couchdb
https://github.com/bdionne/bitstore/tree/master/src/search/

2011/3/28 Andrew Stuart (SuperCoders) <an...@supercoders.com.au>:
> It would be good to know if full text search is coming as a core feature and
> if yes, approximately when - does anyone know?
>
> Even an approximate timeframe would be good.
>
> thanks
>



-- 
http://www.gerardodiiorio.com
Lo scopo del lavoro è quello di guadagnarsi il tempo libero  (Aristotele)
The purpose of the job is to gain leisure  (Aristotele)

Re: Full text search - is it coming? If yes, approx when.

Posted by Olafur Arason <ol...@olafura.com>.
There does not seem to be much understanding that this could be a killer
feature. People are now relying on Lucene which monitors the _changes
feed.

Cloudant has done it's own implementation which I gather through the
information they have published makes a view out of all your word,
they recommend java view because you can then reuse the lexer from
Lucene. Then I think they are reusing the reader of the view to make
their query. They have a similar syntax as Lucene for the query interface.
They are still working on this and I think they don't have that much
incentive to opensource it right away. But they have in past both
opensourced there technology like BigCouch so I think it's more a
matter of when rather then if.

I think this is a good solution for a fulltext search. But I don't think that
the java view does not have direct access to the data so it could be
slow. But cloudant does clustering on view generation so that helps.

But there is also general problem with the current view system where
search technology could be used.

The view are really good at sorting but people are using them to
do key matches which they are not designed for. They beginkey and
endkey are for sorting ranges and are not good for matching which
most resources online are pointing to.

For example when you do:
beginkey = ["key11", "key21"]
endkey = ["key19", "key21"]

You get ["key11","key22"], ["key11", "key23"] ... ["key12","key21"],
["key12","key22"]...
which makes sense when looking up sorting ranges but not using it to
match keys. But you can have a range match lookup but only on the
last key and never on two keys. So this would work:

beginkey = ["key21", "key11"]
endkey = ["key21", "key19"]

The current view interface could be augmented to accept queries
and could make them much more powerful then they currently are
and just using the keys for sorting and selecting which values you
want shown which they are designed to do and do really well.

This would be a killer feature and could use the new infrastructure
from Cloudant search.

And don't tell me the Elastic or Lucene interface could do anything
close to this :)

Regards,
Olafur Arason

On Mon, Mar 28, 2011 at 04:31, Andrew Stuart (SuperCoders)
<an...@supercoders.com.au> wrote:
> It would be good to know if full text search is coming as a core feature and
> if yes, approximately when - does anyone know?
>
> Even an approximate timeframe would be good.
>
> thanks
>

Full text search - is it coming? If yes, approx when.

Posted by "Andrew Stuart (SuperCoders)" <an...@supercoders.com.au>.
It would be good to know if full text search is coming as a core  
feature and if yes, approximately when - does anyone know?

Even an approximate timeframe would be good.

thanks

RE: blocking during view generation

Posted by Nils Breunese <N....@vpro.nl>.
http://permalink.gmane.org/gmane.comp.db.couchdb.user/9917 lists a couple of reasons.

Nils.
________________________________________
Van: kowsik [kowsik@gmail.com]
Verzonden: maandag 28 maart 2011 6:26
Aan: Mark Hahn
CC: user@couchdb.apache.org; Mark Hahn; Andrew Stuart (SuperCoders)
Onderwerp: Re: blocking during view generation

(...)

BTW, is there a reason why we couch is still using spidermonkey as the
only option while v8 has gotten faster and faster?

K.
---
http://blitz.io
http://twitter.com/pcapr
------------------------------------------------------------------------
 VPRO   www.vpro.nl
------------------------------------------------------------------------

Re: blocking during view generation

Posted by kowsik <ko...@gmail.com>.
Each time a document has been updated or added, the view server is
invoked with that document and *all* of the views in a given design.
So in some ways, this is a cpu/memory trade off. The 'cpu' being the
cost of deserializing the document within the view server. It's done
once and the various map functions are invoked with the same document.
If your document is fairly large, then this minimizes the cpu cost.
But if you have lots of views in the same design document, then the
key-values emit'd by your views can suck up lots of memory.

At least that's what I make of it. Maybe the couch creators can
comment on this more.

BTW, is there a reason why we couch is still using spidermonkey as the
only option while v8 has gotten faster and faster?

K.
---
http://blitz.io
http://twitter.com/pcapr

On Wed, Mar 23, 2011 at 12:36 AM, Mark Hahn <ma...@boutiquing.com> wrote:
> I am a newbie, but I don't really understand why the views are divided
> into designs in the first place.
>
> I can understand what a design means intuitively, but can someone
> explain what it is good for?  I know it is good for solving my
> original question, but that is fixing a negative.  Is there anything
> positive about a design?
>
> On Tue, Mar 22, 2011 at 10:16 PM, kowsik <ko...@gmail.com> wrote:
>> On Tue, Mar 22, 2011 at 9:31 PM, Mark Hahn <ma...@elleh.com> wrote:
>>>> The recommended way is to add a new design/view (with the change) have
>>> CouchDB take its time to index it and then switch over to this view.
>>>
>>> This solves everything for me.
>>>
>>>> I had the impression from the discussion that the server blocked more widely than just the view being generated
>>>
>>> It does.   All views in a design are blocked when you change the
>>> source code of one view. I only have one design so that was all views
>>> on the server for me.
>>
>> I go back and forth on this one. To me, it's logical (spoken like
>> spock!) to have a design that represents a collection of views that
>> are related. This also means that the view server gets handed *all*
>> the views in the design for each new/updated document, which makes the
>> emit's fast. OTOH, having one view/design means that you can
>> change/version a single view without affecting anything else, but the
>> overall indexing takes longer. On one project I have one
>> view-per-design while on the other it's multiple views-per-design. Go
>> figure. I'm still trying to figure out the optimal setting. :)
>>
>> K.
>> ---
>> http://blitz.io
>> http://twitter.com/pcapr
>>
>>>
>>> On Tue, Mar 22, 2011 at 9:16 PM, Andrew Stuart (SuperCoders)
>>> <an...@supercoders.com.au> wrote:
>>>> Ah well that makes sense.  I had the impression from the discussion that the
>>>> server blocked more widely than just the view being generated which would
>>>> have been very strange.
>>>>
>>>>
>>>> On 23/03/2011, at 3:12 PM, kowsik wrote:
>>>>
>>>> On Tue, Mar 22, 2011 at 8:51 PM, Andrew Stuart (SuperCoders)
>>>> <an...@supercoders.com.au> wrote:
>>>>>
>>>>> Say I put 100,000 documents into a production database.  Is that database
>>>>> usable whilst the views are updated to include those new documents?
>>>>
>>>> The recommended way is to add a new design/view (with the change) have
>>>> CouchDB take its time to index it and then switch over to this view.
>>>> This way the existing views are super fast and nothing gets blocked.
>>>> I'm sure this is in the wiki or the book.
>>>>
>>>> And yes, the rest of the views are happily available without blocking
>>>> while this particular new view is "catching up".
>>>>
>>>>> What exactly is blocked and under what circumstances?
>>>>
>>>> Only the access to the new view which is getting indexed. Nothing else
>>>> blocks.
>>>>
>>>> K.
>>>> ---
>>>> http://blitz.io
>>>> http://twitter.com/pcapr
>>>> --
>>>> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
>>>> filtering.http://www.mailguard.com.au/mg
>>>> Click here to report this message as spam:
>>>> https://login.mailguard.com.au/report/1BXm6do62R/6XbNWx99CirvTmBxZ7aJxt/0
>>>>
>>>
>>
>
>
>
> --
> Mark Hahn
> Website Manager
> mark@boutiquing.com
> 949-229-1012
>

Re: blocking during view generation

Posted by Mark Hahn <ma...@boutiquing.com>.
I am a newbie, but I don't really understand why the views are divided
into designs in the first place.

I can understand what a design means intuitively, but can someone
explain what it is good for?  I know it is good for solving my
original question, but that is fixing a negative.  Is there anything
positive about a design?

On Tue, Mar 22, 2011 at 10:16 PM, kowsik <ko...@gmail.com> wrote:
> On Tue, Mar 22, 2011 at 9:31 PM, Mark Hahn <ma...@elleh.com> wrote:
>>> The recommended way is to add a new design/view (with the change) have
>> CouchDB take its time to index it and then switch over to this view.
>>
>> This solves everything for me.
>>
>>> I had the impression from the discussion that the server blocked more widely than just the view being generated
>>
>> It does.   All views in a design are blocked when you change the
>> source code of one view. I only have one design so that was all views
>> on the server for me.
>
> I go back and forth on this one. To me, it's logical (spoken like
> spock!) to have a design that represents a collection of views that
> are related. This also means that the view server gets handed *all*
> the views in the design for each new/updated document, which makes the
> emit's fast. OTOH, having one view/design means that you can
> change/version a single view without affecting anything else, but the
> overall indexing takes longer. On one project I have one
> view-per-design while on the other it's multiple views-per-design. Go
> figure. I'm still trying to figure out the optimal setting. :)
>
> K.
> ---
> http://blitz.io
> http://twitter.com/pcapr
>
>>
>> On Tue, Mar 22, 2011 at 9:16 PM, Andrew Stuart (SuperCoders)
>> <an...@supercoders.com.au> wrote:
>>> Ah well that makes sense.  I had the impression from the discussion that the
>>> server blocked more widely than just the view being generated which would
>>> have been very strange.
>>>
>>>
>>> On 23/03/2011, at 3:12 PM, kowsik wrote:
>>>
>>> On Tue, Mar 22, 2011 at 8:51 PM, Andrew Stuart (SuperCoders)
>>> <an...@supercoders.com.au> wrote:
>>>>
>>>> Say I put 100,000 documents into a production database.  Is that database
>>>> usable whilst the views are updated to include those new documents?
>>>
>>> The recommended way is to add a new design/view (with the change) have
>>> CouchDB take its time to index it and then switch over to this view.
>>> This way the existing views are super fast and nothing gets blocked.
>>> I'm sure this is in the wiki or the book.
>>>
>>> And yes, the rest of the views are happily available without blocking
>>> while this particular new view is "catching up".
>>>
>>>> What exactly is blocked and under what circumstances?
>>>
>>> Only the access to the new view which is getting indexed. Nothing else
>>> blocks.
>>>
>>> K.
>>> ---
>>> http://blitz.io
>>> http://twitter.com/pcapr
>>> --
>>> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
>>> filtering.http://www.mailguard.com.au/mg
>>> Click here to report this message as spam:
>>> https://login.mailguard.com.au/report/1BXm6do62R/6XbNWx99CirvTmBxZ7aJxt/0
>>>
>>
>



-- 
Mark Hahn
Website Manager
mark@boutiquing.com
949-229-1012

Re: blocking during view generation

Posted by kowsik <ko...@gmail.com>.
On Tue, Mar 22, 2011 at 9:31 PM, Mark Hahn <ma...@elleh.com> wrote:
>> The recommended way is to add a new design/view (with the change) have
> CouchDB take its time to index it and then switch over to this view.
>
> This solves everything for me.
>
>> I had the impression from the discussion that the server blocked more widely than just the view being generated
>
> It does.   All views in a design are blocked when you change the
> source code of one view. I only have one design so that was all views
> on the server for me.

I go back and forth on this one. To me, it's logical (spoken like
spock!) to have a design that represents a collection of views that
are related. This also means that the view server gets handed *all*
the views in the design for each new/updated document, which makes the
emit's fast. OTOH, having one view/design means that you can
change/version a single view without affecting anything else, but the
overall indexing takes longer. On one project I have one
view-per-design while on the other it's multiple views-per-design. Go
figure. I'm still trying to figure out the optimal setting. :)

K.
---
http://blitz.io
http://twitter.com/pcapr

>
> On Tue, Mar 22, 2011 at 9:16 PM, Andrew Stuart (SuperCoders)
> <an...@supercoders.com.au> wrote:
>> Ah well that makes sense.  I had the impression from the discussion that the
>> server blocked more widely than just the view being generated which would
>> have been very strange.
>>
>>
>> On 23/03/2011, at 3:12 PM, kowsik wrote:
>>
>> On Tue, Mar 22, 2011 at 8:51 PM, Andrew Stuart (SuperCoders)
>> <an...@supercoders.com.au> wrote:
>>>
>>> Say I put 100,000 documents into a production database.  Is that database
>>> usable whilst the views are updated to include those new documents?
>>
>> The recommended way is to add a new design/view (with the change) have
>> CouchDB take its time to index it and then switch over to this view.
>> This way the existing views are super fast and nothing gets blocked.
>> I'm sure this is in the wiki or the book.
>>
>> And yes, the rest of the views are happily available without blocking
>> while this particular new view is "catching up".
>>
>>> What exactly is blocked and under what circumstances?
>>
>> Only the access to the new view which is getting indexed. Nothing else
>> blocks.
>>
>> K.
>> ---
>> http://blitz.io
>> http://twitter.com/pcapr
>> --
>> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
>> filtering.http://www.mailguard.com.au/mg
>> Click here to report this message as spam:
>> https://login.mailguard.com.au/report/1BXm6do62R/6XbNWx99CirvTmBxZ7aJxt/0
>>
>

Re: blocking during view generation

Posted by Mark Hahn <ma...@elleh.com>.
> The recommended way is to add a new design/view (with the change) have
CouchDB take its time to index it and then switch over to this view.

This solves everything for me.

> I had the impression from the discussion that the server blocked more widely than just the view being generated

It does.   All views in a design are blocked when you change the
source code of one view. I only have one design so that was all views
on the server for me.


On Tue, Mar 22, 2011 at 9:16 PM, Andrew Stuart (SuperCoders)
<an...@supercoders.com.au> wrote:
> Ah well that makes sense.  I had the impression from the discussion that the
> server blocked more widely than just the view being generated which would
> have been very strange.
>
>
> On 23/03/2011, at 3:12 PM, kowsik wrote:
>
> On Tue, Mar 22, 2011 at 8:51 PM, Andrew Stuart (SuperCoders)
> <an...@supercoders.com.au> wrote:
>>
>> Say I put 100,000 documents into a production database.  Is that database
>> usable whilst the views are updated to include those new documents?
>
> The recommended way is to add a new design/view (with the change) have
> CouchDB take its time to index it and then switch over to this view.
> This way the existing views are super fast and nothing gets blocked.
> I'm sure this is in the wiki or the book.
>
> And yes, the rest of the views are happily available without blocking
> while this particular new view is "catching up".
>
>> What exactly is blocked and under what circumstances?
>
> Only the access to the new view which is getting indexed. Nothing else
> blocks.
>
> K.
> ---
> http://blitz.io
> http://twitter.com/pcapr
> --
> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
> filtering.http://www.mailguard.com.au/mg
> Click here to report this message as spam:
> https://login.mailguard.com.au/report/1BXm6do62R/6XbNWx99CirvTmBxZ7aJxt/0
>

Re: blocking during view generation

Posted by "Andrew Stuart (SuperCoders)" <an...@supercoders.com.au>.
Ah well that makes sense.  I had the impression from the discussion  
that the server blocked more widely than just the view being generated  
which would have been very strange.


On 23/03/2011, at 3:12 PM, kowsik wrote:

On Tue, Mar 22, 2011 at 8:51 PM, Andrew Stuart (SuperCoders)
<an...@supercoders.com.au> wrote:
> Say I put 100,000 documents into a production database.  Is that  
> database
> usable whilst the views are updated to include those new documents?

The recommended way is to add a new design/view (with the change) have
CouchDB take its time to index it and then switch over to this view.
This way the existing views are super fast and nothing gets blocked.
I'm sure this is in the wiki or the book.

And yes, the rest of the views are happily available without blocking
while this particular new view is "catching up".

> What exactly is blocked and under what circumstances?

Only the access to the new view which is getting indexed. Nothing else  
blocks.

K.
---
http://blitz.io
http://twitter.com/pcapr
-- 
Message  protected by MailGuard: e-mail anti-virus, anti-spam and  
content filtering.http://www.mailguard.com.au/mg
Click here to report this message as spam:
https://login.mailguard.com.au/report/1BXm6do62R/6XbNWx99CirvTmBxZ7aJxt/0

Re: blocking during view generation

Posted by kowsik <ko...@gmail.com>.
On Tue, Mar 22, 2011 at 8:51 PM, Andrew Stuart (SuperCoders)
<an...@supercoders.com.au> wrote:
> Say I put 100,000 documents into a production database.  Is that database
> usable whilst the views are updated to include those new documents?

The recommended way is to add a new design/view (with the change) have
CouchDB take its time to index it and then switch over to this view.
This way the existing views are super fast and nothing gets blocked.
I'm sure this is in the wiki or the book.

And yes, the rest of the views are happily available without blocking
while this particular new view is "catching up".

> What exactly is blocked and under what circumstances?

Only the access to the new view which is getting indexed. Nothing else blocks.

K.
---
http://blitz.io
http://twitter.com/pcapr

Re: blocking during view generation

Posted by "Andrew Stuart (SuperCoders)" <an...@supercoders.com.au>.
Say I put 100,000 documents into a production database.  Is that  
database usable whilst the views are updated to include those new  
documents?

What exactly is blocked and under what circumstances?

On 23/03/2011, at 2:42 PM, Mark Hahn wrote:

Just to make it clear, I am only talking about the process that
happens when I change the source code of a view.  I wasn't surprised
to see accesses to that view being blocked, but seeing all access to
all views being blocked was stupefying.  Why rebuild views that
haven't changed?


On Tue, Mar 22, 2011 at 8:34 PM, kowsik <ko...@gmail.com> wrote:
> On Tue, Mar 22, 2011 at 8:28 PM, Andrew Stuart (SuperCoders)
> <an...@supercoders.com.au> wrote:
>> This one seems hard to believe - is it true that CouchDB blocks the  
>> server
>> whilst updating views?
>>
>> View updates can be alot of work for a server.
>>
>> So in reality, queries to the server pause whilst views are updated?
>>
>> This doesn't seem practical for any production usage.
>
> See this blog: http://labs.mudynamics.com/2011/03/10/blitzio-couchdb-in-production/
>
> As long as you have a worker of sorts that watches the _changes feed
> to "catch up" on the index, things work great. Just like any other
> piece of software, you have to have some understanding how CouchDB
> works internally. There is a stale=ok view query which will get you
> the older revisions so your UI doesn't block at all.
>
>> Can someone confirm that this is true, that during production a  
>> server will
>> block whilst views are updated?
>
> See above. :) Yes, couch doesn't do background view indexing, though I
> tend to agree that if there was an Erlang task of sorts that did this
> automatically, end users can relax even more.
>
>> Does anyone else see this as a major issue or am I missing  
>> something? I'm
>> happy to say I have missed the point many times before :-)
>>
>> I'm with Mark - I can't think of any other type of modern server  
>> that stops
>> processing to get something else done.
>
> K.
> ---
> http://blitz.io
> http://twitter.com/pcapr
>



-- 
Mark Hahn
Website Manager
mark@boutiquing.com
949-229-1012
-- 
Message  protected by MailGuard: e-mail anti-virus, anti-spam and  
content filtering.http://www.mailguard.com.au/mg
Click here to report this message as spam:
https://login.mailguard.com.au/report/1BXlCjitcm/T3k8rQus8PI7jCWYQ6SES/1.6

Re: blocking during view generation

Posted by Mark Hahn <ma...@boutiquing.com>.
Just to make it clear, I am only talking about the process that
happens when I change the source code of a view.  I wasn't surprised
to see accesses to that view being blocked, but seeing all access to
all views being blocked was stupefying.  Why rebuild views that
haven't changed?


On Tue, Mar 22, 2011 at 8:34 PM, kowsik <ko...@gmail.com> wrote:
> On Tue, Mar 22, 2011 at 8:28 PM, Andrew Stuart (SuperCoders)
> <an...@supercoders.com.au> wrote:
>> This one seems hard to believe - is it true that CouchDB blocks the server
>> whilst updating views?
>>
>> View updates can be alot of work for a server.
>>
>> So in reality, queries to the server pause whilst views are updated?
>>
>> This doesn't seem practical for any production usage.
>
> See this blog: http://labs.mudynamics.com/2011/03/10/blitzio-couchdb-in-production/
>
> As long as you have a worker of sorts that watches the _changes feed
> to "catch up" on the index, things work great. Just like any other
> piece of software, you have to have some understanding how CouchDB
> works internally. There is a stale=ok view query which will get you
> the older revisions so your UI doesn't block at all.
>
>> Can someone confirm that this is true, that during production a server will
>> block whilst views are updated?
>
> See above. :) Yes, couch doesn't do background view indexing, though I
> tend to agree that if there was an Erlang task of sorts that did this
> automatically, end users can relax even more.
>
>> Does anyone else see this as a major issue or am I missing something? I'm
>> happy to say I have missed the point many times before :-)
>>
>> I'm with Mark - I can't think of any other type of modern server that stops
>> processing to get something else done.
>
> K.
> ---
> http://blitz.io
> http://twitter.com/pcapr
>



-- 
Mark Hahn
Website Manager
mark@boutiquing.com
949-229-1012

Re: blocking during view generation

Posted by kowsik <ko...@gmail.com>.
On Tue, Mar 22, 2011 at 8:28 PM, Andrew Stuart (SuperCoders)
<an...@supercoders.com.au> wrote:
> This one seems hard to believe - is it true that CouchDB blocks the server
> whilst updating views?
>
> View updates can be alot of work for a server.
>
> So in reality, queries to the server pause whilst views are updated?
>
> This doesn't seem practical for any production usage.

See this blog: http://labs.mudynamics.com/2011/03/10/blitzio-couchdb-in-production/

As long as you have a worker of sorts that watches the _changes feed
to "catch up" on the index, things work great. Just like any other
piece of software, you have to have some understanding how CouchDB
works internally. There is a stale=ok view query which will get you
the older revisions so your UI doesn't block at all.

> Can someone confirm that this is true, that during production a server will
> block whilst views are updated?

See above. :) Yes, couch doesn't do background view indexing, though I
tend to agree that if there was an Erlang task of sorts that did this
automatically, end users can relax even more.

> Does anyone else see this as a major issue or am I missing something? I'm
> happy to say I have missed the point many times before :-)
>
> I'm with Mark - I can't think of any other type of modern server that stops
> processing to get something else done.

K.
---
http://blitz.io
http://twitter.com/pcapr

Re: blocking during view generation

Posted by "Andrew Stuart (SuperCoders)" <an...@supercoders.com.au>.
This one seems hard to believe - is it true that CouchDB blocks the  
server whilst updating views?

View updates can be alot of work for a server.

So in reality, queries to the server pause whilst views are updated?

This doesn't seem practical for any production usage.

Can someone confirm that this is true, that during production a server  
will block whilst views are updated?

Does anyone else see this as a major issue or am I missing something?  
I'm happy to say I have missed the point many times before :-)

I'm with Mark - I can't think of any other type of modern server that  
stops processing to get something else done.

thanks

as



On 23/03/2011, at 2:15 PM, Mark Hahn wrote:

Can some please direct me to where this behavior is documented?  I
just went over the technical description again and this is not
mentioned there.  Here is the closest thing I could find and it
doesn't seem to agree with my experience. (From
http://couchdb.apache.org/docs/overview.html).

"CouchDB’s core design is largely optimized around the need for
efficient, incremental creation of views and their indexes."

What is "incremental" about blocking everything for a long time while
views/indexes are created?

I don't mean to be bitchy, and I am sure I will be able to use one of
the many suggestions to ameliorate this problem, but I should not have
to be in production after months of development before discovering
this.


On Tue, Mar 22, 2011 at 7:48 PM, Mark Hahn <ma...@boutiquing.com> wrote:
> Apparently I have a lot of solutions to work with.  I was quite
> worried after first seeing this.
>
> P.S. I studied couch at great lengths for months and I never saw this,
> or at least I didn't see the implications.  It needs to be discussed
> at more length or more specifically in the manual (book?).
>
> P.P.S.  Is the book ever going to be updated?  I'd pay for a hard  
> copy.
>
>
> On Tue, Mar 22, 2011 at 1:15 AM, Gabor Ratky <rg...@rgabostyle.com>  
> wrote:
>> If blocking and blocking only is the problem and you're OK with  
>> stale data, passing ?stale=ok might solve your problem with  
>> blocking. See http://wiki.apache.org/couchdb/HTTP_view_API for more  
>> details.
>>
>> Gabor
>>
>> On Mar 21, 2011, at 6:24 PM, Mark Hahn wrote:
>>
>>> I have noticed that all view queries are blocked for some time  
>>> when I just
>>> change one view.  This seems like it is going to be a serious  
>>> problem when
>>> my db grows.  Is there someway to avoid this in production?  I've  
>>> never seen
>>> mysql or any other db block for any reason.
>>
>>
>
>
>
> --
> Mark Hahn
> Website Manager
> mark@boutiquing.com
> 949-229-1012
>



-- 
Mark Hahn
Website Manager
mark@boutiquing.com
949-229-1012
-- 
Message  protected by MailGuard: e-mail anti-virus, anti-spam and  
content filtering.http://www.mailguard.com.au/mg
Click here to report this message as spam:
https://login.mailguard.com.au/report/1BXl6zGpTb/5bQ3uCBNOauUp9RuSq5d5L/0

Re: blocking during view generation

Posted by Mark Hahn <ma...@boutiquing.com>.
Can some please direct me to where this behavior is documented?  I
just went over the technical description again and this is not
mentioned there.  Here is the closest thing I could find and it
doesn't seem to agree with my experience. (From
http://couchdb.apache.org/docs/overview.html).

"CouchDB’s core design is largely optimized around the need for
efficient, incremental creation of views and their indexes."

What is "incremental" about blocking everything for a long time while
views/indexes are created?

I don't mean to be bitchy, and I am sure I will be able to use one of
the many suggestions to ameliorate this problem, but I should not have
to be in production after months of development before discovering
this.


On Tue, Mar 22, 2011 at 7:48 PM, Mark Hahn <ma...@boutiquing.com> wrote:
> Apparently I have a lot of solutions to work with.  I was quite
> worried after first seeing this.
>
> P.S. I studied couch at great lengths for months and I never saw this,
> or at least I didn't see the implications.  It needs to be discussed
> at more length or more specifically in the manual (book?).
>
> P.P.S.  Is the book ever going to be updated?  I'd pay for a hard copy.
>
>
> On Tue, Mar 22, 2011 at 1:15 AM, Gabor Ratky <rg...@rgabostyle.com> wrote:
>> If blocking and blocking only is the problem and you're OK with stale data, passing ?stale=ok might solve your problem with blocking. See http://wiki.apache.org/couchdb/HTTP_view_API for more details.
>>
>> Gabor
>>
>> On Mar 21, 2011, at 6:24 PM, Mark Hahn wrote:
>>
>>> I have noticed that all view queries are blocked for some time when I just
>>> change one view.  This seems like it is going to be a serious problem when
>>> my db grows.  Is there someway to avoid this in production?  I've never seen
>>> mysql or any other db block for any reason.
>>
>>
>
>
>
> --
> Mark Hahn
> Website Manager
> mark@boutiquing.com
> 949-229-1012
>



-- 
Mark Hahn
Website Manager
mark@boutiquing.com
949-229-1012

Re: blocking during view generation

Posted by Mark Hahn <ma...@boutiquing.com>.
Apparently I have a lot of solutions to work with.  I was quite
worried after first seeing this.

P.S. I studied couch at great lengths for months and I never saw this,
or at least I didn't see the implications.  It needs to be discussed
at more length or more specifically in the manual (book?).

P.P.S.  Is the book ever going to be updated?  I'd pay for a hard copy.


On Tue, Mar 22, 2011 at 1:15 AM, Gabor Ratky <rg...@rgabostyle.com> wrote:
> If blocking and blocking only is the problem and you're OK with stale data, passing ?stale=ok might solve your problem with blocking. See http://wiki.apache.org/couchdb/HTTP_view_API for more details.
>
> Gabor
>
> On Mar 21, 2011, at 6:24 PM, Mark Hahn wrote:
>
>> I have noticed that all view queries are blocked for some time when I just
>> change one view.  This seems like it is going to be a serious problem when
>> my db grows.  Is there someway to avoid this in production?  I've never seen
>> mysql or any other db block for any reason.
>
>



-- 
Mark Hahn
Website Manager
mark@boutiquing.com
949-229-1012

Re: blocking during view generation

Posted by Gabor Ratky <rg...@rgabostyle.com>.
If blocking and blocking only is the problem and you're OK with stale data, passing ?stale=ok might solve your problem with blocking. See http://wiki.apache.org/couchdb/HTTP_view_API for more details.

Gabor

On Mar 21, 2011, at 6:24 PM, Mark Hahn wrote:

> I have noticed that all view queries are blocked for some time when I just
> change one view.  This seems like it is going to be a serious problem when
> my db grows.  Is there someway to avoid this in production?  I've never seen
> mysql or any other db block for any reason.