You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Mage <ma...@mage.hu> on 2011/04/29 20:34:44 UTC
triggering all (necessary) view update
Hello,
there should be a way to trigger an update on all views in a database.
I imagine a master-slave setup for a website. There can be one or
several slaves.
The code for the website gets updated. So do the views. (For example,
couchrest_model updates the view functions on every little code change).
Putting the new design documents to the master causes a big read-delay
on all the slaves while they are rebuilding the b-trees from scratch.
I would put the new design documents to a slave, let it rebuild the
views (the ones which were changed) then replicate the design document
changes to the master. This way the clients can read the other slaves or
the master while the updated slave rebuilds. After that the updated
slave can serve the clients while the others are rebuilding b-tress. No
delay.
I suppose that I can write a short script that finds all the views in a
database and asks one document from each, however it is not as elegant
as having a request which does the same.
Mage
Re: triggering all (necessary) view update
Posted by "prathviraj.s.m meenkeri" <pr...@gmail.com>.
Hi,
I have installed couchdb its working fine,
i have problem with the database connectivity with the frontend design,
want to access data from the couchdb using ruby,
can any body help me working with this ,
Thanks
Re: triggering all (necessary) view update
Posted by Robert Newson <ro...@gmail.com>.
the stale=ok feature is for requesting view results without blocking
to update, not related to compaction. I was answering Jim's question.
B.
On 30 April 2011 17:20, Mage <ma...@mage.hu> wrote:
> Hello Robert,
>
> I am not sure I see right, is this related to compaction? I see no
> downtime right after I start compaction. CouchDB responds fast when it
> is compacting (at least in my test case).
>
> The long wait happens when I modify a view and I query it first time.
> This is what I would trigger directly for all the views after updating
> one slave node's views. So users of the website wouldn't have downtime.
>
> Do you mean that compaction scheduler will handle view rebuild too? Now
> compation doesn't build views.
>
> Mage
>
>
> On 04/30/2011 04:32 PM, Robert Newson wrote:
>> Magie,
>>
>> I didn't read your post as offensive, don't worry. The problem is
>> mostly that compaction is an intensive process so scheduling it
>> automatically is tricky; the right schedule will vary by use case. I
>> can see the scheduler becoming very complicated in order to
>> accommodate as many cases as possible.
>>
>> I think it would be better done externally, even if that's in the form
>> of some kind of official couchdb plugin or addon. Once the next couple
>> of releases are behind us (1.0.3 and 1.1.0), we'll be reorganizing the
>> source tree to make things more manageable (and separated). That would
>> be a good time to consider a compaction scheduler add-on.
>>
>> B.
>>
>> On 30 April 2011 13:14, Mage <ma...@mage.hu> wrote:
>>> On 04/29/2011 10:36 PM, Robert Newson wrote:
>>>> "I suppose that I can write a short script that finds all the views in a
>>>> database and asks one document from each, however it is not as elegant
>>>> as having a request which does the same."
>>>>
>>>> You only find it more elegant because you won't have to look at the code. :)
>>> I didn't mean to be offensive.
>>>
>>> I don't know erlang yet so I have no idea how much work would be
>>> required to implement this in CouchDB. There is a lot of things to
>>> rethink after the long years spent with sql.
>>>
>>> Anyway, I'will script this.
>>>
>>> Mage
>>>
>>>
>>>
>
>
Re: triggering all (necessary) view update
Posted by Mage <ma...@mage.hu>.
Hello Robert,
I am not sure I see right, is this related to compaction? I see no
downtime right after I start compaction. CouchDB responds fast when it
is compacting (at least in my test case).
The long wait happens when I modify a view and I query it first time.
This is what I would trigger directly for all the views after updating
one slave node's views. So users of the website wouldn't have downtime.
Do you mean that compaction scheduler will handle view rebuild too? Now
compation doesn't build views.
Mage
On 04/30/2011 04:32 PM, Robert Newson wrote:
> Magie,
>
> I didn't read your post as offensive, don't worry. The problem is
> mostly that compaction is an intensive process so scheduling it
> automatically is tricky; the right schedule will vary by use case. I
> can see the scheduler becoming very complicated in order to
> accommodate as many cases as possible.
>
> I think it would be better done externally, even if that's in the form
> of some kind of official couchdb plugin or addon. Once the next couple
> of releases are behind us (1.0.3 and 1.1.0), we'll be reorganizing the
> source tree to make things more manageable (and separated). That would
> be a good time to consider a compaction scheduler add-on.
>
> B.
>
> On 30 April 2011 13:14, Mage <ma...@mage.hu> wrote:
>> On 04/29/2011 10:36 PM, Robert Newson wrote:
>>> "I suppose that I can write a short script that finds all the views in a
>>> database and asks one document from each, however it is not as elegant
>>> as having a request which does the same."
>>>
>>> You only find it more elegant because you won't have to look at the code. :)
>> I didn't mean to be offensive.
>>
>> I don't know erlang yet so I have no idea how much work would be
>> required to implement this in CouchDB. There is a lot of things to
>> rethink after the long years spent with sql.
>>
>> Anyway, I'will script this.
>>
>> Mage
>>
>>
>>
Re: triggering all (necessary) view update
Posted by Robert Newson <ro...@gmail.com>.
add ?stale=ok to prevent blocking for the updates to be applied.
http://wiki.apache.org/couchdb/HTTP_view_API#Querying_Options
B.
On 30 April 2011 17:08, Jim Klo <ji...@sri.com> wrote:
> I've been watching this thread, and thanks for bit of roadmap update - I'd love to see a better plugin architecture.
>
> Back to updating views, along the same lines of triggering updates, is there a way to make the view updates non-blocking? I can handle scripting the updates and eventual consistency - but I can't handle the 2 - 5 minutes of blocking it takes for each view to update after a few thousand docs are added.
>
> - Jim
>
> On Apr 30, 2011, at 7:32 AM, Robert Newson <ro...@gmail.com> wrote:
>
>> Magie,
>>
>> I didn't read your post as offensive, don't worry. The problem is
>> mostly that compaction is an intensive process so scheduling it
>> automatically is tricky; the right schedule will vary by use case. I
>> can see the scheduler becoming very complicated in order to
>> accommodate as many cases as possible.
>>
>> I think it would be better done externally, even if that's in the form
>> of some kind of official couchdb plugin or addon. Once the next couple
>> of releases are behind us (1.0.3 and 1.1.0), we'll be reorganizing the
>> source tree to make things more manageable (and separated). That would
>> be a good time to consider a compaction scheduler add-on.
>>
>> B.
>>
>> On 30 April 2011 13:14, Mage <ma...@mage.hu> wrote:
>>> On 04/29/2011 10:36 PM, Robert Newson wrote:
>>>> "I suppose that I can write a short script that finds all the views in a
>>>> database and asks one document from each, however it is not as elegant
>>>> as having a request which does the same."
>>>>
>>>> You only find it more elegant because you won't have to look at the code. :)
>>> I didn't mean to be offensive.
>>>
>>> I don't know erlang yet so I have no idea how much work would be
>>> required to implement this in CouchDB. There is a lot of things to
>>> rethink after the long years spent with sql.
>>>
>>> Anyway, I'will script this.
>>>
>>> Mage
>>>
>>>
>>>
>
Re: triggering all (necessary) view update
Posted by Sean Copenhaver <se...@gmail.com>.
I'm coming in late on this thread but if you want to make sure you are not
blocked from querying views while they update I would suggest using the
approach outlined in the wiki to write a temporary design document, then
once the views are finished you write the real design document. View indexes
are unique to design document contents, so couchdb will share the index.
http://wiki.apache.org/couchdb/How_to_deploy_view_changes_in_a_live_environment
That paired with an external process monitoring _changes API for design
document changes and I think you can basically fully automate this.
On Sat, Apr 30, 2011 at 12:30 PM, Mage <ma...@mage.hu> wrote:
> Hello Jim,
>
> this seems to be related to triggering the view-update for the whole
> database. I think it should help you too. Even with external scripting.
>
> Mage
>
> On 04/30/2011 06:08 PM, Jim Klo wrote:
> > I can handle scripting the updates and eventual consistency - but I
> can't handle the 2 - 5 minutes of blocking it takes for each view to update
> after a few thousand docs are added.
> >
> > - Jim
>
>
--
“The limits of language are the limits of one's world. “ -Ludwig von
Wittgenstein
Re: triggering all (necessary) view update
Posted by Mage <ma...@mage.hu>.
Hello Jim,
this seems to be related to triggering the view-update for the whole
database. I think it should help you too. Even with external scripting.
Mage
On 04/30/2011 06:08 PM, Jim Klo wrote:
> I can handle scripting the updates and eventual consistency - but I can't handle the 2 - 5 minutes of blocking it takes for each view to update after a few thousand docs are added.
>
> - Jim
Re: triggering all (necessary) view update
Posted by Jim Klo <ji...@sri.com>.
I've been watching this thread, and thanks for bit of roadmap update - I'd love to see a better plugin architecture.
Back to updating views, along the same lines of triggering updates, is there a way to make the view updates non-blocking? I can handle scripting the updates and eventual consistency - but I can't handle the 2 - 5 minutes of blocking it takes for each view to update after a few thousand docs are added.
- Jim
On Apr 30, 2011, at 7:32 AM, Robert Newson <ro...@gmail.com> wrote:
> Magie,
>
> I didn't read your post as offensive, don't worry. The problem is
> mostly that compaction is an intensive process so scheduling it
> automatically is tricky; the right schedule will vary by use case. I
> can see the scheduler becoming very complicated in order to
> accommodate as many cases as possible.
>
> I think it would be better done externally, even if that's in the form
> of some kind of official couchdb plugin or addon. Once the next couple
> of releases are behind us (1.0.3 and 1.1.0), we'll be reorganizing the
> source tree to make things more manageable (and separated). That would
> be a good time to consider a compaction scheduler add-on.
>
> B.
>
> On 30 April 2011 13:14, Mage <ma...@mage.hu> wrote:
>> On 04/29/2011 10:36 PM, Robert Newson wrote:
>>> "I suppose that I can write a short script that finds all the views in a
>>> database and asks one document from each, however it is not as elegant
>>> as having a request which does the same."
>>>
>>> You only find it more elegant because you won't have to look at the code. :)
>> I didn't mean to be offensive.
>>
>> I don't know erlang yet so I have no idea how much work would be
>> required to implement this in CouchDB. There is a lot of things to
>> rethink after the long years spent with sql.
>>
>> Anyway, I'will script this.
>>
>> Mage
>>
>>
>>
Re: triggering all (necessary) view update
Posted by Robert Newson <ro...@gmail.com>.
Magie,
I didn't read your post as offensive, don't worry. The problem is
mostly that compaction is an intensive process so scheduling it
automatically is tricky; the right schedule will vary by use case. I
can see the scheduler becoming very complicated in order to
accommodate as many cases as possible.
I think it would be better done externally, even if that's in the form
of some kind of official couchdb plugin or addon. Once the next couple
of releases are behind us (1.0.3 and 1.1.0), we'll be reorganizing the
source tree to make things more manageable (and separated). That would
be a good time to consider a compaction scheduler add-on.
B.
On 30 April 2011 13:14, Mage <ma...@mage.hu> wrote:
> On 04/29/2011 10:36 PM, Robert Newson wrote:
>> "I suppose that I can write a short script that finds all the views in a
>> database and asks one document from each, however it is not as elegant
>> as having a request which does the same."
>>
>> You only find it more elegant because you won't have to look at the code. :)
> I didn't mean to be offensive.
>
> I don't know erlang yet so I have no idea how much work would be
> required to implement this in CouchDB. There is a lot of things to
> rethink after the long years spent with sql.
>
> Anyway, I'will script this.
>
> Mage
>
>
>
Re: triggering all (necessary) view update
Posted by Mage <ma...@mage.hu>.
On 04/29/2011 10:36 PM, Robert Newson wrote:
> "I suppose that I can write a short script that finds all the views in a
> database and asks one document from each, however it is not as elegant
> as having a request which does the same."
>
> You only find it more elegant because you won't have to look at the code. :)
I didn't mean to be offensive.
I don't know erlang yet so I have no idea how much work would be
required to implement this in CouchDB. There is a lot of things to
rethink after the long years spent with sql.
Anyway, I'will script this.
Mage
Re: triggering all (necessary) view update
Posted by Robert Newson <ro...@gmail.com>.
"I suppose that I can write a short script that finds all the views in a
database and asks one document from each, however it is not as elegant
as having a request which does the same."
You only find it more elegant because you won't have to look at the code. :)
This is trivially scriptable from the outside, I see no good reason to add this.
B.
On 29 April 2011 20:31, Zachary Zolton <za...@gmail.com> wrote:
> Since CouchDB updates all views in the same design doc at the same
> time, you only have to query a row from one view per design doc to
> update the view indexes.
>
> On Fri, Apr 29, 2011 at 1:34 PM, Mage <ma...@mage.hu> wrote:
>>
>> Hello,
>>
>> there should be a way to trigger an update on all views in a database.
>>
>> I imagine a master-slave setup for a website. There can be one or
>> several slaves.
>>
>> The code for the website gets updated. So do the views. (For example,
>> couchrest_model updates the view functions on every little code change).
>>
>> Putting the new design documents to the master causes a big read-delay
>> on all the slaves while they are rebuilding the b-trees from scratch.
>>
>> I would put the new design documents to a slave, let it rebuild the
>> views (the ones which were changed) then replicate the design document
>> changes to the master. This way the clients can read the other slaves or
>> the master while the updated slave rebuilds. After that the updated
>> slave can serve the clients while the others are rebuilding b-tress. No
>> delay.
>>
>> I suppose that I can write a short script that finds all the views in a
>> database and asks one document from each, however it is not as elegant
>> as having a request which does the same.
>>
>> Mage
>
Re: triggering all (necessary) view update
Posted by Zachary Zolton <za...@gmail.com>.
Since CouchDB updates all views in the same design doc at the same
time, you only have to query a row from one view per design doc to
update the view indexes.
On Fri, Apr 29, 2011 at 1:34 PM, Mage <ma...@mage.hu> wrote:
>
> Hello,
>
> there should be a way to trigger an update on all views in a database.
>
> I imagine a master-slave setup for a website. There can be one or
> several slaves.
>
> The code for the website gets updated. So do the views. (For example,
> couchrest_model updates the view functions on every little code change).
>
> Putting the new design documents to the master causes a big read-delay
> on all the slaves while they are rebuilding the b-trees from scratch.
>
> I would put the new design documents to a slave, let it rebuild the
> views (the ones which were changed) then replicate the design document
> changes to the master. This way the clients can read the other slaves or
> the master while the updated slave rebuilds. After that the updated
> slave can serve the clients while the others are rebuilding b-tress. No
> delay.
>
> I suppose that I can write a short script that finds all the views in a
> database and asks one document from each, however it is not as elegant
> as having a request which does the same.
>
> Mage