You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Yogesh Khambia <yk...@gmail.com> on 2011/06/16 00:42:40 UTC
view index generation for single document and bulk documents
Hi all,
Currently, I am doing performance tests with database in CouchDB 1.0.1,
where a script is continuously writing single documents to the CouchDB
database.
I had the issue of the user being penalized for reading the view in CouchDB
database, by updating the view indexes.To improve on the latency for the
view index generation, I wrote a function which generates the view indexes
for each new update made to the database.
The latency for view index creation has been improved. However, I read from
the FAQ on the CouchDB wiki that "The reason not to integrate each doc as it
comes in is that it is horribly inefficient and CouchDB is designed to do
view index updates very fast, so batching is a good idea."
It will be really helpful if somebody can answer me on following:
- If view index generation for each new single document insert is not a
good approach?
- How does the single document insert and bulk document insert affect the
CouchDB view generation, where:
1. a daemon script updates view index for each new document insert.
2. a daemon script updates view index for bulk document
Thank you.
--
Best Regards,
Yogesh Khambia
Postgraduate Design Engineer
Mobile: +31 626 217 381
Email: y.khambia@tue.nl
Re: view index generation for single document and bulk documents
Posted by Yogesh Khambia <yk...@gmail.com>.
Hi,
I apologize for the last post, as I made a mistake:
The correct measurements for writing speed are as follows:
CouchDB-1.0.1 = 20-25 documents/sec
CouchDB-1.1.0 = 5 documents/sec
That's why I am sticking with CouchDB-1.0.1. I will be providing the
platform details, crash logs, where downloaded from by Monday.
Thanks.
On Sat, Jun 18, 2011 at 3:27 AM, Dave Cottlehuber <da...@muse.net.nz> wrote:
> On 18 June 2011 10:34, Yogesh Khambia <yk...@gmail.com> wrote:
> >
> > I am sticking with CouchDB-1.0.1 for my project now as I am using it for
> > quite sometime and it is stable. For Windows it crashed (we know that
> > Windows is not officially supported platform). so for now, I am not using
> > 'stale=update_after'.
>
> We're working on getting it there. Can you give a few more details on the
> crash?
> - what version, downloaded from where?
> - what platform/version?
> - what you were doing at the time of crash?
> - logs would be great from %COUCH%/var/log/couchdb/couch.log
>
> Anything large please post a link rather than content.
>
> A+
> Dave
>
--
Best Regards,
Yogesh Khambia
Postgraduate Design Engineer
Mobile: +31 626 217 381
Email: y.khambia@tue.nl
Re: view index generation for single document and bulk documents
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 18 June 2011 10:34, Yogesh Khambia <yk...@gmail.com> wrote:
>
> I am sticking with CouchDB-1.0.1 for my project now as I am using it for
> quite sometime and it is stable. For Windows it crashed (we know that
> Windows is not officially supported platform). so for now, I am not using
> 'stale=update_after'.
We're working on getting it there. Can you give a few more details on the crash?
- what version, downloaded from where?
- what platform/version?
- what you were doing at the time of crash?
- logs would be great from %COUCH%/var/log/couchdb/couch.log
Anything large please post a link rather than content.
A+
Dave
Re: view index generation for single document and bulk documents
Posted by Yogesh Khambia <yk...@gmail.com>.
Hi Dave,
Thanks a lot for the link to the blogpost.
So far, I am seeing the improvements with the daemon generating the view
index after each new document update to the database.
As earlier while using Chrome's webkit for latency measurments, I could see
PENDING as the status for the HTTP GET request.
However, with the daemon script updating the view index for each new
document, it shows the progress of time, instead of PENDING status.
I am expecting that now the time taken is because of the lookup and
retrieval of the data.
I have done some benchmarking, although more results and the correct
hardware specification I can provide by Monday.
Here what I have measured:
*Virtual Machine Server (2 cores) with other processes(AMQP process , MySQL
process , CouchDB process ) running*
*Ubuntu server(Single core)*
Number of document
360517 documents
1,428,190 documents
Time for generating first view index(no view index in cache)
3 hrs
3.3 hrs
Speed of Map function(as I am not using Reduce function)
34 documents/sec
120 documents/sec
Add 10 more documents
(view index generation time)
294 ms
83.3 ms
My design of the database is as follows:
{
"testcasename" = "MY_TEST",
"testresult" = "pass",
"product" = "HP",
"series" = "10.1.2.200",
"time" = [2011, 6, 18, 1, 14, 50]
}
I have used the CouchDB self generated id. Each write to the database
involves adding such documents to the database.
I am very much interested in knowing the following
- How the does the model of the database affect the CouchDB performance for?
1. A flat structure as shown above, and
2. A nested structure, where the objects are nested in another object
and thereby having deep nesting.
- For better performance, which is better JavaScript MapReduce vs Erlang
MapReduce?
I will be happy to share more details, if needed.
Any suggestions, tips will be of great help to me.
Thanks.
On Thu, Jun 16, 2011 at 10:27 AM, Dave Cottlehuber <da...@muse.net.nz> wrote:
> On 16 June 2011 10:42, Yogesh Khambia <yk...@gmail.com> wrote:
> > Hi all,
> >
> > Currently, I am doing performance tests with database in CouchDB 1.0.1,
> > where a script is continuously writing single documents to the CouchDB
> > database.
> > I had the issue of the user being penalized for reading the view in
> CouchDB
> > database, by updating the view indexes.To improve on the latency for the
> > view index generation, I wrote a function which generates the view
> indexes
> > for each new update made to the database.
> > The latency for view index creation has been improved. However, I read
> from
> > the FAQ on the CouchDB wiki that "The reason not to integrate each doc as
> it
> > comes in is that it is horribly inefficient and CouchDB is designed to do
> > view index updates very fast, so batching is a good idea."
> > It will be really helpful if somebody can answer me on following:
> >
> > - If view index generation for each new single document insert is not a
> > good approach?
>
> Hi Yogesh
>
> I think the wiki is pretty clear "it is horribly inefficient".
>
> The reason is that for both docs and views, separate B-tree DB files
> need to be updated. If you bulk-load docs this allows couch to do the
> b-tree balancing all at once for that bulk-load, rather than each time
> per doc. This is a lot more efficient.
> http://horicky.blogspot.com/2008/10/couchdb-implementation.html covers
> well how this works under the hood.
>
> > - How does the single document insert and bulk document insert affect the
> > CouchDB view generation, where:
> >
> > 1. a daemon script updates view index for each new document insert.
> > 2. a daemon script updates view index for bulk document
>
> IIRC all views in the same ddoc block while couchdb updates it. Other
> than that the same points above apply; however this will vary heavily,
> especially how your doc ids are sequenced and what your views output,
> hardware, OS, etc etc. Perhaps worth doing some benchmarking and/or
> providing more info on your use case.
>
> Somebody more familiar with the code may be able to add more if you need
> it.
>
> A+
> Dave
>
--
Best Regards,
Yogesh Khambia
Postgraduate Design Engineer
Mobile: +31 626 217 381
Email: y.khambia@tue.nl
Re: view index generation for single document and bulk documents
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 16 June 2011 10:42, Yogesh Khambia <yk...@gmail.com> wrote:
> Hi all,
>
> Currently, I am doing performance tests with database in CouchDB 1.0.1,
> where a script is continuously writing single documents to the CouchDB
> database.
> I had the issue of the user being penalized for reading the view in CouchDB
> database, by updating the view indexes.To improve on the latency for the
> view index generation, I wrote a function which generates the view indexes
> for each new update made to the database.
> The latency for view index creation has been improved. However, I read from
> the FAQ on the CouchDB wiki that "The reason not to integrate each doc as it
> comes in is that it is horribly inefficient and CouchDB is designed to do
> view index updates very fast, so batching is a good idea."
> It will be really helpful if somebody can answer me on following:
>
> - If view index generation for each new single document insert is not a
> good approach?
Hi Yogesh
I think the wiki is pretty clear "it is horribly inefficient".
The reason is that for both docs and views, separate B-tree DB files
need to be updated. If you bulk-load docs this allows couch to do the
b-tree balancing all at once for that bulk-load, rather than each time
per doc. This is a lot more efficient.
http://horicky.blogspot.com/2008/10/couchdb-implementation.html covers
well how this works under the hood.
> - How does the single document insert and bulk document insert affect the
> CouchDB view generation, where:
>
> 1. a daemon script updates view index for each new document insert.
> 2. a daemon script updates view index for bulk document
IIRC all views in the same ddoc block while couchdb updates it. Other
than that the same points above apply; however this will vary heavily,
especially how your doc ids are sequenced and what your views output,
hardware, OS, etc etc. Perhaps worth doing some benchmarking and/or
providing more info on your use case.
Somebody more familiar with the code may be able to add more if you need it.
A+
Dave
Re: conflict when deleting a document
Posted by Randall Leeds <ra...@gmail.com>.
On Thu, Jun 16, 2011 at 01:31, Bernhard Schauer
<be...@openforce.com> wrote:
> thx Robert. I thought about this, but I would like to keep the design
> documents.
> And as this runs as part of test-setup perfomance is not a big problem.
>
Why not just re-load the design documents in the setup phase of your test?
> On 2011-06-16 10:09, Robert Newson wrote:
>>
>> Why not just delete the database? It'll be faster.
>>
>> B.
>>
>> On 16 June 2011 08:43, Bernhard Schauer<be...@openforce.com>
>> wrote:
>>>
>>> no, I only have views.
>>>
>>> On 2011-06-16 08:49, Marcello Nuccio wrote:
>>>>
>>>> Do you have validate_doc_update functions in your design documents?
>>>> If yes, does delete works if you remove them?
>>>>
>>>> Marcello
>>>>
>>>> 2011/6/16 Bernhard Schauer<be...@openforce.com>:
>>>>>
>>>>> Hallo all,
>>>>>
>>>>> I have a wired problem when deleting documents. For test purposes I
>>>>> want
>>>>> to
>>>>> delete all documents in a Database, and then insert a well known set of
>>>>> Documents. I do this using Ektorp (Java).
>>>>>
>>>>> Inserting is no problem. But when it comes to deleting the documents I
>>>>> get a
>>>>> conflict.
>>>>> What I actually do is get the ids and revisions of all documents (the
>>>>> built-in all view) and put them in a bulk operation to delete them.
>>>>>
>>>>> Interestingly I can not delete those documents via futon either.
>>>>> When query one of those conflicting documents with revs and revs_info
>>>>> it
>>>>> shows me that there is only one revision.
>>>>>
>>>>> Even more funny, I can not reproduce this in a DB that has no views.
>>>>> There
>>>>> it deleting works. The DB I actually want to work with (and that causes
>>>>> the
>>>>> problem) has several design documents with views.
>>>>>
>>>>> Anyone any idea on that?
>>>>>
>>>>> thx Bernhard
>>>>>
>>>>>
>>>>>
>
Can you provide a paste of the error?
Re: conflict when deleting a document
Posted by Bernhard Schauer <be...@openforce.com>.
thx Robert. I thought about this, but I would like to keep the design
documents.
And as this runs as part of test-setup perfomance is not a big problem.
On 2011-06-16 10:09, Robert Newson wrote:
> Why not just delete the database? It'll be faster.
>
> B.
>
> On 16 June 2011 08:43, Bernhard Schauer<be...@openforce.com> wrote:
>> no, I only have views.
>>
>> On 2011-06-16 08:49, Marcello Nuccio wrote:
>>> Do you have validate_doc_update functions in your design documents?
>>> If yes, does delete works if you remove them?
>>>
>>> Marcello
>>>
>>> 2011/6/16 Bernhard Schauer<be...@openforce.com>:
>>>> Hallo all,
>>>>
>>>> I have a wired problem when deleting documents. For test purposes I want
>>>> to
>>>> delete all documents in a Database, and then insert a well known set of
>>>> Documents. I do this using Ektorp (Java).
>>>>
>>>> Inserting is no problem. But when it comes to deleting the documents I
>>>> get a
>>>> conflict.
>>>> What I actually do is get the ids and revisions of all documents (the
>>>> built-in all view) and put them in a bulk operation to delete them.
>>>>
>>>> Interestingly I can not delete those documents via futon either.
>>>> When query one of those conflicting documents with revs and revs_info it
>>>> shows me that there is only one revision.
>>>>
>>>> Even more funny, I can not reproduce this in a DB that has no views.
>>>> There
>>>> it deleting works. The DB I actually want to work with (and that causes
>>>> the
>>>> problem) has several design documents with views.
>>>>
>>>> Anyone any idea on that?
>>>>
>>>> thx Bernhard
>>>>
>>>>
>>>>
Re: conflict when deleting a document
Posted by Robert Newson <ro...@gmail.com>.
Why not just delete the database? It'll be faster.
B.
On 16 June 2011 08:43, Bernhard Schauer <be...@openforce.com> wrote:
> no, I only have views.
>
> On 2011-06-16 08:49, Marcello Nuccio wrote:
>>
>> Do you have validate_doc_update functions in your design documents?
>> If yes, does delete works if you remove them?
>>
>> Marcello
>>
>> 2011/6/16 Bernhard Schauer<be...@openforce.com>:
>>>
>>> Hallo all,
>>>
>>> I have a wired problem when deleting documents. For test purposes I want
>>> to
>>> delete all documents in a Database, and then insert a well known set of
>>> Documents. I do this using Ektorp (Java).
>>>
>>> Inserting is no problem. But when it comes to deleting the documents I
>>> get a
>>> conflict.
>>> What I actually do is get the ids and revisions of all documents (the
>>> built-in all view) and put them in a bulk operation to delete them.
>>>
>>> Interestingly I can not delete those documents via futon either.
>>> When query one of those conflicting documents with revs and revs_info it
>>> shows me that there is only one revision.
>>>
>>> Even more funny, I can not reproduce this in a DB that has no views.
>>> There
>>> it deleting works. The DB I actually want to work with (and that causes
>>> the
>>> problem) has several design documents with views.
>>>
>>> Anyone any idea on that?
>>>
>>> thx Bernhard
>>>
>>>
>>>
>
Re: conflict when deleting a document
Posted by Bernhard Schauer <be...@openforce.com>.
no, I only have views.
On 2011-06-16 08:49, Marcello Nuccio wrote:
> Do you have validate_doc_update functions in your design documents?
> If yes, does delete works if you remove them?
>
> Marcello
>
> 2011/6/16 Bernhard Schauer<be...@openforce.com>:
>> Hallo all,
>>
>> I have a wired problem when deleting documents. For test purposes I want to
>> delete all documents in a Database, and then insert a well known set of
>> Documents. I do this using Ektorp (Java).
>>
>> Inserting is no problem. But when it comes to deleting the documents I get a
>> conflict.
>> What I actually do is get the ids and revisions of all documents (the
>> built-in all view) and put them in a bulk operation to delete them.
>>
>> Interestingly I can not delete those documents via futon either.
>> When query one of those conflicting documents with revs and revs_info it
>> shows me that there is only one revision.
>>
>> Even more funny, I can not reproduce this in a DB that has no views. There
>> it deleting works. The DB I actually want to work with (and that causes the
>> problem) has several design documents with views.
>>
>> Anyone any idea on that?
>>
>> thx Bernhard
>>
>>
>>
Re: conflict when deleting a document
Posted by Marcello Nuccio <ma...@gmail.com>.
Do you have validate_doc_update functions in your design documents?
If yes, does delete works if you remove them?
Marcello
2011/6/16 Bernhard Schauer <be...@openforce.com>:
> Hallo all,
>
> I have a wired problem when deleting documents. For test purposes I want to
> delete all documents in a Database, and then insert a well known set of
> Documents. I do this using Ektorp (Java).
>
> Inserting is no problem. But when it comes to deleting the documents I get a
> conflict.
> What I actually do is get the ids and revisions of all documents (the
> built-in all view) and put them in a bulk operation to delete them.
>
> Interestingly I can not delete those documents via futon either.
> When query one of those conflicting documents with revs and revs_info it
> shows me that there is only one revision.
>
> Even more funny, I can not reproduce this in a DB that has no views. There
> it deleting works. The DB I actually want to work with (and that causes the
> problem) has several design documents with views.
>
> Anyone any idea on that?
>
> thx Bernhard
>
>
>
conflict when deleting a document
Posted by Bernhard Schauer <be...@openforce.com>.
Hallo all,
I have a wired problem when deleting documents. For test purposes I want
to delete all documents in a Database, and then insert a well known set
of Documents. I do this using Ektorp (Java).
Inserting is no problem. But when it comes to deleting the documents I
get a conflict.
What I actually do is get the ids and revisions of all documents (the
built-in all view) and put them in a bulk operation to delete them.
Interestingly I can not delete those documents via futon either.
When query one of those conflicting documents with revs and revs_info it
shows me that there is only one revision.
Even more funny, I can not reproduce this in a DB that has no views.
There it deleting works. The DB I actually want to work with (and that
causes the problem) has several design documents with views.
Anyone any idea on that?
thx Bernhard
Re: view index generation for single document and bulk documents
Posted by Yogesh Khambia <yk...@gmail.com>.
Hi Marcello,
Thanks for information.
In fact the information about the data being updated to the last record is
required, with the condition that end user should not have a notifiable
waiting ime (not more than 20 sec).
For testing, I did measurements with CouchDB-1.0.1 and CouchDB-1.1.0 for the
clean installation database on Ubuntu machine(single core) with only CouchDB
running on the machine. A writer from my another machine writes documents to
the database.
I observed a significant difference between the writing speed as follows:
CouchDB-1.0.1
CouchDB-1.1.0
Writing speed
5 Docs/sec
20-25 docs/sec
I am sticking with CouchDB-1.0.1 for my project now as I am using it for
quite sometime and it is stable. For Windows it crashed (we know that
Windows is not officially supported platform). so for now, I am not using
'stale=update_after'.
On Thu, Jun 16, 2011 at 9:36 AM, Marcello Nuccio
<ma...@gmail.com>wrote:
> 2011/6/16 Yogesh Khambia <yk...@gmail.com>:
> > Hi Marcello,
> >
> > Thanks for the tip.
> > I will be testing with CouchDB-1.1 soon.
> > One point I want to add:
> > In my case as the database is being updated quite often, the user is
> > interested in the view on the latest data available at any time. Querying
> > the view with 'stale=update_after' parameter will result in stalling the
> > view and then rebuilding the view index, therefore, it will not give the
> > view on the latest data.
>
> It will give the view on the latest indexed data and it will ensure
> that the index is kept updated. It's a trade-off. What is the most
> important for your users: fast response time or information updated to
> the last second?
>
> Marcello
>
> >
> > On Thu, Jun 16, 2011 at 7:40 AM, Marcello Nuccio
> > <ma...@gmail.com>wrote:
> >
> >> Hi Yogesh,
> >> I cannot answer your questions, but I think you should repeat your
> >> tests with CouchDB-1.1 and querying the view with `stale=update_after`
> >> parameter.
> >>
> >> Read
> >>
> http://docs.couchbase.org/couchdb-release-1.1/index.html#coudhdb-release-1.1-updateafter
> >> for more information.
> >>
> >> Bye,
> >> Marcello
> >>
> >> 2011/6/16 Yogesh Khambia <yk...@gmail.com>:
> >> > Hi all,
> >> >
> >> > Currently, I am doing performance tests with database in CouchDB
> 1.0.1,
> >> > where a script is continuously writing single documents to the CouchDB
> >> > database.
> >> > I had the issue of the user being penalized for reading the view in
> >> CouchDB
> >> > database, by updating the view indexes.To improve on the latency for
> the
> >> > view index generation, I wrote a function which generates the view
> >> indexes
> >> > for each new update made to the database.
> >> > The latency for view index creation has been improved. However, I read
> >> from
> >> > the FAQ on the CouchDB wiki that "The reason not to integrate each doc
> as
> >> it
> >> > comes in is that it is horribly inefficient and CouchDB is designed to
> do
> >> > view index updates very fast, so batching is a good idea."
> >> > It will be really helpful if somebody can answer me on following:
> >> >
> >> > - If view index generation for each new single document insert is not
> a
> >> > good approach?
> >> >
> >> > - How does the single document insert and bulk document insert affect
> the
> >> > CouchDB view generation, where:
> >> >
> >> > 1. a daemon script updates view index for each new document insert.
> >> > 2. a daemon script updates view index for bulk document
> >> >
> >> > Thank you.
> >> >
> >> >
> >> > --
> >> > Best Regards,
> >> > Yogesh Khambia
> >> > Postgraduate Design Engineer
> >> > Mobile: +31 626 217 381
> >> > Email: y.khambia@tue.nl
> >> >
> >>
> >
> >
> >
> > --
> > Best Regards,
> > Yogesh Khambia
> > Postgraduate Design Engineer
> > Mobile: +31 626 217 381
> > Email: y.khambia@tue.nl
> >
>
--
Best Regards,
Yogesh Khambia
Postgraduate Design Engineer
Mobile: +31 626 217 381
Email: y.khambia@tue.nl
Re: view index generation for single document and bulk documents
Posted by Marcello Nuccio <ma...@gmail.com>.
2011/6/16 Yogesh Khambia <yk...@gmail.com>:
> Hi Marcello,
>
> Thanks for the tip.
> I will be testing with CouchDB-1.1 soon.
> One point I want to add:
> In my case as the database is being updated quite often, the user is
> interested in the view on the latest data available at any time. Querying
> the view with 'stale=update_after' parameter will result in stalling the
> view and then rebuilding the view index, therefore, it will not give the
> view on the latest data.
It will give the view on the latest indexed data and it will ensure
that the index is kept updated. It's a trade-off. What is the most
important for your users: fast response time or information updated to
the last second?
Marcello
>
> On Thu, Jun 16, 2011 at 7:40 AM, Marcello Nuccio
> <ma...@gmail.com>wrote:
>
>> Hi Yogesh,
>> I cannot answer your questions, but I think you should repeat your
>> tests with CouchDB-1.1 and querying the view with `stale=update_after`
>> parameter.
>>
>> Read
>> http://docs.couchbase.org/couchdb-release-1.1/index.html#coudhdb-release-1.1-updateafter
>> for more information.
>>
>> Bye,
>> Marcello
>>
>> 2011/6/16 Yogesh Khambia <yk...@gmail.com>:
>> > Hi all,
>> >
>> > Currently, I am doing performance tests with database in CouchDB 1.0.1,
>> > where a script is continuously writing single documents to the CouchDB
>> > database.
>> > I had the issue of the user being penalized for reading the view in
>> CouchDB
>> > database, by updating the view indexes.To improve on the latency for the
>> > view index generation, I wrote a function which generates the view
>> indexes
>> > for each new update made to the database.
>> > The latency for view index creation has been improved. However, I read
>> from
>> > the FAQ on the CouchDB wiki that "The reason not to integrate each doc as
>> it
>> > comes in is that it is horribly inefficient and CouchDB is designed to do
>> > view index updates very fast, so batching is a good idea."
>> > It will be really helpful if somebody can answer me on following:
>> >
>> > - If view index generation for each new single document insert is not a
>> > good approach?
>> >
>> > - How does the single document insert and bulk document insert affect the
>> > CouchDB view generation, where:
>> >
>> > 1. a daemon script updates view index for each new document insert.
>> > 2. a daemon script updates view index for bulk document
>> >
>> > Thank you.
>> >
>> >
>> > --
>> > Best Regards,
>> > Yogesh Khambia
>> > Postgraduate Design Engineer
>> > Mobile: +31 626 217 381
>> > Email: y.khambia@tue.nl
>> >
>>
>
>
>
> --
> Best Regards,
> Yogesh Khambia
> Postgraduate Design Engineer
> Mobile: +31 626 217 381
> Email: y.khambia@tue.nl
>
Re: view index generation for single document and bulk documents
Posted by Yogesh Khambia <yk...@gmail.com>.
Hi Marcello,
Thanks for the tip.
I will be testing with CouchDB-1.1 soon.
One point I want to add:
In my case as the database is being updated quite often, the user is
interested in the view on the latest data available at any time. Querying
the view with 'stale=update_after' parameter will result in stalling the
view and then rebuilding the view index, therefore, it will not give the
view on the latest data.
On Thu, Jun 16, 2011 at 7:40 AM, Marcello Nuccio
<ma...@gmail.com>wrote:
> Hi Yogesh,
> I cannot answer your questions, but I think you should repeat your
> tests with CouchDB-1.1 and querying the view with `stale=update_after`
> parameter.
>
> Read
> http://docs.couchbase.org/couchdb-release-1.1/index.html#coudhdb-release-1.1-updateafter
> for more information.
>
> Bye,
> Marcello
>
> 2011/6/16 Yogesh Khambia <yk...@gmail.com>:
> > Hi all,
> >
> > Currently, I am doing performance tests with database in CouchDB 1.0.1,
> > where a script is continuously writing single documents to the CouchDB
> > database.
> > I had the issue of the user being penalized for reading the view in
> CouchDB
> > database, by updating the view indexes.To improve on the latency for the
> > view index generation, I wrote a function which generates the view
> indexes
> > for each new update made to the database.
> > The latency for view index creation has been improved. However, I read
> from
> > the FAQ on the CouchDB wiki that "The reason not to integrate each doc as
> it
> > comes in is that it is horribly inefficient and CouchDB is designed to do
> > view index updates very fast, so batching is a good idea."
> > It will be really helpful if somebody can answer me on following:
> >
> > - If view index generation for each new single document insert is not a
> > good approach?
> >
> > - How does the single document insert and bulk document insert affect the
> > CouchDB view generation, where:
> >
> > 1. a daemon script updates view index for each new document insert.
> > 2. a daemon script updates view index for bulk document
> >
> > Thank you.
> >
> >
> > --
> > Best Regards,
> > Yogesh Khambia
> > Postgraduate Design Engineer
> > Mobile: +31 626 217 381
> > Email: y.khambia@tue.nl
> >
>
--
Best Regards,
Yogesh Khambia
Postgraduate Design Engineer
Mobile: +31 626 217 381
Email: y.khambia@tue.nl
Re: view index generation for single document and bulk documents
Posted by Marcello Nuccio <ma...@gmail.com>.
Hi Yogesh,
I cannot answer your questions, but I think you should repeat your
tests with CouchDB-1.1 and querying the view with `stale=update_after`
parameter.
Read http://docs.couchbase.org/couchdb-release-1.1/index.html#coudhdb-release-1.1-updateafter
for more information.
Bye,
Marcello
2011/6/16 Yogesh Khambia <yk...@gmail.com>:
> Hi all,
>
> Currently, I am doing performance tests with database in CouchDB 1.0.1,
> where a script is continuously writing single documents to the CouchDB
> database.
> I had the issue of the user being penalized for reading the view in CouchDB
> database, by updating the view indexes.To improve on the latency for the
> view index generation, I wrote a function which generates the view indexes
> for each new update made to the database.
> The latency for view index creation has been improved. However, I read from
> the FAQ on the CouchDB wiki that "The reason not to integrate each doc as it
> comes in is that it is horribly inefficient and CouchDB is designed to do
> view index updates very fast, so batching is a good idea."
> It will be really helpful if somebody can answer me on following:
>
> - If view index generation for each new single document insert is not a
> good approach?
>
> - How does the single document insert and bulk document insert affect the
> CouchDB view generation, where:
>
> 1. a daemon script updates view index for each new document insert.
> 2. a daemon script updates view index for bulk document
>
> Thank you.
>
>
> --
> Best Regards,
> Yogesh Khambia
> Postgraduate Design Engineer
> Mobile: +31 626 217 381
> Email: y.khambia@tue.nl
>