You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Daniel Gonzalez <go...@gonvaled.com> on 2012/03/13 23:45:39 UTC
Creating a database with lots of documents and updating a view
Hi,
I am creating a database with lots of documents (3 million).
I have a view in the database:
function(doc) {
if (doc.PORTED_NUMBER) emit(doc.PORTED_NUMBER, doc.RECEIVING_OPERATOR);
}
To speed up view creation, I am doing the following (Strategy A)
1. Define view
2. Insert 1000 documents
3. Access the view
4. Goto 2
And I repeat this process until all documents have been inserted.
I have read that this is faster than my previous strategy (Strategy B,
obsolete):
1. Insert all documents
2. Define view
3. Access view
My problem is that, in my current Strategy A, step 3 is taking longer and
longer. Currently I have around 300 thousand documents inserted and view
access is taking around 120s.
The evolution of the delay in view access has been:
2012-03-13 23:01:40,405 - __main__ - INFO - -
BulkSend >> requested= 1000 ok= 1000 errors= 0
2012-03-13 23:03:29,589 - __main__ - INFO - - View
ready, ellapsed 109
2012-03-13 23:03:32,945 - __main__ - INFO - -
BulkSend >> requested= 1000 ok= 1000 errors= 0
2012-03-13 23:05:31,699 - __main__ - INFO - - View
ready, ellapsed 118
2012-03-13 23:05:35,106 - __main__ - INFO - -
BulkSend >> requested= 1000 ok= 1000 errors= 0
2012-03-13 23:07:28,392 - __main__ - INFO - - View
ready, ellapsed 113
2012-03-13 23:07:31,663 - __main__ - INFO - -
BulkSend >> requested= 1000 ok= 1000 errors= 0
2012-03-13 23:09:26,929 - __main__ - INFO - - View
ready, ellapsed 115
2012-03-13 23:09:30,572 - __main__ - INFO - -
BulkSend >> requested= 1000 ok= 1000 errors= 0
2012-03-13 23:11:27,490 - __main__ - INFO - - View
ready, ellapsed 116
2012-03-13 23:11:30,784 - __main__ - INFO - -
BulkSend >> requested= 1000 ok= 1000 errors= 0
2012-03-13 23:13:21,575 - __main__ - INFO - - View
ready, ellapsed 110
2012-03-13 23:13:24,937 - __main__ - INFO - -
BulkSend >> requested= 1000 ok= 1000 errors= 0
2012-03-13 23:15:23,519 - __main__ - INFO - - View
ready, ellapsed 118
2012-03-13 23:15:26,836 - __main__ - INFO - -
BulkSend >> requested= 1000 ok= 1000 errors= 0
2012-03-13 23:17:23,036 - __main__ - INFO - - View
ready, ellapsed 116
2012-03-13 23:17:26,310 - __main__ - INFO - -
BulkSend >> requested= 1000 ok= 1000 errors= 0
It started with around 1s, and it is increasing more or less monotonically.
It is already running since 7 hours ago, and only 300000 documents have
been imported and indexed.
If everything continues like this (I do not know what kind of matematical
function this is following, but for me it seems like an exponential
function), importing the 3 million of documents is going to take forever.
Is there a way to speed this up?
Thanks!
Daniel
Re: Creating a database with lots of documents and updating a view
Posted by Jens Alfke <je...@couchbase.com>.
On Mar 14, 2012, at 4:13 PM, Matthieu Rakotojaona wrote:
But when (and if) you have a reduce function, all the intermediary
rereduce results will have to be updated, right ?
I don’t think so. CouchDB has some clever optimizations where it stores intermediate reduce results in the b-tree nodes. So I think the only ones that need to be recalculated are the ones in b-tree nodes that are affected by the insertion, i.e. usually only a few.
—Jens
Re: Creating a database with lots of documents and updating a view
Posted by Mark Hahn <ma...@hahnca.com>.
> : I just have one increasing
integer being coded using a base64 ordered dictionary:
If you use the id type that has the date in the front you will be ordered
also.
Re: Creating a database with lots of documents and updating a view
Posted by Jason Smith <jh...@iriscouch.com>.
On Fri, Mar 16, 2012 at 9:23 AM, Daniel Gonzalez <go...@gonvaled.com> wrote:
> What I would
> *very* much like to know is: how many documents have already been
> indexed? How can I know this? Is there any request that I can send to
> CouchDB to get this information? Log files?
Yes, you can GET /_active_tasks to see the progress as JSON data. The
Futon user interface also represents this information in the "Status"
tab on the right.
--
Iris Couch
Re: Creating a database with lots of documents and updating a view
Posted by Daniel Gonzalez <go...@gonvaled.com>.
> Number 1 and 2 are true and *should* show logarithmically-increasing
> costs per document (i.e. not too bad).
>
> 500k updates per day is 6 updates per second which CouchDB can easily
> maintain and index.
>
> In that light, would you agree that you are currently enduring the
> long sunk cost of adding and indexing all those documents? (By
> comparison, mkfs.ext3 is painfully slow for large volumes, but we
> don't mind because we know it's a one-off cost.)
>
> Having said all of that, this is one of the problems BigCouch solves.
> It is pretty much API-compatible with CouchDB so you might investigate
> that option too. In my opinion, roughly speaking, you get better
> performance at a cost of a little more operational (sysadmin) work.
>
> --
> Iris Couch
Most of my problems have been reduced by using a smaller (and ordered)
doc_id. Database creation time has decreased, as well as size.
I am not sure how I will handle updates. Using an increasing doc_id
when the database is new is quite easy: I just have one increasing
integer being coded using a base64 ordered dictionary:
"-@0123456789aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ"
Doing the same on updates (modifications, deletions, insertions), will
be more challenging.
Now regarding view creation time: I understand that there is a
one-time overhead indexing the newly created database. I have just
triggered my view in this 22 million database, and will be waiting for
it to finish for some time. I have triggered it like this:
curl -X GET localhost:5984/mydb/_design/tools/_view/by-number | grep -c .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:21:46 --:--:-- 0
And curl is happily waiting for the view to return some results. It is
nice that I can see how much time indexing is taking. What I would
*very* much like to know is: how many documents have already been
indexed? How can I know this? Is there any request that I can send to
CouchDB to get this information? Log files?
Thanks,
Daniel Gonzalez
Re: Creating a database with lots of documents and updating a view
Posted by Jason Smith <jh...@iriscouch.com>.
On Thu, Mar 15, 2012 at 1:21 PM, Daniel Gonzalez <go...@gonvaled.com> wrote:
> Hi CGS,
>
> As you can see from the other threads that I have opened, the problem seems
> to be threefold:
>
> 1. The bigger the database gets, the more size couchdb needs *per
> document*
> 2. The bigger the database gets, the more time it takes to insert new
> documents
> 3. The bigger the database gets, the longer it gets to generate the
> views. The dependency is not lineal.
Number 1 and 2 are true and *should* show logarithmically-increasing
costs per document (i.e. not too bad).
500k updates per day is 6 updates per second which CouchDB can easily
maintain and index.
In that light, would you agree that you are currently enduring the
long sunk cost of adding and indexing all those documents? (By
comparison, mkfs.ext3 is painfully slow for large volumes, but we
don't mind because we know it's a one-off cost.)
Having said all of that, this is one of the problems BigCouch solves.
It is pretty much API-compatible with CouchDB so you might investigate
that option too. In my opinion, roughly speaking, you get better
performance at a cost of a little more operational (sysadmin) work.
--
Iris Couch
Re: Creating a database with lots of documents and updating a view
Posted by Daniel Gonzalez <go...@gonvaled.com>.
Hi CGS,
As you can see from the other threads that I have opened, the problem seems
to be threefold:
1. The bigger the database gets, the more size couchdb needs *per
document*
2. The bigger the database gets, the more time it takes to insert new
documents
3. The bigger the database gets, the longer it gets to generate the
views. The dependency is not lineal.
We are talking here about a database which will be created once (around 20
million documents in all), and will be updated regularly, to the tune of
500 thousand documents per day, mostly additions and edits (very few
deletes, if at all).
How are people structuring big data with couchdb?
Thanks,
Daniel
On Thu, Mar 15, 2012 at 12:13 PM, CGS <cg...@gmail.com> wrote:
> I understand your reluctance and I agree with you, but it seems that the
> bottleneck is at building the view and to that I see no other way to speed
> up the things but only trying to build it in parallel. That's why I gave
> that option. Of course, if you use more computing elements it will build
> faster. But now a question: do you need to build such a view quite often?
> Meaning, do you expect further a high rate of document insertion or just
> once and after that the rate is relatively low? Because if it is one time
> shot, then it may be worth waiting once to have the index built. Otherwise,
> if you expect high frequency insertion, a round robin (or more evolved load
> distributor algorithm) may be a nice choice to give each database the
> required time to make the insertion and update the view. I know that means
> more work for you, but it also builds a more robust application.
>
> This option is just for the sake of discussing different solutions to your
> problem.
>
> Regards,
> CGS
>
>
>
> On Thu, Mar 15, 2012 at 10:48 AM, Daniel Gonzalez <gonvaled@gonvaled.com
> >wrote:
>
> > Hi Cgs,
> >
> > I suppose I could come up with a partition strategy that distributes
> > the load: I could do what you suggest, or I could also use separate
> > hosts. But all this would transfer complexity to my application, and I
> > want to avoid that.
> >
> > Thanks
> > Daniel
> >
> > On Thu, Mar 15, 2012 at 12:41 AM, CGS <cg...@gmail.com> wrote:
> > > Hi,
> > >
> > > Sorry for interfering, Daniel, but do you really need all those
> documents
> > > in a single database? I mean is it mandatory to have them there?
> Because
> > if
> > > not, one can split that 3M database in 10 - 100 smaller databases, each
> > > with its own view, and concatenate the result after that (if you cannot
> > use
> > > pagination, otherwise, instruct your view to take only a certain number
> > of
> > > results per displayed page) using JS or whatever else if you have an
> > > application which needs that result. In this way you can have a certain
> > > level of parallelism which may speed the overall process.
> > >
> > > Just a 2c idea.
> > >
> > > CGS
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Mar 15, 2012 at 12:13 AM, Matthieu Rakotojaona <
> > > matthieu.rakotojaona@gmail.com> wrote:
> > >
> > >> On Wed, Mar 14, 2012 at 10:19 PM, Christopher Sebastian
> > >> <cs...@gmail.com> wrote:
> > >> > I am also new to couchdb, but I don't believe the information from
> > >> Matthieu
> > >> > Rakotojaona is correct. It is my understanding that pretty much
> > >> everything
> > >> > in CouchDB (including views) uses incremental updates. So adding
> new
> > >> > documents to the database does NOT cause all view leaves to be
> > traversed
> > >> --
> > >> > the view is updated incrementally.
> > >> >
> > >> > Is this correct?
> > >> >
> > >> > ~Christopher Sebastian
> > >>
> > >> I might have spoken a little bit too fast. Indeed when you add new
> > >> documents, they are automatically passed (or rather they will be when
> > >> the next query comes) to the map function, which doesn't traverse the
> > >> already indexed db.
> > >>
> > >> But when (and if) you have a reduce function, all the intermediary
> > >> rereduce results will have to be updated, right ?
> > >>
> > >> --
> > >> Matthieu RAKOTOJAONA
> > >>
> >
>
Re: Creating a database with lots of documents and updating a view
Posted by CGS <cg...@gmail.com>.
I understand your reluctance and I agree with you, but it seems that the
bottleneck is at building the view and to that I see no other way to speed
up the things but only trying to build it in parallel. That's why I gave
that option. Of course, if you use more computing elements it will build
faster. But now a question: do you need to build such a view quite often?
Meaning, do you expect further a high rate of document insertion or just
once and after that the rate is relatively low? Because if it is one time
shot, then it may be worth waiting once to have the index built. Otherwise,
if you expect high frequency insertion, a round robin (or more evolved load
distributor algorithm) may be a nice choice to give each database the
required time to make the insertion and update the view. I know that means
more work for you, but it also builds a more robust application.
This option is just for the sake of discussing different solutions to your
problem.
Regards,
CGS
On Thu, Mar 15, 2012 at 10:48 AM, Daniel Gonzalez <go...@gonvaled.com>wrote:
> Hi Cgs,
>
> I suppose I could come up with a partition strategy that distributes
> the load: I could do what you suggest, or I could also use separate
> hosts. But all this would transfer complexity to my application, and I
> want to avoid that.
>
> Thanks
> Daniel
>
> On Thu, Mar 15, 2012 at 12:41 AM, CGS <cg...@gmail.com> wrote:
> > Hi,
> >
> > Sorry for interfering, Daniel, but do you really need all those documents
> > in a single database? I mean is it mandatory to have them there? Because
> if
> > not, one can split that 3M database in 10 - 100 smaller databases, each
> > with its own view, and concatenate the result after that (if you cannot
> use
> > pagination, otherwise, instruct your view to take only a certain number
> of
> > results per displayed page) using JS or whatever else if you have an
> > application which needs that result. In this way you can have a certain
> > level of parallelism which may speed the overall process.
> >
> > Just a 2c idea.
> >
> > CGS
> >
> >
> >
> >
> >
> > On Thu, Mar 15, 2012 at 12:13 AM, Matthieu Rakotojaona <
> > matthieu.rakotojaona@gmail.com> wrote:
> >
> >> On Wed, Mar 14, 2012 at 10:19 PM, Christopher Sebastian
> >> <cs...@gmail.com> wrote:
> >> > I am also new to couchdb, but I don't believe the information from
> >> Matthieu
> >> > Rakotojaona is correct. It is my understanding that pretty much
> >> everything
> >> > in CouchDB (including views) uses incremental updates. So adding new
> >> > documents to the database does NOT cause all view leaves to be
> traversed
> >> --
> >> > the view is updated incrementally.
> >> >
> >> > Is this correct?
> >> >
> >> > ~Christopher Sebastian
> >>
> >> I might have spoken a little bit too fast. Indeed when you add new
> >> documents, they are automatically passed (or rather they will be when
> >> the next query comes) to the map function, which doesn't traverse the
> >> already indexed db.
> >>
> >> But when (and if) you have a reduce function, all the intermediary
> >> rereduce results will have to be updated, right ?
> >>
> >> --
> >> Matthieu RAKOTOJAONA
> >>
>
Re: Creating a database with lots of documents and updating a view
Posted by Daniel Gonzalez <go...@gonvaled.com>.
Hi Cgs,
I suppose I could come up with a partition strategy that distributes
the load: I could do what you suggest, or I could also use separate
hosts. But all this would transfer complexity to my application, and I
want to avoid that.
Thanks
Daniel
On Thu, Mar 15, 2012 at 12:41 AM, CGS <cg...@gmail.com> wrote:
> Hi,
>
> Sorry for interfering, Daniel, but do you really need all those documents
> in a single database? I mean is it mandatory to have them there? Because if
> not, one can split that 3M database in 10 - 100 smaller databases, each
> with its own view, and concatenate the result after that (if you cannot use
> pagination, otherwise, instruct your view to take only a certain number of
> results per displayed page) using JS or whatever else if you have an
> application which needs that result. In this way you can have a certain
> level of parallelism which may speed the overall process.
>
> Just a 2c idea.
>
> CGS
>
>
>
>
>
> On Thu, Mar 15, 2012 at 12:13 AM, Matthieu Rakotojaona <
> matthieu.rakotojaona@gmail.com> wrote:
>
>> On Wed, Mar 14, 2012 at 10:19 PM, Christopher Sebastian
>> <cs...@gmail.com> wrote:
>> > I am also new to couchdb, but I don't believe the information from
>> Matthieu
>> > Rakotojaona is correct. It is my understanding that pretty much
>> everything
>> > in CouchDB (including views) uses incremental updates. So adding new
>> > documents to the database does NOT cause all view leaves to be traversed
>> --
>> > the view is updated incrementally.
>> >
>> > Is this correct?
>> >
>> > ~Christopher Sebastian
>>
>> I might have spoken a little bit too fast. Indeed when you add new
>> documents, they are automatically passed (or rather they will be when
>> the next query comes) to the map function, which doesn't traverse the
>> already indexed db.
>>
>> But when (and if) you have a reduce function, all the intermediary
>> rereduce results will have to be updated, right ?
>>
>> --
>> Matthieu RAKOTOJAONA
>>
Re: Creating a database with lots of documents and updating a view
Posted by CGS <cg...@gmail.com>.
Hi,
Sorry for interfering, Daniel, but do you really need all those documents
in a single database? I mean is it mandatory to have them there? Because if
not, one can split that 3M database in 10 - 100 smaller databases, each
with its own view, and concatenate the result after that (if you cannot use
pagination, otherwise, instruct your view to take only a certain number of
results per displayed page) using JS or whatever else if you have an
application which needs that result. In this way you can have a certain
level of parallelism which may speed the overall process.
Just a 2c idea.
CGS
On Thu, Mar 15, 2012 at 12:13 AM, Matthieu Rakotojaona <
matthieu.rakotojaona@gmail.com> wrote:
> On Wed, Mar 14, 2012 at 10:19 PM, Christopher Sebastian
> <cs...@gmail.com> wrote:
> > I am also new to couchdb, but I don't believe the information from
> Matthieu
> > Rakotojaona is correct. It is my understanding that pretty much
> everything
> > in CouchDB (including views) uses incremental updates. So adding new
> > documents to the database does NOT cause all view leaves to be traversed
> --
> > the view is updated incrementally.
> >
> > Is this correct?
> >
> > ~Christopher Sebastian
>
> I might have spoken a little bit too fast. Indeed when you add new
> documents, they are automatically passed (or rather they will be when
> the next query comes) to the map function, which doesn't traverse the
> already indexed db.
>
> But when (and if) you have a reduce function, all the intermediary
> rereduce results will have to be updated, right ?
>
> --
> Matthieu RAKOTOJAONA
>
Re: Creating a database with lots of documents and updating a view
Posted by Matthieu Rakotojaona <ma...@gmail.com>.
On Wed, Mar 14, 2012 at 10:19 PM, Christopher Sebastian
<cs...@gmail.com> wrote:
> I am also new to couchdb, but I don't believe the information from Matthieu
> Rakotojaona is correct. It is my understanding that pretty much everything
> in CouchDB (including views) uses incremental updates. So adding new
> documents to the database does NOT cause all view leaves to be traversed --
> the view is updated incrementally.
>
> Is this correct?
>
> ~Christopher Sebastian
I might have spoken a little bit too fast. Indeed when you add new
documents, they are automatically passed (or rather they will be when
the next query comes) to the map function, which doesn't traverse the
already indexed db.
But when (and if) you have a reduce function, all the intermediary
rereduce results will have to be updated, right ?
--
Matthieu RAKOTOJAONA
Re: Creating a database with lots of documents and updating a view
Posted by Christopher Sebastian <cs...@gmail.com>.
I am also new to couchdb, but I don't believe the information from Matthieu
Rakotojaona is correct. It is my understanding that pretty much everything
in CouchDB (including views) uses incremental updates. So adding new
documents to the database does NOT cause all view leaves to be traversed --
the view is updated incrementally.
Is this correct?
~Christopher Sebastian
On Wed, Mar 14, 2012 at 2:51 AM, Matthieu Rakotojaona <
matthieu.rakotojaona@gmail.com> wrote:
> You may try to compact the view (and the db) after each insert, but
> this will dramatically increase the overall insert duration.
>
> I don't know very much about couchdb, but when you update the view you
> will have to traverse all your precedently created view leaves.
> Tidying it up will make the whole process faster, but each "tidy up"
> will be longer and longer.
> --
> Matthieu RAKOTOJAONA
>
--
Written using Gmail Motion.
http://gmail.com/motion
Re: Creating a database with lots of documents and updating a view
Posted by Matthieu Rakotojaona <ma...@gmail.com>.
You may try to compact the view (and the db) after each insert, but
this will dramatically increase the overall insert duration.
I don't know very much about couchdb, but when you update the view you
will have to traverse all your precedently created view leaves.
Tidying it up will make the whole process faster, but each "tidy up"
will be longer and longer.
--
Matthieu RAKOTOJAONA
Re: Creating a database with lots of documents and updating a view
Posted by Robert Newson <rn...@apache.org>.
You can see progress in /_active_tasks or in Futon, it should show the
update sequence number. The four year old page you link to might have
been accurate in 2008 but I don't think it's true now. You should
expect view building to slow down in an O(log n) curve as befits a
b+tree. The numbers you see with very few documents are
unrealistically high as everything fits in the disk cache. If you were
to graph it you would see a very high peak that quickly softens into a
curve.
B.
On 13 March 2012 23:08, Daniel Gonzalez <go...@gonvaled.com> wrote:
> Hi,
>
> I have no reduce on the view, and that is my only view.
> I *am* doing bulk inserts (1000 documents), and after each bulk insert, I
> access the view. (my assumption is that this will be faster than accessing
> the view once at the end of inserting the 3 million documents)
>
> I know that I will get here very varying numbers, but: what is the expected
> view indexing time for the view that I posted and for an amount of 3
> million documents?
>
> How can I monitor view creation? (how many documents have been already
> indexed)
>
> I got the idea that "bulk insert + view access + repeat" was faster that
> "full insert + view access" here:
> http://iamseanmurphy.com/2008/09/08/couchdb-view-generation/
>
> Thanks,
> Daniel
>
> On Tue, Mar 13, 2012 at 11:58 PM, Robert Newson <rn...@apache.org> wrote:
>
>> The view build is already batched. In my opinion your strategy A can
>> only ever be slower or the same speed as B.
>>
>> Try inserting the docs using _bulk_docs, it'll go much faster. I'd
>> fill the database up and hit the view at the end for the fastest build
>> time, but I'd still expect it take a while to build the view the first
>> time.
>>
>> Do you have a reduce on the view? Are there other views in the same
>> design document?
>>
>> B.
>>
>> On 13 March 2012 22:45, Daniel Gonzalez <go...@gonvaled.com> wrote:
>> > Hi,
>> >
>> > I am creating a database with lots of documents (3 million).
>> > I have a view in the database:
>> >
>> > function(doc) {
>> > if (doc.PORTED_NUMBER) emit(doc.PORTED_NUMBER,
>> doc.RECEIVING_OPERATOR);
>> > }
>> >
>> > To speed up view creation, I am doing the following (Strategy A)
>> >
>> > 1. Define view
>> > 2. Insert 1000 documents
>> > 3. Access the view
>> > 4. Goto 2
>> >
>> > And I repeat this process until all documents have been inserted.
>> >
>> > I have read that this is faster than my previous strategy (Strategy B,
>> > obsolete):
>> >
>> > 1. Insert all documents
>> > 2. Define view
>> > 3. Access view
>> >
>> > My problem is that, in my current Strategy A, step 3 is taking longer and
>> > longer. Currently I have around 300 thousand documents inserted and view
>> > access is taking around 120s.
>> > The evolution of the delay in view access has been:
>> >
>> > 2012-03-13 23:01:40,405 - __main__ - INFO - -
>> > BulkSend >> requested= 1000 ok= 1000 errors= 0
>> > 2012-03-13 23:03:29,589 - __main__ - INFO - -
>> View
>> > ready, ellapsed 109
>> > 2012-03-13 23:03:32,945 - __main__ - INFO - -
>> > BulkSend >> requested= 1000 ok= 1000 errors= 0
>> > 2012-03-13 23:05:31,699 - __main__ - INFO - -
>> View
>> > ready, ellapsed 118
>> > 2012-03-13 23:05:35,106 - __main__ - INFO - -
>> > BulkSend >> requested= 1000 ok= 1000 errors= 0
>> > 2012-03-13 23:07:28,392 - __main__ - INFO - -
>> View
>> > ready, ellapsed 113
>> > 2012-03-13 23:07:31,663 - __main__ - INFO - -
>> > BulkSend >> requested= 1000 ok= 1000 errors= 0
>> > 2012-03-13 23:09:26,929 - __main__ - INFO - -
>> View
>> > ready, ellapsed 115
>> > 2012-03-13 23:09:30,572 - __main__ - INFO - -
>> > BulkSend >> requested= 1000 ok= 1000 errors= 0
>> > 2012-03-13 23:11:27,490 - __main__ - INFO - -
>> View
>> > ready, ellapsed 116
>> > 2012-03-13 23:11:30,784 - __main__ - INFO - -
>> > BulkSend >> requested= 1000 ok= 1000 errors= 0
>> > 2012-03-13 23:13:21,575 - __main__ - INFO - -
>> View
>> > ready, ellapsed 110
>> > 2012-03-13 23:13:24,937 - __main__ - INFO - -
>> > BulkSend >> requested= 1000 ok= 1000 errors= 0
>> > 2012-03-13 23:15:23,519 - __main__ - INFO - -
>> View
>> > ready, ellapsed 118
>> > 2012-03-13 23:15:26,836 - __main__ - INFO - -
>> > BulkSend >> requested= 1000 ok= 1000 errors= 0
>> > 2012-03-13 23:17:23,036 - __main__ - INFO - -
>> View
>> > ready, ellapsed 116
>> > 2012-03-13 23:17:26,310 - __main__ - INFO - -
>> > BulkSend >> requested= 1000 ok= 1000 errors= 0
>> >
>> > It started with around 1s, and it is increasing more or less
>> monotonically.
>> > It is already running since 7 hours ago, and only 300000 documents have
>> > been imported and indexed.
>> > If everything continues like this (I do not know what kind of matematical
>> > function this is following, but for me it seems like an exponential
>> > function), importing the 3 million of documents is going to take forever.
>> >
>> > Is there a way to speed this up?
>> >
>> > Thanks!
>> > Daniel
>>
Re: Creating a database with lots of documents and updating a view
Posted by Daniel Gonzalez <go...@gonvaled.com>.
Hi,
I have no reduce on the view, and that is my only view.
I *am* doing bulk inserts (1000 documents), and after each bulk insert, I
access the view. (my assumption is that this will be faster than accessing
the view once at the end of inserting the 3 million documents)
I know that I will get here very varying numbers, but: what is the expected
view indexing time for the view that I posted and for an amount of 3
million documents?
How can I monitor view creation? (how many documents have been already
indexed)
I got the idea that "bulk insert + view access + repeat" was faster that
"full insert + view access" here:
http://iamseanmurphy.com/2008/09/08/couchdb-view-generation/
Thanks,
Daniel
On Tue, Mar 13, 2012 at 11:58 PM, Robert Newson <rn...@apache.org> wrote:
> The view build is already batched. In my opinion your strategy A can
> only ever be slower or the same speed as B.
>
> Try inserting the docs using _bulk_docs, it'll go much faster. I'd
> fill the database up and hit the view at the end for the fastest build
> time, but I'd still expect it take a while to build the view the first
> time.
>
> Do you have a reduce on the view? Are there other views in the same
> design document?
>
> B.
>
> On 13 March 2012 22:45, Daniel Gonzalez <go...@gonvaled.com> wrote:
> > Hi,
> >
> > I am creating a database with lots of documents (3 million).
> > I have a view in the database:
> >
> > function(doc) {
> > if (doc.PORTED_NUMBER) emit(doc.PORTED_NUMBER,
> doc.RECEIVING_OPERATOR);
> > }
> >
> > To speed up view creation, I am doing the following (Strategy A)
> >
> > 1. Define view
> > 2. Insert 1000 documents
> > 3. Access the view
> > 4. Goto 2
> >
> > And I repeat this process until all documents have been inserted.
> >
> > I have read that this is faster than my previous strategy (Strategy B,
> > obsolete):
> >
> > 1. Insert all documents
> > 2. Define view
> > 3. Access view
> >
> > My problem is that, in my current Strategy A, step 3 is taking longer and
> > longer. Currently I have around 300 thousand documents inserted and view
> > access is taking around 120s.
> > The evolution of the delay in view access has been:
> >
> > 2012-03-13 23:01:40,405 - __main__ - INFO - -
> > BulkSend >> requested= 1000 ok= 1000 errors= 0
> > 2012-03-13 23:03:29,589 - __main__ - INFO - -
> View
> > ready, ellapsed 109
> > 2012-03-13 23:03:32,945 - __main__ - INFO - -
> > BulkSend >> requested= 1000 ok= 1000 errors= 0
> > 2012-03-13 23:05:31,699 - __main__ - INFO - -
> View
> > ready, ellapsed 118
> > 2012-03-13 23:05:35,106 - __main__ - INFO - -
> > BulkSend >> requested= 1000 ok= 1000 errors= 0
> > 2012-03-13 23:07:28,392 - __main__ - INFO - -
> View
> > ready, ellapsed 113
> > 2012-03-13 23:07:31,663 - __main__ - INFO - -
> > BulkSend >> requested= 1000 ok= 1000 errors= 0
> > 2012-03-13 23:09:26,929 - __main__ - INFO - -
> View
> > ready, ellapsed 115
> > 2012-03-13 23:09:30,572 - __main__ - INFO - -
> > BulkSend >> requested= 1000 ok= 1000 errors= 0
> > 2012-03-13 23:11:27,490 - __main__ - INFO - -
> View
> > ready, ellapsed 116
> > 2012-03-13 23:11:30,784 - __main__ - INFO - -
> > BulkSend >> requested= 1000 ok= 1000 errors= 0
> > 2012-03-13 23:13:21,575 - __main__ - INFO - -
> View
> > ready, ellapsed 110
> > 2012-03-13 23:13:24,937 - __main__ - INFO - -
> > BulkSend >> requested= 1000 ok= 1000 errors= 0
> > 2012-03-13 23:15:23,519 - __main__ - INFO - -
> View
> > ready, ellapsed 118
> > 2012-03-13 23:15:26,836 - __main__ - INFO - -
> > BulkSend >> requested= 1000 ok= 1000 errors= 0
> > 2012-03-13 23:17:23,036 - __main__ - INFO - -
> View
> > ready, ellapsed 116
> > 2012-03-13 23:17:26,310 - __main__ - INFO - -
> > BulkSend >> requested= 1000 ok= 1000 errors= 0
> >
> > It started with around 1s, and it is increasing more or less
> monotonically.
> > It is already running since 7 hours ago, and only 300000 documents have
> > been imported and indexed.
> > If everything continues like this (I do not know what kind of matematical
> > function this is following, but for me it seems like an exponential
> > function), importing the 3 million of documents is going to take forever.
> >
> > Is there a way to speed this up?
> >
> > Thanks!
> > Daniel
>
Re: Creating a database with lots of documents and updating a view
Posted by Robert Newson <rn...@apache.org>.
The view build is already batched. In my opinion your strategy A can
only ever be slower or the same speed as B.
Try inserting the docs using _bulk_docs, it'll go much faster. I'd
fill the database up and hit the view at the end for the fastest build
time, but I'd still expect it take a while to build the view the first
time.
Do you have a reduce on the view? Are there other views in the same
design document?
B.
On 13 March 2012 22:45, Daniel Gonzalez <go...@gonvaled.com> wrote:
> Hi,
>
> I am creating a database with lots of documents (3 million).
> I have a view in the database:
>
> function(doc) {
> if (doc.PORTED_NUMBER) emit(doc.PORTED_NUMBER, doc.RECEIVING_OPERATOR);
> }
>
> To speed up view creation, I am doing the following (Strategy A)
>
> 1. Define view
> 2. Insert 1000 documents
> 3. Access the view
> 4. Goto 2
>
> And I repeat this process until all documents have been inserted.
>
> I have read that this is faster than my previous strategy (Strategy B,
> obsolete):
>
> 1. Insert all documents
> 2. Define view
> 3. Access view
>
> My problem is that, in my current Strategy A, step 3 is taking longer and
> longer. Currently I have around 300 thousand documents inserted and view
> access is taking around 120s.
> The evolution of the delay in view access has been:
>
> 2012-03-13 23:01:40,405 - __main__ - INFO - -
> BulkSend >> requested= 1000 ok= 1000 errors= 0
> 2012-03-13 23:03:29,589 - __main__ - INFO - - View
> ready, ellapsed 109
> 2012-03-13 23:03:32,945 - __main__ - INFO - -
> BulkSend >> requested= 1000 ok= 1000 errors= 0
> 2012-03-13 23:05:31,699 - __main__ - INFO - - View
> ready, ellapsed 118
> 2012-03-13 23:05:35,106 - __main__ - INFO - -
> BulkSend >> requested= 1000 ok= 1000 errors= 0
> 2012-03-13 23:07:28,392 - __main__ - INFO - - View
> ready, ellapsed 113
> 2012-03-13 23:07:31,663 - __main__ - INFO - -
> BulkSend >> requested= 1000 ok= 1000 errors= 0
> 2012-03-13 23:09:26,929 - __main__ - INFO - - View
> ready, ellapsed 115
> 2012-03-13 23:09:30,572 - __main__ - INFO - -
> BulkSend >> requested= 1000 ok= 1000 errors= 0
> 2012-03-13 23:11:27,490 - __main__ - INFO - - View
> ready, ellapsed 116
> 2012-03-13 23:11:30,784 - __main__ - INFO - -
> BulkSend >> requested= 1000 ok= 1000 errors= 0
> 2012-03-13 23:13:21,575 - __main__ - INFO - - View
> ready, ellapsed 110
> 2012-03-13 23:13:24,937 - __main__ - INFO - -
> BulkSend >> requested= 1000 ok= 1000 errors= 0
> 2012-03-13 23:15:23,519 - __main__ - INFO - - View
> ready, ellapsed 118
> 2012-03-13 23:15:26,836 - __main__ - INFO - -
> BulkSend >> requested= 1000 ok= 1000 errors= 0
> 2012-03-13 23:17:23,036 - __main__ - INFO - - View
> ready, ellapsed 116
> 2012-03-13 23:17:26,310 - __main__ - INFO - -
> BulkSend >> requested= 1000 ok= 1000 errors= 0
>
> It started with around 1s, and it is increasing more or less monotonically.
> It is already running since 7 hours ago, and only 300000 documents have
> been imported and indexed.
> If everything continues like this (I do not know what kind of matematical
> function this is following, but for me it seems like an exponential
> function), importing the 3 million of documents is going to take forever.
>
> Is there a way to speed this up?
>
> Thanks!
> Daniel