Bulk inserts

[Luis Miguel Silva <lu...@gmail.com>]

Dear all,

Does anybody on this list have experience with bulk inserts? ...from
Perl's CouchDB::Client module?

I'm doing some benchmarks and found some references to "bulk inserts" here:
http://guide.couchdb.org/draft/performance.html#inserts
"Bulk docs is the API to use when you are importing a corpus of data
using a scripting language. It can be 10 to 100 times faster than
individual bulk updates and is just as easy to work with from most
languages."

Unfortunately, this doc doesn't seem to have examples on how to do
bulk inserts...

Thank you,
Luis

Re: Bulk inserts

[Calle Dybedahl <ca...@init.se>]

On 5 apr 2011, at 02:32, Luis Miguel Silva wrote:

> Dear all,
> 
> Does anybody on this list have experience with bulk inserts? ...from
> Perl's CouchDB::Client module?
> 

Yes. It's really easy, just give a reference to a list of doc objects to the bulkStore method. And it is significantly faster than doing inserts singly, if you have a lot of them. If nothing else, the bulk interface doesn't have to set up and tear down hundreds or thousands of TCP connections.

Also, if  you're using the CouchDB::Client Perl module, you may want to have a look at the https://github.com/cdybedahl/couchdb-client repository at GitHub. I was using the module actively at a customer until the end of 2010, and fixed a few bugs and suboptimal behaviors. I tried to feed them upstream, but there doesn't seem to have been a new release since I sent them.
-- 
Calle Dybedahl
calle@init.se -*- +46 703 - 970 612

Re: Bulk inserts

[Luis Miguel Silva <lu...@gmail.com>]

Cool! It would be great if the module supported it but i don't think it does...
Thank you for pointing me to the API (that other document referenced it but it didn't provide any examples)!

Thanks,
Luis Miguel Silva

On Apr 4, 2011, at 6:52 PM, Sean Copenhaver <se...@gmail.com> wrote:

> I don't have any experience but if the module is giving you a hard time you may want to just script out the request yourself. 
> 
> Anyway the wiki has a simple example using curl:
> 
> http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API#Modify_Multiple_Documents_With_a_Single_Request
> 
> 
> 
> On Apr 4, 2011, at 8:32 PM, Luis Miguel Silva <lu...@gmail.com> wrote:
> 
>> Dear all,
>> 
>> Does anybody on this list have experience with bulk inserts? ...from
>> Perl's CouchDB::Client module?
>> 
>> I'm doing some benchmarks and found some references to "bulk inserts" here:
>> http://guide.couchdb.org/draft/performance.html#inserts
>> "Bulk docs is the API to use when you are importing a corpus of data
>> using a scripting language. It can be 10 to 100 times faster than
>> individual bulk updates and is just as easy to work with from most
>> languages."
>> 
>> Unfortunately, this doc doesn't seem to have examples on how to do
>> bulk inserts...
>> 
>> Thank you,
>> Luis

Re: Bulk inserts

[Sean Copenhaver <se...@gmail.com>]

I don't have any experience but if the module is giving you a hard time you may want to just script out the request yourself. 

Anyway the wiki has a simple example using curl:

http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API#Modify_Multiple_Documents_With_a_Single_Request



On Apr 4, 2011, at 8:32 PM, Luis Miguel Silva <lu...@gmail.com> wrote:

> Dear all,
> 
> Does anybody on this list have experience with bulk inserts? ...from
> Perl's CouchDB::Client module?
> 
> I'm doing some benchmarks and found some references to "bulk inserts" here:
> http://guide.couchdb.org/draft/performance.html#inserts
> "Bulk docs is the API to use when you are importing a corpus of data
> using a scripting language. It can be 10 to 100 times faster than
> individual bulk updates and is just as easy to work with from most
> languages."
> 
> Unfortunately, this doc doesn't seem to have examples on how to do
> bulk inserts...
> 
> Thank you,
> Luis