You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Tom Nichols <tm...@gmail.com> on 2009/05/04 17:40:53 UTC

Insert performance

Hi, I have some questions about insert performance.

I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
attached a huge EBS volume to it and mounted it where CouchDB's data
files are stored.  I fired up about ruby scripts running inserts and
after a weekend I only have about 30GB/ 12M rows of data...  Which
seems small.  'top' tells me that my CPU is only about 30% utilized.

Any idea what I might be doing wrong?  I pretty much just followed
these instructions:
http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2

My ruby script looks like this:
#!/usr/bin/env ruby
#Script to load random data into CouchDB

require 'rubygems'
require 'couchrest'

db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
puts "Created database: #{ARGV[0]}"

max = 9999999999999999
while 1
        puts 'loading...'
        for val in 0..max
                db.save_doc({ :key => val, 'val one' => "val ${val}",
'val2' => "#{ARGV[1]} #{val}" })
        end
end


Thanks in advance...

Re: Insert performance

Posted by Chris Anderson <jc...@apache.org>.
On Mon, May 4, 2009 at 10:20 AM, Tom Nichols <tm...@gmail.com> wrote:
> well, if I set "batch" to true, I all of my load scripts die after a
> short amount of time with this error:
>
> /var/lib/gems/1.8/gems/couchrest-0.24/lib/couchrest/monkeypatches.rb:41:in
> `rbuf_fill': uninitialized constant Timeout::TimeoutError (NameError)
>        from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
>        from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
>        from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
>
> Regardless, it still seems like there is a bottleneck on the server
> end.  Did I mention I'm running the 'load' scripts locally?  So it's
> not network latency that is causing the slowness.  Any other ideas?
>

You're probably best of using explicit bulk_docs saves with an array
of documents. That way you know how much you are passing to CouchDB at
a time. With smallish docs (less than a few kb) you can usually do
around 1000 at a time to get the best insert performance.

> Thanks.
> -Tom
>
>
> On Mon, May 4, 2009 at 12:19 PM, Zachary Zolton
> <za...@gmail.com> wrote:
>> Yeah, the optional second argument —for usign bulk save semantics—
>> defaults to false.
>>
>> Also, there's an option where you can set how many documents to batch
>> save at a time. I don't remember the default, but I've had good luck
>> saving with anywhere between 500 and 2000 docs.
>>
>> On Mon, May 4, 2009 at 11:13 AM, Tom Nichols <tm...@gmail.com> wrote:
>>> Thanks.  I'm using save_doc, I just need to pass 'true' as a second argument?
>>>
>>> I posted the question here because I assumed the performance
>>> bottleneck was on the CouchDB end, not my ruby script.  Am I wrong? I
>>> assumed if I was running 20 "slow" ruby scripts they would peg the
>>> CPU.  The fact that I'm not seeing that makes me think there is some
>>> blocking/ synchronization that is making the CouchDB server slow....?
>>>
>>> Thanks again.
>>> -Tom
>>>
>>> On Mon, May 4, 2009 at 11:58 AM, Zachary Zolton
>>> <za...@gmail.com> wrote:
>>>> Short answer: use db.save_doc(hash, true) for bulk_docs behavior.
>>>>
>>>> Also, consider moving this thread to the CouchRest Google Group:
>>>> http://groups.google.com/group/couchrest/topics
>>>>
>>>> Cheers,
>>>> zdzolton
>>>>
>>>> On Mon, May 4, 2009 at 10:40 AM, Tom Nichols <tm...@gmail.com> wrote:
>>>>> Hi, I have some questions about insert performance.
>>>>>
>>>>> I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
>>>>> attached a huge EBS volume to it and mounted it where CouchDB's data
>>>>> files are stored.  I fired up about ruby scripts running inserts and
>>>>> after a weekend I only have about 30GB/ 12M rows of data...  Which
>>>>> seems small.  'top' tells me that my CPU is only about 30% utilized.
>>>>>
>>>>> Any idea what I might be doing wrong?  I pretty much just followed
>>>>> these instructions:
>>>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>>>>
>>>>> My ruby script looks like this:
>>>>> #!/usr/bin/env ruby
>>>>> #Script to load random data into CouchDB
>>>>>
>>>>> require 'rubygems'
>>>>> require 'couchrest'
>>>>>
>>>>> db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
>>>>> puts "Created database: #{ARGV[0]}"
>>>>>
>>>>> max = 9999999999999999
>>>>> while 1
>>>>>        puts 'loading...'
>>>>>        for val in 0..max
>>>>>                db.save_doc({ :key => val, 'val one' => "val ${val}",
>>>>> 'val2' => "#{ARGV[1]} #{val}" })
>>>>>        end
>>>>> end
>>>>>
>>>>>
>>>>> Thanks in advance...
>>>>>
>>>>
>>>
>>
>



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Re: Insert performance

Posted by Tom Nichols <tm...@gmail.com>.
well, if I set "batch" to true, I all of my load scripts die after a
short amount of time with this error:

/var/lib/gems/1.8/gems/couchrest-0.24/lib/couchrest/monkeypatches.rb:41:in
`rbuf_fill': uninitialized constant Timeout::TimeoutError (NameError)
        from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
        from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
        from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'

Regardless, it still seems like there is a bottleneck on the server
end.  Did I mention I'm running the 'load' scripts locally?  So it's
not network latency that is causing the slowness.  Any other ideas?

Thanks.
-Tom


On Mon, May 4, 2009 at 12:19 PM, Zachary Zolton
<za...@gmail.com> wrote:
> Yeah, the optional second argument —for usign bulk save semantics—
> defaults to false.
>
> Also, there's an option where you can set how many documents to batch
> save at a time. I don't remember the default, but I've had good luck
> saving with anywhere between 500 and 2000 docs.
>
> On Mon, May 4, 2009 at 11:13 AM, Tom Nichols <tm...@gmail.com> wrote:
>> Thanks.  I'm using save_doc, I just need to pass 'true' as a second argument?
>>
>> I posted the question here because I assumed the performance
>> bottleneck was on the CouchDB end, not my ruby script.  Am I wrong? I
>> assumed if I was running 20 "slow" ruby scripts they would peg the
>> CPU.  The fact that I'm not seeing that makes me think there is some
>> blocking/ synchronization that is making the CouchDB server slow....?
>>
>> Thanks again.
>> -Tom
>>
>> On Mon, May 4, 2009 at 11:58 AM, Zachary Zolton
>> <za...@gmail.com> wrote:
>>> Short answer: use db.save_doc(hash, true) for bulk_docs behavior.
>>>
>>> Also, consider moving this thread to the CouchRest Google Group:
>>> http://groups.google.com/group/couchrest/topics
>>>
>>> Cheers,
>>> zdzolton
>>>
>>> On Mon, May 4, 2009 at 10:40 AM, Tom Nichols <tm...@gmail.com> wrote:
>>>> Hi, I have some questions about insert performance.
>>>>
>>>> I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
>>>> attached a huge EBS volume to it and mounted it where CouchDB's data
>>>> files are stored.  I fired up about ruby scripts running inserts and
>>>> after a weekend I only have about 30GB/ 12M rows of data...  Which
>>>> seems small.  'top' tells me that my CPU is only about 30% utilized.
>>>>
>>>> Any idea what I might be doing wrong?  I pretty much just followed
>>>> these instructions:
>>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>>>
>>>> My ruby script looks like this:
>>>> #!/usr/bin/env ruby
>>>> #Script to load random data into CouchDB
>>>>
>>>> require 'rubygems'
>>>> require 'couchrest'
>>>>
>>>> db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
>>>> puts "Created database: #{ARGV[0]}"
>>>>
>>>> max = 9999999999999999
>>>> while 1
>>>>        puts 'loading...'
>>>>        for val in 0..max
>>>>                db.save_doc({ :key => val, 'val one' => "val ${val}",
>>>> 'val2' => "#{ARGV[1]} #{val}" })
>>>>        end
>>>> end
>>>>
>>>>
>>>> Thanks in advance...
>>>>
>>>
>>
>

Re: Insert performance

Posted by Zachary Zolton <za...@gmail.com>.
Yeah, the optional second argument —for usign bulk save semantics—
defaults to false.

Also, there's an option where you can set how many documents to batch
save at a time. I don't remember the default, but I've had good luck
saving with anywhere between 500 and 2000 docs.

On Mon, May 4, 2009 at 11:13 AM, Tom Nichols <tm...@gmail.com> wrote:
> Thanks.  I'm using save_doc, I just need to pass 'true' as a second argument?
>
> I posted the question here because I assumed the performance
> bottleneck was on the CouchDB end, not my ruby script.  Am I wrong? I
> assumed if I was running 20 "slow" ruby scripts they would peg the
> CPU.  The fact that I'm not seeing that makes me think there is some
> blocking/ synchronization that is making the CouchDB server slow....?
>
> Thanks again.
> -Tom
>
> On Mon, May 4, 2009 at 11:58 AM, Zachary Zolton
> <za...@gmail.com> wrote:
>> Short answer: use db.save_doc(hash, true) for bulk_docs behavior.
>>
>> Also, consider moving this thread to the CouchRest Google Group:
>> http://groups.google.com/group/couchrest/topics
>>
>> Cheers,
>> zdzolton
>>
>> On Mon, May 4, 2009 at 10:40 AM, Tom Nichols <tm...@gmail.com> wrote:
>>> Hi, I have some questions about insert performance.
>>>
>>> I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
>>> attached a huge EBS volume to it and mounted it where CouchDB's data
>>> files are stored.  I fired up about ruby scripts running inserts and
>>> after a weekend I only have about 30GB/ 12M rows of data...  Which
>>> seems small.  'top' tells me that my CPU is only about 30% utilized.
>>>
>>> Any idea what I might be doing wrong?  I pretty much just followed
>>> these instructions:
>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>>
>>> My ruby script looks like this:
>>> #!/usr/bin/env ruby
>>> #Script to load random data into CouchDB
>>>
>>> require 'rubygems'
>>> require 'couchrest'
>>>
>>> db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
>>> puts "Created database: #{ARGV[0]}"
>>>
>>> max = 9999999999999999
>>> while 1
>>>        puts 'loading...'
>>>        for val in 0..max
>>>                db.save_doc({ :key => val, 'val one' => "val ${val}",
>>> 'val2' => "#{ARGV[1]} #{val}" })
>>>        end
>>> end
>>>
>>>
>>> Thanks in advance...
>>>
>>
>

Re: Insert performance

Posted by Tom Nichols <tm...@gmail.com>.
Thanks.  I'm using save_doc, I just need to pass 'true' as a second argument?

I posted the question here because I assumed the performance
bottleneck was on the CouchDB end, not my ruby script.  Am I wrong? I
assumed if I was running 20 "slow" ruby scripts they would peg the
CPU.  The fact that I'm not seeing that makes me think there is some
blocking/ synchronization that is making the CouchDB server slow....?

Thanks again.
-Tom

On Mon, May 4, 2009 at 11:58 AM, Zachary Zolton
<za...@gmail.com> wrote:
> Short answer: use db.save_doc(hash, true) for bulk_docs behavior.
>
> Also, consider moving this thread to the CouchRest Google Group:
> http://groups.google.com/group/couchrest/topics
>
> Cheers,
> zdzolton
>
> On Mon, May 4, 2009 at 10:40 AM, Tom Nichols <tm...@gmail.com> wrote:
>> Hi, I have some questions about insert performance.
>>
>> I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
>> attached a huge EBS volume to it and mounted it where CouchDB's data
>> files are stored.  I fired up about ruby scripts running inserts and
>> after a weekend I only have about 30GB/ 12M rows of data...  Which
>> seems small.  'top' tells me that my CPU is only about 30% utilized.
>>
>> Any idea what I might be doing wrong?  I pretty much just followed
>> these instructions:
>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>
>> My ruby script looks like this:
>> #!/usr/bin/env ruby
>> #Script to load random data into CouchDB
>>
>> require 'rubygems'
>> require 'couchrest'
>>
>> db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
>> puts "Created database: #{ARGV[0]}"
>>
>> max = 9999999999999999
>> while 1
>>        puts 'loading...'
>>        for val in 0..max
>>                db.save_doc({ :key => val, 'val one' => "val ${val}",
>> 'val2' => "#{ARGV[1]} #{val}" })
>>        end
>> end
>>
>>
>> Thanks in advance...
>>
>

Re: Insert performance

Posted by Zachary Zolton <za...@gmail.com>.
Short answer: use db.save_doc(hash, true) for bulk_docs behavior.

Also, consider moving this thread to the CouchRest Google Group:
http://groups.google.com/group/couchrest/topics

Cheers,
zdzolton

On Mon, May 4, 2009 at 10:40 AM, Tom Nichols <tm...@gmail.com> wrote:
> Hi, I have some questions about insert performance.
>
> I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
> attached a huge EBS volume to it and mounted it where CouchDB's data
> files are stored.  I fired up about ruby scripts running inserts and
> after a weekend I only have about 30GB/ 12M rows of data...  Which
> seems small.  'top' tells me that my CPU is only about 30% utilized.
>
> Any idea what I might be doing wrong?  I pretty much just followed
> these instructions:
> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>
> My ruby script looks like this:
> #!/usr/bin/env ruby
> #Script to load random data into CouchDB
>
> require 'rubygems'
> require 'couchrest'
>
> db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
> puts "Created database: #{ARGV[0]}"
>
> max = 9999999999999999
> while 1
>        puts 'loading...'
>        for val in 0..max
>                db.save_doc({ :key => val, 'val one' => "val ${val}",
> 'val2' => "#{ARGV[1]} #{val}" })
>        end
> end
>
>
> Thanks in advance...
>

Re: Insert performance

Posted by Adam Kocoloski <ko...@apache.org>.
Hi Tom, those numbers seems to be in the right ballpark to me.  The  
single-core / dual-core distinction isn't quite right for EC2, where  
it's all about "compute units".  Regardless of the actual CPUs used, a  
c1.medium should get 5x the cycles of an m1.small.  Amazon's  
hypervisor will "steal" cycles from your m1.small instance to preserve  
this property if it has to.

The fact that your wallclock time dropped by 4.3x when you upgraded to  
a c1.medium suggests that you were mostly CPU bound.  Cheers,

Adam

On May 6, 2009, at 9:39 AM, Tom Nichols wrote:

> Hi Chris, here are my results:
> m1.small running Ubuntu Intrepid:
> real    1m3.049s
> user    0m25.040s
> sys     0m2.240s
>
> c1.medium instance:
> real    0m14.752s
> user    0m23.990s
> sys     0m2.220s
>
> In practice, the c1.medium seemed to run my ruby scripts a bit faster,
> giving a thoughput of about 0.98 MB/s, or 730 docs/s.  I thought the
> m1.small instance was still dual-core; if it's a single-core machine
> then that at least partly explains why running a bunch of load scripts
> on the same box as CouchDB would be so slow.
>
> Thoughts?  Is this the performance I should expect?
>
> Thanks.
> -Tom
>
>
> On Tue, May 5, 2009 at 3:33 PM, Chris Anderson <jc...@apache.org>  
> wrote:
>> On Tue, May 5, 2009 at 10:43 AM, Tom Nichols <tm...@gmail.com>  
>> wrote:
>>> So I did a rough calculation and it looks like I'm getting less than
>>> 1MB/s throughput in CouchDB --
>>>
>>> 3072 MB total / 6900 sec = 0.445 MB/s
>>>
>>> So if the disk throughput is ~20 to 30 MB/s then the bottleneck is
>>> somewhere in the database.  It's obviously not going to be anywhere
>>> close to raw disk I/O speeds but this still seems incredibly slow.
>>> Granted, I'm using a small instance...  I'll try a c1.medium and see
>>> if the results are drastically different.
>>>
>>
>> Can you try running this benchmark script and see what you get for
>> insert performance:
>>
>> http://gist.github.com/79279
>>
>>> On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jhs@proven-corporation.com 
>>> > wrote:
>>>> Tom Nichols wrote:
>>>>>
>>>>> Hi, I have some questions about insert performance.
>>>>>
>>>>> I have a single CouchDB 0.9.0 node running on small EC2  
>>>>> instance.  I
>>>>> attached a huge EBS volume to it and mounted it where CouchDB's  
>>>>> data
>>>>> files are stored.  I fired up about ruby scripts running inserts  
>>>>> and
>>>>> after a weekend I only have about 30GB/ 12M rows of data...  Which
>>>>> seems small.  'top' tells me that my CPU is only about 30%  
>>>>> utilized.
>>>>>
>>>>> Any idea what I might be doing wrong?  I pretty much just followed
>>>>> these instructions:
>>>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>>>
>>>> Hi, Tom.  I believe I read somewhere before that the smallest EC2  
>>>> instances
>>>> have a slower and/or higher-latency connection to EBS, so you  
>>>> might want to
>>>> consider a large instance, or maybe even a high-memory small  
>>>> instance and
>>>> see whether you get better "hardware" performance.
>>>>
>>>> Although strangely, when googling it, the first article I found  
>>>> says that
>>>> their benchmarks found no difference between EBS or even the  
>>>> ephemeral
>>>> filesystem.
>>>>
>>>> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
>>>>
>>>> On the other hand, here is a forum posting and a random benchmark  
>>>> indicating
>>>> that more expensive instances get better throughput:
>>>>
>>>> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
>>>> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
>>>>
>>>> --
>>>> Jason Smith
>>>> Proven Corporation
>>>> Bangkok, Thailand
>>>> http://www.proven-corporation.com
>>>>
>>>
>>
>>
>>
>> --
>> Chris Anderson
>> http://jchrisa.net
>> http://couch.io
>>


Re: Insert performance

Posted by Tom Nichols <tm...@gmail.com>.
Hi Chris, here are my results:
m1.small running Ubuntu Intrepid:
real    1m3.049s
user    0m25.040s
sys     0m2.240s

c1.medium instance:
real    0m14.752s
user    0m23.990s
sys     0m2.220s

In practice, the c1.medium seemed to run my ruby scripts a bit faster,
giving a thoughput of about 0.98 MB/s, or 730 docs/s.  I thought the
m1.small instance was still dual-core; if it's a single-core machine
then that at least partly explains why running a bunch of load scripts
on the same box as CouchDB would be so slow.

Thoughts?  Is this the performance I should expect?

Thanks.
-Tom


On Tue, May 5, 2009 at 3:33 PM, Chris Anderson <jc...@apache.org> wrote:
> On Tue, May 5, 2009 at 10:43 AM, Tom Nichols <tm...@gmail.com> wrote:
>> So I did a rough calculation and it looks like I'm getting less than
>> 1MB/s throughput in CouchDB --
>>
>> 3072 MB total / 6900 sec = 0.445 MB/s
>>
>> So if the disk throughput is ~20 to 30 MB/s then the bottleneck is
>> somewhere in the database.  It's obviously not going to be anywhere
>> close to raw disk I/O speeds but this still seems incredibly slow.
>> Granted, I'm using a small instance...  I'll try a c1.medium and see
>> if the results are drastically different.
>>
>
> Can you try running this benchmark script and see what you get for
> insert performance:
>
> http://gist.github.com/79279
>
>> On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jh...@proven-corporation.com> wrote:
>>> Tom Nichols wrote:
>>>>
>>>> Hi, I have some questions about insert performance.
>>>>
>>>> I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
>>>> attached a huge EBS volume to it and mounted it where CouchDB's data
>>>> files are stored.  I fired up about ruby scripts running inserts and
>>>> after a weekend I only have about 30GB/ 12M rows of data...  Which
>>>> seems small.  'top' tells me that my CPU is only about 30% utilized.
>>>>
>>>> Any idea what I might be doing wrong?  I pretty much just followed
>>>> these instructions:
>>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>>
>>> Hi, Tom.  I believe I read somewhere before that the smallest EC2 instances
>>> have a slower and/or higher-latency connection to EBS, so you might want to
>>> consider a large instance, or maybe even a high-memory small instance and
>>> see whether you get better "hardware" performance.
>>>
>>> Although strangely, when googling it, the first article I found says that
>>> their benchmarks found no difference between EBS or even the ephemeral
>>> filesystem.
>>>
>>> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
>>>
>>> On the other hand, here is a forum posting and a random benchmark indicating
>>> that more expensive instances get better throughput:
>>>
>>> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
>>> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
>>>
>>> --
>>> Jason Smith
>>> Proven Corporation
>>> Bangkok, Thailand
>>> http://www.proven-corporation.com
>>>
>>
>
>
>
> --
> Chris Anderson
> http://jchrisa.net
> http://couch.io
>

Re: Insert performance

Posted by Chris Anderson <jc...@apache.org>.
On Tue, May 5, 2009 at 10:43 AM, Tom Nichols <tm...@gmail.com> wrote:
> So I did a rough calculation and it looks like I'm getting less than
> 1MB/s throughput in CouchDB --
>
> 3072 MB total / 6900 sec = 0.445 MB/s
>
> So if the disk throughput is ~20 to 30 MB/s then the bottleneck is
> somewhere in the database.  It's obviously not going to be anywhere
> close to raw disk I/O speeds but this still seems incredibly slow.
> Granted, I'm using a small instance...  I'll try a c1.medium and see
> if the results are drastically different.
>

Can you try running this benchmark script and see what you get for
insert performance:

http://gist.github.com/79279

> On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jh...@proven-corporation.com> wrote:
>> Tom Nichols wrote:
>>>
>>> Hi, I have some questions about insert performance.
>>>
>>> I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
>>> attached a huge EBS volume to it and mounted it where CouchDB's data
>>> files are stored.  I fired up about ruby scripts running inserts and
>>> after a weekend I only have about 30GB/ 12M rows of data...  Which
>>> seems small.  'top' tells me that my CPU is only about 30% utilized.
>>>
>>> Any idea what I might be doing wrong?  I pretty much just followed
>>> these instructions:
>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>
>> Hi, Tom.  I believe I read somewhere before that the smallest EC2 instances
>> have a slower and/or higher-latency connection to EBS, so you might want to
>> consider a large instance, or maybe even a high-memory small instance and
>> see whether you get better "hardware" performance.
>>
>> Although strangely, when googling it, the first article I found says that
>> their benchmarks found no difference between EBS or even the ephemeral
>> filesystem.
>>
>> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
>>
>> On the other hand, here is a forum posting and a random benchmark indicating
>> that more expensive instances get better throughput:
>>
>> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
>> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
>>
>> --
>> Jason Smith
>> Proven Corporation
>> Bangkok, Thailand
>> http://www.proven-corporation.com
>>
>



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Re: Insert performance

Posted by Tom Nichols <tm...@gmail.com>.
So I did a rough calculation and it looks like I'm getting less than
1MB/s throughput in CouchDB --

3072 MB total / 6900 sec = 0.445 MB/s

So if the disk throughput is ~20 to 30 MB/s then the bottleneck is
somewhere in the database.  It's obviously not going to be anywhere
close to raw disk I/O speeds but this still seems incredibly slow.
Granted, I'm using a small instance...  I'll try a c1.medium and see
if the results are drastically different.

On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jh...@proven-corporation.com> wrote:
> Tom Nichols wrote:
>>
>> Hi, I have some questions about insert performance.
>>
>> I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
>> attached a huge EBS volume to it and mounted it where CouchDB's data
>> files are stored.  I fired up about ruby scripts running inserts and
>> after a weekend I only have about 30GB/ 12M rows of data...  Which
>> seems small.  'top' tells me that my CPU is only about 30% utilized.
>>
>> Any idea what I might be doing wrong?  I pretty much just followed
>> these instructions:
>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>
> Hi, Tom.  I believe I read somewhere before that the smallest EC2 instances
> have a slower and/or higher-latency connection to EBS, so you might want to
> consider a large instance, or maybe even a high-memory small instance and
> see whether you get better "hardware" performance.
>
> Although strangely, when googling it, the first article I found says that
> their benchmarks found no difference between EBS or even the ephemeral
> filesystem.
>
> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
>
> On the other hand, here is a forum posting and a random benchmark indicating
> that more expensive instances get better throughput:
>
> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
>
> --
> Jason Smith
> Proven Corporation
> Bangkok, Thailand
> http://www.proven-corporation.com
>

Re: Insert performance

Posted by Dustin Whitney <du...@gmail.com>.
That's kind of a bummer with respect to the small EC2 instances and
throughput.  What can the CouchDB guys share about their setup with CouchIO
- I'm not sure if I have the name right, but I mean the service where you
guys host the CouchDB on EC2.

-Dustin

On Tue, May 5, 2009 at 12:59 PM, Tom Nichols <tm...@gmail.com> wrote:

> Thanks Jason.
>
> I removed the EBS volume and things seemed a little faster.  CouchDB
> is actually saturating the CPU now, so it looks like I'm getting
> "accurate" benchmarks now.  i.e. I'm actually exercising CouchDB and
> I'm not limited by some hardware/OS latency.
>
> So I had 10 concurrent ruby processes performing batch inserts into
> two test databases.  After 2 hours, I have 1.1M documents (1.5 GB) in
> each database, for a total of 2GB and 2.2 million documents.  Does
> this still seem slow?
>
> Thanks.
> -Tom
>
>
> On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jh...@proven-corporation.com>
> wrote:
> > Tom Nichols wrote:
> >>
> >> Hi, I have some questions about insert performance.
> >>
> >> I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
> >> attached a huge EBS volume to it and mounted it where CouchDB's data
> >> files are stored.  I fired up about ruby scripts running inserts and
> >> after a weekend I only have about 30GB/ 12M rows of data...  Which
> >> seems small.  'top' tells me that my CPU is only about 30% utilized.
> >>
> >> Any idea what I might be doing wrong?  I pretty much just followed
> >> these instructions:
> >> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
> >
> > Hi, Tom.  I believe I read somewhere before that the smallest EC2
> instances
> > have a slower and/or higher-latency connection to EBS, so you might want
> to
> > consider a large instance, or maybe even a high-memory small instance and
> > see whether you get better "hardware" performance.
> >
> > Although strangely, when googling it, the first article I found says that
> > their benchmarks found no difference between EBS or even the ephemeral
> > filesystem.
> >
> >
> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
> >
> > On the other hand, here is a forum posting and a random benchmark
> indicating
> > that more expensive instances get better throughput:
> >
> >
> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
> >
> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
> >
> > --
> > Jason Smith
> > Proven Corporation
> > Bangkok, Thailand
> > http://www.proven-corporation.com
> >
>

Re: Insert performance

Posted by Tom Nichols <tm...@gmail.com>.
Thanks Jason.

I removed the EBS volume and things seemed a little faster.  CouchDB
is actually saturating the CPU now, so it looks like I'm getting
"accurate" benchmarks now.  i.e. I'm actually exercising CouchDB and
I'm not limited by some hardware/OS latency.

So I had 10 concurrent ruby processes performing batch inserts into
two test databases.  After 2 hours, I have 1.1M documents (1.5 GB) in
each database, for a total of 2GB and 2.2 million documents.  Does
this still seem slow?

Thanks.
-Tom


On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jh...@proven-corporation.com> wrote:
> Tom Nichols wrote:
>>
>> Hi, I have some questions about insert performance.
>>
>> I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
>> attached a huge EBS volume to it and mounted it where CouchDB's data
>> files are stored.  I fired up about ruby scripts running inserts and
>> after a weekend I only have about 30GB/ 12M rows of data...  Which
>> seems small.  'top' tells me that my CPU is only about 30% utilized.
>>
>> Any idea what I might be doing wrong?  I pretty much just followed
>> these instructions:
>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>
> Hi, Tom.  I believe I read somewhere before that the smallest EC2 instances
> have a slower and/or higher-latency connection to EBS, so you might want to
> consider a large instance, or maybe even a high-memory small instance and
> see whether you get better "hardware" performance.
>
> Although strangely, when googling it, the first article I found says that
> their benchmarks found no difference between EBS or even the ephemeral
> filesystem.
>
> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
>
> On the other hand, here is a forum posting and a random benchmark indicating
> that more expensive instances get better throughput:
>
> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
>
> --
> Jason Smith
> Proven Corporation
> Bangkok, Thailand
> http://www.proven-corporation.com
>

Re: Insert performance

Posted by Jason Smith <jh...@proven-corporation.com>.
Tom Nichols wrote:
> Hi, I have some questions about insert performance.
> 
> I have a single CouchDB 0.9.0 node running on small EC2 instance.  I
> attached a huge EBS volume to it and mounted it where CouchDB's data
> files are stored.  I fired up about ruby scripts running inserts and
> after a weekend I only have about 30GB/ 12M rows of data...  Which
> seems small.  'top' tells me that my CPU is only about 30% utilized.
> 
> Any idea what I might be doing wrong?  I pretty much just followed
> these instructions:
> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2

Hi, Tom.  I believe I read somewhere before that the smallest EC2 
instances have a slower and/or higher-latency connection to EBS, so you 
might want to consider a large instance, or maybe even a high-memory 
small instance and see whether you get better "hardware" performance.

Although strangely, when googling it, the first article I found says 
that their benchmarks found no difference between EBS or even the 
ephemeral filesystem.

http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/

On the other hand, here is a forum posting and a random benchmark 
indicating that more expensive instances get better throughput:

http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html

-- 
Jason Smith
Proven Corporation
Bangkok, Thailand
http://www.proven-corporation.com