You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Tom Nichols <tm...@gmail.com> on 2009/05/04 17:40:53 UTC
Insert performance
Hi, I have some questions about insert performance.
I have a single CouchDB 0.9.0 node running on small EC2 instance. I
attached a huge EBS volume to it and mounted it where CouchDB's data
files are stored. I fired up about ruby scripts running inserts and
after a weekend I only have about 30GB/ 12M rows of data... Which
seems small. 'top' tells me that my CPU is only about 30% utilized.
Any idea what I might be doing wrong? I pretty much just followed
these instructions:
http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
My ruby script looks like this:
#!/usr/bin/env ruby
#Script to load random data into CouchDB
require 'rubygems'
require 'couchrest'
db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
puts "Created database: #{ARGV[0]}"
max = 9999999999999999
while 1
puts 'loading...'
for val in 0..max
db.save_doc({ :key => val, 'val one' => "val ${val}",
'val2' => "#{ARGV[1]} #{val}" })
end
end
Thanks in advance...
Re: Insert performance
Posted by Chris Anderson <jc...@apache.org>.
On Mon, May 4, 2009 at 10:20 AM, Tom Nichols <tm...@gmail.com> wrote:
> well, if I set "batch" to true, I all of my load scripts die after a
> short amount of time with this error:
>
> /var/lib/gems/1.8/gems/couchrest-0.24/lib/couchrest/monkeypatches.rb:41:in
> `rbuf_fill': uninitialized constant Timeout::TimeoutError (NameError)
> from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
> from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
> from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
>
> Regardless, it still seems like there is a bottleneck on the server
> end. Did I mention I'm running the 'load' scripts locally? So it's
> not network latency that is causing the slowness. Any other ideas?
>
You're probably best of using explicit bulk_docs saves with an array
of documents. That way you know how much you are passing to CouchDB at
a time. With smallish docs (less than a few kb) you can usually do
around 1000 at a time to get the best insert performance.
> Thanks.
> -Tom
>
>
> On Mon, May 4, 2009 at 12:19 PM, Zachary Zolton
> <za...@gmail.com> wrote:
>> Yeah, the optional second argument —for usign bulk save semantics—
>> defaults to false.
>>
>> Also, there's an option where you can set how many documents to batch
>> save at a time. I don't remember the default, but I've had good luck
>> saving with anywhere between 500 and 2000 docs.
>>
>> On Mon, May 4, 2009 at 11:13 AM, Tom Nichols <tm...@gmail.com> wrote:
>>> Thanks. I'm using save_doc, I just need to pass 'true' as a second argument?
>>>
>>> I posted the question here because I assumed the performance
>>> bottleneck was on the CouchDB end, not my ruby script. Am I wrong? I
>>> assumed if I was running 20 "slow" ruby scripts they would peg the
>>> CPU. The fact that I'm not seeing that makes me think there is some
>>> blocking/ synchronization that is making the CouchDB server slow....?
>>>
>>> Thanks again.
>>> -Tom
>>>
>>> On Mon, May 4, 2009 at 11:58 AM, Zachary Zolton
>>> <za...@gmail.com> wrote:
>>>> Short answer: use db.save_doc(hash, true) for bulk_docs behavior.
>>>>
>>>> Also, consider moving this thread to the CouchRest Google Group:
>>>> http://groups.google.com/group/couchrest/topics
>>>>
>>>> Cheers,
>>>> zdzolton
>>>>
>>>> On Mon, May 4, 2009 at 10:40 AM, Tom Nichols <tm...@gmail.com> wrote:
>>>>> Hi, I have some questions about insert performance.
>>>>>
>>>>> I have a single CouchDB 0.9.0 node running on small EC2 instance. I
>>>>> attached a huge EBS volume to it and mounted it where CouchDB's data
>>>>> files are stored. I fired up about ruby scripts running inserts and
>>>>> after a weekend I only have about 30GB/ 12M rows of data... Which
>>>>> seems small. 'top' tells me that my CPU is only about 30% utilized.
>>>>>
>>>>> Any idea what I might be doing wrong? I pretty much just followed
>>>>> these instructions:
>>>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>>>>
>>>>> My ruby script looks like this:
>>>>> #!/usr/bin/env ruby
>>>>> #Script to load random data into CouchDB
>>>>>
>>>>> require 'rubygems'
>>>>> require 'couchrest'
>>>>>
>>>>> db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
>>>>> puts "Created database: #{ARGV[0]}"
>>>>>
>>>>> max = 9999999999999999
>>>>> while 1
>>>>> puts 'loading...'
>>>>> for val in 0..max
>>>>> db.save_doc({ :key => val, 'val one' => "val ${val}",
>>>>> 'val2' => "#{ARGV[1]} #{val}" })
>>>>> end
>>>>> end
>>>>>
>>>>>
>>>>> Thanks in advance...
>>>>>
>>>>
>>>
>>
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Insert performance
Posted by Tom Nichols <tm...@gmail.com>.
well, if I set "batch" to true, I all of my load scripts die after a
short amount of time with this error:
/var/lib/gems/1.8/gems/couchrest-0.24/lib/couchrest/monkeypatches.rb:41:in
`rbuf_fill': uninitialized constant Timeout::TimeoutError (NameError)
from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
Regardless, it still seems like there is a bottleneck on the server
end. Did I mention I'm running the 'load' scripts locally? So it's
not network latency that is causing the slowness. Any other ideas?
Thanks.
-Tom
On Mon, May 4, 2009 at 12:19 PM, Zachary Zolton
<za...@gmail.com> wrote:
> Yeah, the optional second argument —for usign bulk save semantics—
> defaults to false.
>
> Also, there's an option where you can set how many documents to batch
> save at a time. I don't remember the default, but I've had good luck
> saving with anywhere between 500 and 2000 docs.
>
> On Mon, May 4, 2009 at 11:13 AM, Tom Nichols <tm...@gmail.com> wrote:
>> Thanks. I'm using save_doc, I just need to pass 'true' as a second argument?
>>
>> I posted the question here because I assumed the performance
>> bottleneck was on the CouchDB end, not my ruby script. Am I wrong? I
>> assumed if I was running 20 "slow" ruby scripts they would peg the
>> CPU. The fact that I'm not seeing that makes me think there is some
>> blocking/ synchronization that is making the CouchDB server slow....?
>>
>> Thanks again.
>> -Tom
>>
>> On Mon, May 4, 2009 at 11:58 AM, Zachary Zolton
>> <za...@gmail.com> wrote:
>>> Short answer: use db.save_doc(hash, true) for bulk_docs behavior.
>>>
>>> Also, consider moving this thread to the CouchRest Google Group:
>>> http://groups.google.com/group/couchrest/topics
>>>
>>> Cheers,
>>> zdzolton
>>>
>>> On Mon, May 4, 2009 at 10:40 AM, Tom Nichols <tm...@gmail.com> wrote:
>>>> Hi, I have some questions about insert performance.
>>>>
>>>> I have a single CouchDB 0.9.0 node running on small EC2 instance. I
>>>> attached a huge EBS volume to it and mounted it where CouchDB's data
>>>> files are stored. I fired up about ruby scripts running inserts and
>>>> after a weekend I only have about 30GB/ 12M rows of data... Which
>>>> seems small. 'top' tells me that my CPU is only about 30% utilized.
>>>>
>>>> Any idea what I might be doing wrong? I pretty much just followed
>>>> these instructions:
>>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>>>
>>>> My ruby script looks like this:
>>>> #!/usr/bin/env ruby
>>>> #Script to load random data into CouchDB
>>>>
>>>> require 'rubygems'
>>>> require 'couchrest'
>>>>
>>>> db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
>>>> puts "Created database: #{ARGV[0]}"
>>>>
>>>> max = 9999999999999999
>>>> while 1
>>>> puts 'loading...'
>>>> for val in 0..max
>>>> db.save_doc({ :key => val, 'val one' => "val ${val}",
>>>> 'val2' => "#{ARGV[1]} #{val}" })
>>>> end
>>>> end
>>>>
>>>>
>>>> Thanks in advance...
>>>>
>>>
>>
>
Re: Insert performance
Posted by Zachary Zolton <za...@gmail.com>.
Yeah, the optional second argument —for usign bulk save semantics—
defaults to false.
Also, there's an option where you can set how many documents to batch
save at a time. I don't remember the default, but I've had good luck
saving with anywhere between 500 and 2000 docs.
On Mon, May 4, 2009 at 11:13 AM, Tom Nichols <tm...@gmail.com> wrote:
> Thanks. I'm using save_doc, I just need to pass 'true' as a second argument?
>
> I posted the question here because I assumed the performance
> bottleneck was on the CouchDB end, not my ruby script. Am I wrong? I
> assumed if I was running 20 "slow" ruby scripts they would peg the
> CPU. The fact that I'm not seeing that makes me think there is some
> blocking/ synchronization that is making the CouchDB server slow....?
>
> Thanks again.
> -Tom
>
> On Mon, May 4, 2009 at 11:58 AM, Zachary Zolton
> <za...@gmail.com> wrote:
>> Short answer: use db.save_doc(hash, true) for bulk_docs behavior.
>>
>> Also, consider moving this thread to the CouchRest Google Group:
>> http://groups.google.com/group/couchrest/topics
>>
>> Cheers,
>> zdzolton
>>
>> On Mon, May 4, 2009 at 10:40 AM, Tom Nichols <tm...@gmail.com> wrote:
>>> Hi, I have some questions about insert performance.
>>>
>>> I have a single CouchDB 0.9.0 node running on small EC2 instance. I
>>> attached a huge EBS volume to it and mounted it where CouchDB's data
>>> files are stored. I fired up about ruby scripts running inserts and
>>> after a weekend I only have about 30GB/ 12M rows of data... Which
>>> seems small. 'top' tells me that my CPU is only about 30% utilized.
>>>
>>> Any idea what I might be doing wrong? I pretty much just followed
>>> these instructions:
>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>>
>>> My ruby script looks like this:
>>> #!/usr/bin/env ruby
>>> #Script to load random data into CouchDB
>>>
>>> require 'rubygems'
>>> require 'couchrest'
>>>
>>> db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
>>> puts "Created database: #{ARGV[0]}"
>>>
>>> max = 9999999999999999
>>> while 1
>>> puts 'loading...'
>>> for val in 0..max
>>> db.save_doc({ :key => val, 'val one' => "val ${val}",
>>> 'val2' => "#{ARGV[1]} #{val}" })
>>> end
>>> end
>>>
>>>
>>> Thanks in advance...
>>>
>>
>
Re: Insert performance
Posted by Tom Nichols <tm...@gmail.com>.
Thanks. I'm using save_doc, I just need to pass 'true' as a second argument?
I posted the question here because I assumed the performance
bottleneck was on the CouchDB end, not my ruby script. Am I wrong? I
assumed if I was running 20 "slow" ruby scripts they would peg the
CPU. The fact that I'm not seeing that makes me think there is some
blocking/ synchronization that is making the CouchDB server slow....?
Thanks again.
-Tom
On Mon, May 4, 2009 at 11:58 AM, Zachary Zolton
<za...@gmail.com> wrote:
> Short answer: use db.save_doc(hash, true) for bulk_docs behavior.
>
> Also, consider moving this thread to the CouchRest Google Group:
> http://groups.google.com/group/couchrest/topics
>
> Cheers,
> zdzolton
>
> On Mon, May 4, 2009 at 10:40 AM, Tom Nichols <tm...@gmail.com> wrote:
>> Hi, I have some questions about insert performance.
>>
>> I have a single CouchDB 0.9.0 node running on small EC2 instance. I
>> attached a huge EBS volume to it and mounted it where CouchDB's data
>> files are stored. I fired up about ruby scripts running inserts and
>> after a weekend I only have about 30GB/ 12M rows of data... Which
>> seems small. 'top' tells me that my CPU is only about 30% utilized.
>>
>> Any idea what I might be doing wrong? I pretty much just followed
>> these instructions:
>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>
>> My ruby script looks like this:
>> #!/usr/bin/env ruby
>> #Script to load random data into CouchDB
>>
>> require 'rubygems'
>> require 'couchrest'
>>
>> db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
>> puts "Created database: #{ARGV[0]}"
>>
>> max = 9999999999999999
>> while 1
>> puts 'loading...'
>> for val in 0..max
>> db.save_doc({ :key => val, 'val one' => "val ${val}",
>> 'val2' => "#{ARGV[1]} #{val}" })
>> end
>> end
>>
>>
>> Thanks in advance...
>>
>
Re: Insert performance
Posted by Zachary Zolton <za...@gmail.com>.
Short answer: use db.save_doc(hash, true) for bulk_docs behavior.
Also, consider moving this thread to the CouchRest Google Group:
http://groups.google.com/group/couchrest/topics
Cheers,
zdzolton
On Mon, May 4, 2009 at 10:40 AM, Tom Nichols <tm...@gmail.com> wrote:
> Hi, I have some questions about insert performance.
>
> I have a single CouchDB 0.9.0 node running on small EC2 instance. I
> attached a huge EBS volume to it and mounted it where CouchDB's data
> files are stored. I fired up about ruby scripts running inserts and
> after a weekend I only have about 30GB/ 12M rows of data... Which
> seems small. 'top' tells me that my CPU is only about 30% utilized.
>
> Any idea what I might be doing wrong? I pretty much just followed
> these instructions:
> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>
> My ruby script looks like this:
> #!/usr/bin/env ruby
> #Script to load random data into CouchDB
>
> require 'rubygems'
> require 'couchrest'
>
> db = CouchRest.database! "http://127.0.0.1:5984/#{ARGV[0]}"
> puts "Created database: #{ARGV[0]}"
>
> max = 9999999999999999
> while 1
> puts 'loading...'
> for val in 0..max
> db.save_doc({ :key => val, 'val one' => "val ${val}",
> 'val2' => "#{ARGV[1]} #{val}" })
> end
> end
>
>
> Thanks in advance...
>
Re: Insert performance
Posted by Adam Kocoloski <ko...@apache.org>.
Hi Tom, those numbers seems to be in the right ballpark to me. The
single-core / dual-core distinction isn't quite right for EC2, where
it's all about "compute units". Regardless of the actual CPUs used, a
c1.medium should get 5x the cycles of an m1.small. Amazon's
hypervisor will "steal" cycles from your m1.small instance to preserve
this property if it has to.
The fact that your wallclock time dropped by 4.3x when you upgraded to
a c1.medium suggests that you were mostly CPU bound. Cheers,
Adam
On May 6, 2009, at 9:39 AM, Tom Nichols wrote:
> Hi Chris, here are my results:
> m1.small running Ubuntu Intrepid:
> real 1m3.049s
> user 0m25.040s
> sys 0m2.240s
>
> c1.medium instance:
> real 0m14.752s
> user 0m23.990s
> sys 0m2.220s
>
> In practice, the c1.medium seemed to run my ruby scripts a bit faster,
> giving a thoughput of about 0.98 MB/s, or 730 docs/s. I thought the
> m1.small instance was still dual-core; if it's a single-core machine
> then that at least partly explains why running a bunch of load scripts
> on the same box as CouchDB would be so slow.
>
> Thoughts? Is this the performance I should expect?
>
> Thanks.
> -Tom
>
>
> On Tue, May 5, 2009 at 3:33 PM, Chris Anderson <jc...@apache.org>
> wrote:
>> On Tue, May 5, 2009 at 10:43 AM, Tom Nichols <tm...@gmail.com>
>> wrote:
>>> So I did a rough calculation and it looks like I'm getting less than
>>> 1MB/s throughput in CouchDB --
>>>
>>> 3072 MB total / 6900 sec = 0.445 MB/s
>>>
>>> So if the disk throughput is ~20 to 30 MB/s then the bottleneck is
>>> somewhere in the database. It's obviously not going to be anywhere
>>> close to raw disk I/O speeds but this still seems incredibly slow.
>>> Granted, I'm using a small instance... I'll try a c1.medium and see
>>> if the results are drastically different.
>>>
>>
>> Can you try running this benchmark script and see what you get for
>> insert performance:
>>
>> http://gist.github.com/79279
>>
>>> On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jhs@proven-corporation.com
>>> > wrote:
>>>> Tom Nichols wrote:
>>>>>
>>>>> Hi, I have some questions about insert performance.
>>>>>
>>>>> I have a single CouchDB 0.9.0 node running on small EC2
>>>>> instance. I
>>>>> attached a huge EBS volume to it and mounted it where CouchDB's
>>>>> data
>>>>> files are stored. I fired up about ruby scripts running inserts
>>>>> and
>>>>> after a weekend I only have about 30GB/ 12M rows of data... Which
>>>>> seems small. 'top' tells me that my CPU is only about 30%
>>>>> utilized.
>>>>>
>>>>> Any idea what I might be doing wrong? I pretty much just followed
>>>>> these instructions:
>>>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>>>
>>>> Hi, Tom. I believe I read somewhere before that the smallest EC2
>>>> instances
>>>> have a slower and/or higher-latency connection to EBS, so you
>>>> might want to
>>>> consider a large instance, or maybe even a high-memory small
>>>> instance and
>>>> see whether you get better "hardware" performance.
>>>>
>>>> Although strangely, when googling it, the first article I found
>>>> says that
>>>> their benchmarks found no difference between EBS or even the
>>>> ephemeral
>>>> filesystem.
>>>>
>>>> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
>>>>
>>>> On the other hand, here is a forum posting and a random benchmark
>>>> indicating
>>>> that more expensive instances get better throughput:
>>>>
>>>> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
>>>> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
>>>>
>>>> --
>>>> Jason Smith
>>>> Proven Corporation
>>>> Bangkok, Thailand
>>>> http://www.proven-corporation.com
>>>>
>>>
>>
>>
>>
>> --
>> Chris Anderson
>> http://jchrisa.net
>> http://couch.io
>>
Re: Insert performance
Posted by Tom Nichols <tm...@gmail.com>.
Hi Chris, here are my results:
m1.small running Ubuntu Intrepid:
real 1m3.049s
user 0m25.040s
sys 0m2.240s
c1.medium instance:
real 0m14.752s
user 0m23.990s
sys 0m2.220s
In practice, the c1.medium seemed to run my ruby scripts a bit faster,
giving a thoughput of about 0.98 MB/s, or 730 docs/s. I thought the
m1.small instance was still dual-core; if it's a single-core machine
then that at least partly explains why running a bunch of load scripts
on the same box as CouchDB would be so slow.
Thoughts? Is this the performance I should expect?
Thanks.
-Tom
On Tue, May 5, 2009 at 3:33 PM, Chris Anderson <jc...@apache.org> wrote:
> On Tue, May 5, 2009 at 10:43 AM, Tom Nichols <tm...@gmail.com> wrote:
>> So I did a rough calculation and it looks like I'm getting less than
>> 1MB/s throughput in CouchDB --
>>
>> 3072 MB total / 6900 sec = 0.445 MB/s
>>
>> So if the disk throughput is ~20 to 30 MB/s then the bottleneck is
>> somewhere in the database. It's obviously not going to be anywhere
>> close to raw disk I/O speeds but this still seems incredibly slow.
>> Granted, I'm using a small instance... I'll try a c1.medium and see
>> if the results are drastically different.
>>
>
> Can you try running this benchmark script and see what you get for
> insert performance:
>
> http://gist.github.com/79279
>
>> On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jh...@proven-corporation.com> wrote:
>>> Tom Nichols wrote:
>>>>
>>>> Hi, I have some questions about insert performance.
>>>>
>>>> I have a single CouchDB 0.9.0 node running on small EC2 instance. I
>>>> attached a huge EBS volume to it and mounted it where CouchDB's data
>>>> files are stored. I fired up about ruby scripts running inserts and
>>>> after a weekend I only have about 30GB/ 12M rows of data... Which
>>>> seems small. 'top' tells me that my CPU is only about 30% utilized.
>>>>
>>>> Any idea what I might be doing wrong? I pretty much just followed
>>>> these instructions:
>>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>>
>>> Hi, Tom. I believe I read somewhere before that the smallest EC2 instances
>>> have a slower and/or higher-latency connection to EBS, so you might want to
>>> consider a large instance, or maybe even a high-memory small instance and
>>> see whether you get better "hardware" performance.
>>>
>>> Although strangely, when googling it, the first article I found says that
>>> their benchmarks found no difference between EBS or even the ephemeral
>>> filesystem.
>>>
>>> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
>>>
>>> On the other hand, here is a forum posting and a random benchmark indicating
>>> that more expensive instances get better throughput:
>>>
>>> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
>>> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
>>>
>>> --
>>> Jason Smith
>>> Proven Corporation
>>> Bangkok, Thailand
>>> http://www.proven-corporation.com
>>>
>>
>
>
>
> --
> Chris Anderson
> http://jchrisa.net
> http://couch.io
>
Re: Insert performance
Posted by Chris Anderson <jc...@apache.org>.
On Tue, May 5, 2009 at 10:43 AM, Tom Nichols <tm...@gmail.com> wrote:
> So I did a rough calculation and it looks like I'm getting less than
> 1MB/s throughput in CouchDB --
>
> 3072 MB total / 6900 sec = 0.445 MB/s
>
> So if the disk throughput is ~20 to 30 MB/s then the bottleneck is
> somewhere in the database. It's obviously not going to be anywhere
> close to raw disk I/O speeds but this still seems incredibly slow.
> Granted, I'm using a small instance... I'll try a c1.medium and see
> if the results are drastically different.
>
Can you try running this benchmark script and see what you get for
insert performance:
http://gist.github.com/79279
> On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jh...@proven-corporation.com> wrote:
>> Tom Nichols wrote:
>>>
>>> Hi, I have some questions about insert performance.
>>>
>>> I have a single CouchDB 0.9.0 node running on small EC2 instance. I
>>> attached a huge EBS volume to it and mounted it where CouchDB's data
>>> files are stored. I fired up about ruby scripts running inserts and
>>> after a weekend I only have about 30GB/ 12M rows of data... Which
>>> seems small. 'top' tells me that my CPU is only about 30% utilized.
>>>
>>> Any idea what I might be doing wrong? I pretty much just followed
>>> these instructions:
>>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>>
>> Hi, Tom. I believe I read somewhere before that the smallest EC2 instances
>> have a slower and/or higher-latency connection to EBS, so you might want to
>> consider a large instance, or maybe even a high-memory small instance and
>> see whether you get better "hardware" performance.
>>
>> Although strangely, when googling it, the first article I found says that
>> their benchmarks found no difference between EBS or even the ephemeral
>> filesystem.
>>
>> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
>>
>> On the other hand, here is a forum posting and a random benchmark indicating
>> that more expensive instances get better throughput:
>>
>> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
>> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
>>
>> --
>> Jason Smith
>> Proven Corporation
>> Bangkok, Thailand
>> http://www.proven-corporation.com
>>
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Insert performance
Posted by Tom Nichols <tm...@gmail.com>.
So I did a rough calculation and it looks like I'm getting less than
1MB/s throughput in CouchDB --
3072 MB total / 6900 sec = 0.445 MB/s
So if the disk throughput is ~20 to 30 MB/s then the bottleneck is
somewhere in the database. It's obviously not going to be anywhere
close to raw disk I/O speeds but this still seems incredibly slow.
Granted, I'm using a small instance... I'll try a c1.medium and see
if the results are drastically different.
On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jh...@proven-corporation.com> wrote:
> Tom Nichols wrote:
>>
>> Hi, I have some questions about insert performance.
>>
>> I have a single CouchDB 0.9.0 node running on small EC2 instance. I
>> attached a huge EBS volume to it and mounted it where CouchDB's data
>> files are stored. I fired up about ruby scripts running inserts and
>> after a weekend I only have about 30GB/ 12M rows of data... Which
>> seems small. 'top' tells me that my CPU is only about 30% utilized.
>>
>> Any idea what I might be doing wrong? I pretty much just followed
>> these instructions:
>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>
> Hi, Tom. I believe I read somewhere before that the smallest EC2 instances
> have a slower and/or higher-latency connection to EBS, so you might want to
> consider a large instance, or maybe even a high-memory small instance and
> see whether you get better "hardware" performance.
>
> Although strangely, when googling it, the first article I found says that
> their benchmarks found no difference between EBS or even the ephemeral
> filesystem.
>
> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
>
> On the other hand, here is a forum posting and a random benchmark indicating
> that more expensive instances get better throughput:
>
> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
>
> --
> Jason Smith
> Proven Corporation
> Bangkok, Thailand
> http://www.proven-corporation.com
>
Re: Insert performance
Posted by Dustin Whitney <du...@gmail.com>.
That's kind of a bummer with respect to the small EC2 instances and
throughput. What can the CouchDB guys share about their setup with CouchIO
- I'm not sure if I have the name right, but I mean the service where you
guys host the CouchDB on EC2.
-Dustin
On Tue, May 5, 2009 at 12:59 PM, Tom Nichols <tm...@gmail.com> wrote:
> Thanks Jason.
>
> I removed the EBS volume and things seemed a little faster. CouchDB
> is actually saturating the CPU now, so it looks like I'm getting
> "accurate" benchmarks now. i.e. I'm actually exercising CouchDB and
> I'm not limited by some hardware/OS latency.
>
> So I had 10 concurrent ruby processes performing batch inserts into
> two test databases. After 2 hours, I have 1.1M documents (1.5 GB) in
> each database, for a total of 2GB and 2.2 million documents. Does
> this still seem slow?
>
> Thanks.
> -Tom
>
>
> On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jh...@proven-corporation.com>
> wrote:
> > Tom Nichols wrote:
> >>
> >> Hi, I have some questions about insert performance.
> >>
> >> I have a single CouchDB 0.9.0 node running on small EC2 instance. I
> >> attached a huge EBS volume to it and mounted it where CouchDB's data
> >> files are stored. I fired up about ruby scripts running inserts and
> >> after a weekend I only have about 30GB/ 12M rows of data... Which
> >> seems small. 'top' tells me that my CPU is only about 30% utilized.
> >>
> >> Any idea what I might be doing wrong? I pretty much just followed
> >> these instructions:
> >> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
> >
> > Hi, Tom. I believe I read somewhere before that the smallest EC2
> instances
> > have a slower and/or higher-latency connection to EBS, so you might want
> to
> > consider a large instance, or maybe even a high-memory small instance and
> > see whether you get better "hardware" performance.
> >
> > Although strangely, when googling it, the first article I found says that
> > their benchmarks found no difference between EBS or even the ephemeral
> > filesystem.
> >
> >
> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
> >
> > On the other hand, here is a forum posting and a random benchmark
> indicating
> > that more expensive instances get better throughput:
> >
> >
> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
> >
> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
> >
> > --
> > Jason Smith
> > Proven Corporation
> > Bangkok, Thailand
> > http://www.proven-corporation.com
> >
>
Re: Insert performance
Posted by Tom Nichols <tm...@gmail.com>.
Thanks Jason.
I removed the EBS volume and things seemed a little faster. CouchDB
is actually saturating the CPU now, so it looks like I'm getting
"accurate" benchmarks now. i.e. I'm actually exercising CouchDB and
I'm not limited by some hardware/OS latency.
So I had 10 concurrent ruby processes performing batch inserts into
two test databases. After 2 hours, I have 1.1M documents (1.5 GB) in
each database, for a total of 2GB and 2.2 million documents. Does
this still seem slow?
Thanks.
-Tom
On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jh...@proven-corporation.com> wrote:
> Tom Nichols wrote:
>>
>> Hi, I have some questions about insert performance.
>>
>> I have a single CouchDB 0.9.0 node running on small EC2 instance. I
>> attached a huge EBS volume to it and mounted it where CouchDB's data
>> files are stored. I fired up about ruby scripts running inserts and
>> after a weekend I only have about 30GB/ 12M rows of data... Which
>> seems small. 'top' tells me that my CPU is only about 30% utilized.
>>
>> Any idea what I might be doing wrong? I pretty much just followed
>> these instructions:
>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>
> Hi, Tom. I believe I read somewhere before that the smallest EC2 instances
> have a slower and/or higher-latency connection to EBS, so you might want to
> consider a large instance, or maybe even a high-memory small instance and
> see whether you get better "hardware" performance.
>
> Although strangely, when googling it, the first article I found says that
> their benchmarks found no difference between EBS or even the ephemeral
> filesystem.
>
> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
>
> On the other hand, here is a forum posting and a random benchmark indicating
> that more expensive instances get better throughput:
>
> http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
>
> --
> Jason Smith
> Proven Corporation
> Bangkok, Thailand
> http://www.proven-corporation.com
>
Re: Insert performance
Posted by Jason Smith <jh...@proven-corporation.com>.
Tom Nichols wrote:
> Hi, I have some questions about insert performance.
>
> I have a single CouchDB 0.9.0 node running on small EC2 instance. I
> attached a huge EBS volume to it and mounted it where CouchDB's data
> files are stored. I fired up about ruby scripts running inserts and
> after a weekend I only have about 30GB/ 12M rows of data... Which
> seems small. 'top' tells me that my CPU is only about 30% utilized.
>
> Any idea what I might be doing wrong? I pretty much just followed
> these instructions:
> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
Hi, Tom. I believe I read somewhere before that the smallest EC2
instances have a slower and/or higher-latency connection to EBS, so you
might want to consider a large instance, or maybe even a high-memory
small instance and see whether you get better "hardware" performance.
Although strangely, when googling it, the first article I found says
that their benchmarks found no difference between EBS or even the
ephemeral filesystem.
http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/
On the other hand, here is a forum posting and a random benchmark
indicating that more expensive instances get better throughput:
http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197
http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html
--
Jason Smith
Proven Corporation
Bangkok, Thailand
http://www.proven-corporation.com