Is CouchDB the right tool for this?

[Joe Hillenbrand <jo...@gmail.com>]

I'm building an server monitoring application.

The basic idea is that there will be several nodes that will monitor
metrics about themselves and store them as JSON docs locally. Then
there will be a single host server that will collect these metrics for
the use of a web application.

CouchDB seems like a good options for this because each node could
have their own CouchDB database and then eventually consolidate this
data from all the nodes to the host server where it could be queried.

If the host fails, then the nodes can just keep storing their data
locally until it is ready to be received by the host.

I have a faint recollection from when I first learned CouchDB that it
had these kind of replication mechanics built in (something kind of
like UbuntuOne), but when I look over guide and documentation, I don't
find anything like this. Am I mistaken? Is there a better option?

Thanks,

-Joe

Re: Is CouchDB the right tool for this?

[Simon Metson <si...@googlemail.com>]

Hey Joe,
   I've thought about writing something like this off and on for a while, and I think couch is a good fit (depending on what granularity of data you need to store and associated life tike, frequency and volumes - there are better options for some regions of phase space). It's pretty easy to hook up a logger to write json into couch as well as log to files (I have some demo python code somewhere if you're interested) and visualising live data in couch with something like d3 is a piece of cake.
   One thing you might need to consider is having tiered levels of detail, so that machines have very detailed info about themselves but only propagate up a subset to a rack/region server, which in turn propagates up a subset to some master server. Drilling down through the servers is easy assuming you access from inside the network or have some kind of bastion/proxy/VPN. The replication between the servers can be done using couch's replicator (killer feature) but you might need to be clever about how you filter the data; we've found filter functions to be slow enough to cause problems (though it'll be very dependent on what you filter and the rate that data is inserted). 
  I'd be interested in collaborating on something if that's possible. 
Cheers
Simon



On 24 Dec 2011, at 20:54, Joe Hillenbrand <jo...@gmail.com> wrote:

> I'm building an server monitoring application.
> 
> The basic idea is that there will be several nodes that will monitor
> metrics about themselves and store them as JSON docs locally. Then
> there will be a single host server that will collect these metrics for
> the use of a web application.
> 
> CouchDB seems like a good options for this because each node could
> have their own CouchDB database and then eventually consolidate this
> data from all the nodes to the host server where it could be queried.
> 
> If the host fails, then the nodes can just keep storing their data
> locally until it is ready to be received by the host.
> 
> I have a faint recollection from when I first learned CouchDB that it
> had these kind of replication mechanics built in (something kind of
> like UbuntuOne), but when I look over guide and documentation, I don't
> find anything like this. Am I mistaken? Is there a better option?
> 
> Thanks,
> 
> -Joe