You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Nikolai Teofilov <n....@gmail.com> on 2011/03/14 13:41:55 UTC
Monitoring system based on CouchDB?
Hi all,
I need a help on a concept for a monitoring system based on CouchDB. I have master server with a database "status" and several, around 20 nodes machines (the number of machines will grow) performing some logging on a local instances of CouchDB. Each local machine (node) has a database "status" as well and logs realtime data in a single document named as the computer. The idea is to monitor all the data from all node machines in a single database on the master server. The final result should be a database that is a collection of documents named as the computer names of all node machines and each document reflect the current status of the node. Then it will be easy to have long-poll handler for monitoring and rendering the changes of all machines on the master server as a status panel...
I made a local db "status" on each node machine and started a continuous replication to the master server "status" database. The problem is that some of the logging processes on the node machines are working over hours and can involve logs each second. This will reflect on the size of the master database. How to handle this correctly? I can compact the master target database from time to time remotely. Is this a good idea or is there any other way to make continuous replication with kind of auto compact feature? Would a compacting from time to time will affect somehow the other continuous replications?
The second problem I have is once I delete (recreate) the source local "status" database the old instance of the continuous replication does not work for the new created database. I have a kind of workaround be canceling the old continuous replication and triggered it again. This work well but I am not sure if I can expect some other problems somehow.
Any suggestions or thoughts?
Thanks
Nikolai
Re: Monitoring system based on CouchDB?
Posted by Nikolai Teofilov <n....@gmail.com>.
Bob
Thanks, It is useful for a different approach in my system, where I need triggered pull replication from the master server. This is something like measurement system from multiple machines. Where each node can be "plugged" to the system. I did several test on my push approach and I am very happy about the results. I was able to run multiple nodes (10 for now) with push continuous replication to the master. All nodes has a local CouchDB server with a database with single document as a process log for this machine being replicated continuously on the master by push replication from the local CouchDB server. I furthermore was able to trigger after each big change a compaction of the master "status" db in a separated thread from the local machine. It work like a magic! No conflicts no problems at all. The size of the master database was always reasonable. I also tried the long-poll handler on the master database and it works very well. I put a simple couchapp to render the changes in a table form and it works as a real monitor over the whole cluster.
Nikolai
On 14.03.2011, at 22:12, Robert Johnson wrote:
> Nikolai
>
> You would have a rapidly growing database on your nodes as well because each document update on each node would generate a new version.
>
> I offer this:-
>
> Have a document in your master database with the URLs of the nodes and have a cron job to read that document and initiate a one time replication with each node and compress your "master" database (you can use a shell script and curl to do this). Now, you will need to either multi thread the replication cron script or in some way test that the host is available before trying the replication because if the remote host is not available your script will stall for a good few seconds when trying to contact the remote host and if you have a lot of hosts and a short(ish) cron cycle you can get cron job collision.
> Have the status collection job on each node compress the database on each update.
>
> Bob
>
> On 14 Mar 2011, at 12:41, Nikolai Teofilov wrote:
>
>> Hi all,
>>
>> I need a help on a concept for a monitoring system based on CouchDB. I have master server with a database "status" and several, around 20 nodes machines (the number of machines will grow) performing some logging on a local instances of CouchDB. Each local machine (node) has a database "status" as well and logs realtime data in a single document named as the computer. The idea is to monitor all the data from all node machines in a single database on the master server. The final result should be a database that is a collection of documents named as the computer names of all node machines and each document reflect the current status of the node. Then it will be easy to have long-poll handler for monitoring and rendering the changes of all machines on the master server as a status panel...
>>
>> I made a local db "status" on each node machine and started a continuous replication to the master server "status" database. The problem is that some of the logging processes on the node machines are working over hours and can involve logs each second. This will reflect on the size of the master database. How to handle this correctly? I can compact the master target database from time to time remotely. Is this a good idea or is there any other way to make continuous replication with kind of auto compact feature? Would a compacting from time to time will affect somehow the other continuous replications?
>>
>> The second problem I have is once I delete (recreate) the source local "status" database the old instance of the continuous replication does not work for the new created database. I have a kind of workaround be canceling the old continuous replication and triggered it again. This work well but I am not sure if I can expect some other problems somehow.
>>
>> Any suggestions or thoughts?
>>
>> Thanks
>> Nikolai
>
Re: Monitoring system based on CouchDB?
Posted by Robert Johnson <ro...@rowanshire.net>.
Nikolai
You would have a rapidly growing database on your nodes as well because each document update on each node would generate a new version.
I offer this:-
Have a document in your master database with the URLs of the nodes and have a cron job to read that document and initiate a one time replication with each node and compress your "master" database (you can use a shell script and curl to do this). Now, you will need to either multi thread the replication cron script or in some way test that the host is available before trying the replication because if the remote host is not available your script will stall for a good few seconds when trying to contact the remote host and if you have a lot of hosts and a short(ish) cron cycle you can get cron job collision.
Have the status collection job on each node compress the database on each update.
Bob
On 14 Mar 2011, at 12:41, Nikolai Teofilov wrote:
> Hi all,
>
> I need a help on a concept for a monitoring system based on CouchDB. I have master server with a database "status" and several, around 20 nodes machines (the number of machines will grow) performing some logging on a local instances of CouchDB. Each local machine (node) has a database "status" as well and logs realtime data in a single document named as the computer. The idea is to monitor all the data from all node machines in a single database on the master server. The final result should be a database that is a collection of documents named as the computer names of all node machines and each document reflect the current status of the node. Then it will be easy to have long-poll handler for monitoring and rendering the changes of all machines on the master server as a status panel...
>
> I made a local db "status" on each node machine and started a continuous replication to the master server "status" database. The problem is that some of the logging processes on the node machines are working over hours and can involve logs each second. This will reflect on the size of the master database. How to handle this correctly? I can compact the master target database from time to time remotely. Is this a good idea or is there any other way to make continuous replication with kind of auto compact feature? Would a compacting from time to time will affect somehow the other continuous replications?
>
> The second problem I have is once I delete (recreate) the source local "status" database the old instance of the continuous replication does not work for the new created database. I have a kind of workaround be canceling the old continuous replication and triggered it again. This work well but I am not sure if I can expect some other problems somehow.
>
> Any suggestions or thoughts?
>
> Thanks
> Nikolai