How do you watch for node / collector saturation?

[Matthew Rathbone <ma...@foursquare.com>]

Hi all,

We're sending through 10k+ events a second and I need a good way to monitor the nodes & collectors for saturation. 

Does anyone have any good ideas how to do this?

Today I noticed a time when the behavior of the nodes seemed 'bursty', they'd accept like 3000 events, and send them off, but then it would hang for a while (dropping upstream messages), presumably until the downstream node had received everything. This really sucked, so wanted to probe the community on how and what they monitor.

Thanks for any help! 

-- 
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
matthew@foursquare.com (mailto:matthew@foursquare.com) | @rathboma (http://twitter.com/rathboma) | 4sq (http://foursquare.com/rathboma)