You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Jason Webb <jw...@inovem.com> on 2002/08/22 12:07:44 UTC
[PATCH]Simple stats and monitoring for James
Please find enclosed my attempt at a simple stats and monitoring package
for James.
There are two main ways for getting the stats information out at the
moment:
Via. the remote manager,
Or via a stats log file (mainly used for MRTG style monitoring)
If I've missed anything out, or screwed something up, just email me
directly, rather than clog up the list with stuff that most people
couldn't care about.
I've attached a zip file with the patch files as well as the new ones.
Patch files are zip'd flat, the new files in the their correct
directories.
-- Jason
Re: [PATCH]Simple stats and monitoring for James
Posted by Harmeet Bedi <ha...@kodemuse.com>.
Would it be better/possible to leverage logger interface for stats ? Would
it be possible for the API to take a stats object rather than separate
parameters. One advantage is that string construction is done only if stats
logging is enabled.
Would it be possible to use int instead of float. The advantage is that int
based counters can be shared across threads without the need to synchronize.
Instead of
import ...*;
It is prefered to have the fully qualified class name.
It is painful, but here is a way to reduce the pain:
jad -p -pi1000 <classname> | grep "^import"
where jad is java decompiler. Excellent tool.
> If I've missed anything out, or screwed something up, just email me
> directly, rather than clog up the list with stuff that most people
> couldn't care about.
Looks like a very necessary thing for a system to have. I am sending email
to the list because I thought some of the issues like jad, api etc. may
interest others.
Harmeet
----- Original Message -----
From: "Jason Webb" <jw...@inovem.com>
Subject: [PATCH]Simple stats and monitoring for James
> Please find enclosed my attempt at a simple stats and monitoring package
> for James.
> There are two main ways for getting the stats information out at the
> moment:
> Via. the remote manager,
> Or via a stats log file (mainly used for MRTG style monitoring)
>
> If I've missed anything out, or screwed something up, just email me
> directly, rather than clog up the list with stuff that most people
> couldn't care about.
>
> I've attached a zip file with the patch files as well as the new ones.
> Patch files are zip'd flat, the new files in the their correct
> directories.
>
> -- Jason
>
----------------------------------------------------------------------------
----
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by "Noel J. Bergman" <no...@devtech.com>.
Jason,
> I was trying to write a high-level implementation so I didn't have to
> care about the underlying storage, but I think on balance you're
> correct. The performance would suck :-(
High-level doesn't mean building on (too) low-level primitives. You want
the ability to allow to subsystems that can do things efficiently. You need
to be a framework for them to use, not the otherway around.
For example, lets say that you want a report of who has messages waiting,
how many, and how big are the mailboxes? You can either do a lot of work,
or you can allow a JDBC repository to execute:
select count(*) as MessagesWaiting,
sum(length(message_body)) as MailBoxSize,
repository_name as User
from inbox
group by repository_name;
Mind you, we should distinguish between reports such as this, and data
gathering, e.g., a connection counter. But this does illustrate why you
would want each subsystem to play an active role, and register its
capabilities. Are you familar with the /proc system in linux?
I'll respond to everything else later when I've some time.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by Danny Angus <da...@apache.org>.
> I take it from your comments below that a subsystem (POP3, SMTP etc)
> would be responsible for doing it's own stats logging?
Yes, don't forget that the subsystems are seperately usable, you can leave
out protocols you don't want, and activate those you do.
You might want to turn off POP3 and NNTP and just use SMTP, or add IMAP.
> Would this model cope with such things as collecting the number of
> emails in a users account or counting the number of messages in the
> spool? These actions would often be generated on a time interval, rather
> than continously generated (I want to know how many emails an hour I
> delivered, for example). I supposed this could be the job of a stats
> timer that would gather such information and update the stats registry
> as appropriate.
If you did it right it would :-) perhaps you should have seperate stats
logging into a stats repository by each subsystem "block" and a new block to
collate those simple stats into sensible summary information, which, if you
design your data well, could span more than one blocks stats, although I
cannot see an immediate use for this.
> The other thing that any implementor of the stats interface would need
> to be aware of is the volume of data being stored. Such information
> needs to be regularly written to disk, otherwise the VM will simply blow
> up in your face. The Qmail system I worked on was chatty, but all it did
> was write to disk based log files straight away, and leave any stats
> analysis to other near-line programs. I would not want to see mail
> processing hammered due to logging taking excessive CPU,memory or I/O.
> Counters are fine in memory, but more heavy information (email size, to
> , from etc) needs to written to disk. I might be belabouring the point
> here, but there also needs to be someway of getting the information out
> of James into a useful format.
Of course, and James is well provided with examples of persistent storage,
after all thats a big part of what it does, passing mail and account
information into and out of several persistant storage repositories, many of
which can be quickly change in their implementation by thanks to the
intelligent use of Avalon's many parts (I'm not even going to say which
bits, I get it wrong too often :)
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by Jason Webb <jw...@inovem.com>.
I take it from your comments below that a subsystem (POP3, SMTP etc)
would be responsible for doing it's own stats logging?
Would this model cope with such things as collecting the number of
emails in a users account or counting the number of messages in the
spool? These actions would often be generated on a time interval, rather
than continously generated (I want to know how many emails an hour I
delivered, for example). I supposed this could be the job of a stats
timer that would gather such information and update the stats registry
as appropriate.
The other thing that any implementor of the stats interface would need
to be aware of is the volume of data being stored. Such information
needs to be regularly written to disk, otherwise the VM will simply blow
up in your face. The Qmail system I worked on was chatty, but all it did
was write to disk based log files straight away, and leave any stats
analysis to other near-line programs. I would not want to see mail
processing hammered due to logging taking excessive CPU,memory or I/O.
Counters are fine in memory, but more heavy information (email size, to
, from etc) needs to written to disk. I might be belabouring the point
here, but there also needs to be someway of getting the information out
of James into a useful format.
But on the whole I like the idea.
-- Jason
> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: 23 August 2002 20:59
> To: James Developers List
> Subject: RE: [PATCH]Simple stats and monitoring for James
>
>
> Jason,
>
> The implementation of mailInboundQueue() is a bit off. If
> such information is desired, the behavior should be delegated
> to the actual user repository, which can optimize it. For
> example, with a JDBC repository, the value is "select
> count(*) from ...", rather than selecting and iterating over
> a row set.
>
I was trying to write a high-level implementation so I didn't have to
care
about the underlying storage, but I think on balance you're correct.
The performance would suck :-(
> You should be prepared to see the other delegated statistics.
> For example, I expect that the user repository data model
> will eventually support per user quota / billing related statistics.
>
> Based upon this, I'd like to propose that now that you've
> provided a Q&D PoC to show that a stats service is A Good
> Thing, that we take a step back and look at the model. I'd
> like to suggest that a Statistics Server maintain a top level
> Registry of Statistic objects. There can be a basic set of
> common Statistic classes, such as those typically used for
> counting some item (incident or traffic count), and other
> items. A basic Statistic interface might be something like:
>
> public interface Statistic
> {
> /* it might be nice to know which bucket this is
> without having to
> * keep track of the key separately. */
> String getKey();
>
> /* Describe this for logger/admin console */
> String getDescription();
>
> /* String representation of the contents of the bucket */
> String toString();
> }
>
> You could also add direct support for rendering to the
> various output interfaces.
>
> A statistics registry might be something like:
>
> public interface StatisticRegistry
> {
> // Existing stat object for String; else null
> Statistic getInstance(String);
>
> // Existing stat object for String; else install
> Class.newInstance();
> Statistic getInstance(String, Class);
>
> // Existing stat object for String; else install parameter
> Statistic getInstance(String, Object);
> }
>
> Objects that contribute to a given statistic will
> acquire/register an appropriate Statistic instance. The
> instance may be one of several common types, such as a
> counter, or something custom. Those accumulating or
> analyzing the statistic will further need to know what type
> of Statistic it is so that they can mutate it; others may
> only care about rendering it. I can think of several common
> types off-hand, and a few custom ones (e.g., queue length,
> which is a read-only real-time value that could have current,
> max and avg components).
>
> This is a quick strawman to promote discussion.
>
> --- Noel
>
> -----Original Message-----
> From: Jason Webb [mailto:jw@inovem.com]
> Sent: Friday, August 23, 2002 6:51
> To: 'James Developers List'
> Subject: RE: [PATCH]Simple stats and monitoring for James
>
> I've fixed the embarrassing problems you mentioned...
>
> I will extended the stats package to allow any subsystem (NNTP, POP3
> etc) to log their own stats.
>
> This will be of the form:
>
> public void logSubSystemStats(String subsystemName,String
> key,float value);
>
> These stats will be written out a ONE log file (at the
> moment) as and when they are received. I will use another
> thread to do a "write-behind" style buffer to ensure the main
> server is not crippled when stats logging is active.
>
> The type of items that would use this style could include:
> Connection counts for all sub-systems NNTP message posts (key
> is groupname?) User account mail counts
>
> However, I will keep the mail send & receive logging, as the
> mail objects contain enough information to make the above
> method useless. The bulk of the logging we want is the
> "who-sent-what-to-who" type thing. They will be logged to a
> file that contains just mail tranmission stats.
>
> -- Jason
>
>
> --
> To unsubscribe, e-mail:
> <mailto:james-dev-> unsubscribe@jakarta.apache.org>
> For
> additional commands,
> e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by "Noel J. Bergman" <no...@devtech.com>.
Jason,
The implementation of mailInboundQueue() is a bit off. If such information
is desired, the behavior should be delegated to the actual user repository,
which can optimize it. For example, with a JDBC repository, the value is
"select count(*) from ...", rather than selecting and iterating over a row
set.
You should be prepared to see the other delegated statistics. For example,
I expect that the user repository data model will eventually support per
user quota / billing related statistics.
Based upon this, I'd like to propose that now that you've provided a Q&D PoC
to show that a stats service is A Good Thing, that we take a step back and
look at the model. I'd like to suggest that a Statistics Server maintain a
top level Registry of Statistic objects. There can be a basic set of common
Statistic classes, such as those typically used for counting some item
(incident or traffic count), and other items. A basic Statistic interface
might be something like:
public interface Statistic
{
/* it might be nice to know which bucket this is without having to
* keep track of the key separately. */
String getKey();
/* Describe this for logger/admin console */
String getDescription();
/* String representation of the contents of the bucket */
String toString();
}
You could also add direct support for rendering to the various output
interfaces.
A statistics registry might be something like:
public interface StatisticRegistry
{
// Existing stat object for String; else null
Statistic getInstance(String);
// Existing stat object for String; else install Class.newInstance();
Statistic getInstance(String, Class);
// Existing stat object for String; else install parameter
Statistic getInstance(String, Object);
}
Objects that contribute to a given statistic will acquire/register an
appropriate Statistic instance. The instance may be one of several common
types, such as a counter, or something custom. Those accumulating or
analyzing the statistic will further need to know what type of Statistic it
is so that they can mutate it; others may only care about rendering it. I
can think of several common types off-hand, and a few custom ones (e.g.,
queue length, which is a read-only real-time value that could have current,
max and avg components).
This is a quick strawman to promote discussion.
--- Noel
-----Original Message-----
From: Jason Webb [mailto:jw@inovem.com]
Sent: Friday, August 23, 2002 6:51
To: 'James Developers List'
Subject: RE: [PATCH]Simple stats and monitoring for James
I've fixed the embarrassing problems you mentioned...
I will extended the stats package to allow any subsystem (NNTP, POP3
etc) to log their own stats.
This will be of the form:
public void logSubSystemStats(String subsystemName,String key,float
value);
These stats will be written out a ONE log file (at the moment) as and
when they are received. I will use another thread to do a "write-behind"
style buffer to ensure the main server is not crippled when stats
logging is active.
The type of items that would use this style could include:
Connection counts for all sub-systems
NNTP message posts (key is groupname?)
User account mail counts
However, I will keep the mail send & receive logging, as the mail
objects contain enough information to make the above method useless. The
bulk of the logging we want is the "who-sent-what-to-who" type thing.
They will be logged to a file that contains just mail tranmission stats.
-- Jason
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by Jason Webb <jw...@inovem.com>.
I've fixed the embarrassing problems you mentioned...
I will extended the stats package to allow any subsystem (NNTP, POP3
etc) to log their own stats.
This will be of the form:
public void logSubSystemStats(String subsystemName,String key,float
value);
These stats will be written out a ONE log file (at the moment) as and
when they are received. I will use another thread to do a "write-behind"
style buffer to ensure the main server is not crippled when stats
logging is active.
The type of items that would use this style could include:
Connection counts for all sub-systems
NNTP message posts (key is groupname?)
User account mail counts
However, I will keep the mail send & receive logging, as the mail
objects contain enough information to make the above method useless. The
bulk of the logging we want is the "who-sent-what-to-who" type thing.
They will be logged to a file that contains just mail tranmission stats.
-- Jason
> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: 23 August 2002 01:26
> To: James Developers List
> Cc: Jason Webb
> Subject: RE: [PATCH]Simple stats and monitoring for James
>
>
> Jason,
>
> 1) You created StatsRepository.ROLE and then replicated the
> string literal rather than use the class constant.
>
> 2) "Foo" is probably not a good default value.
>
> 3) You used System.out.println instead of the logging mechanism.
>
> I didn't look much further than that for this go 'round. I
> figure that you'll have a revised version soon enough.
>
> Rather than hardcode the stats the way you did in the
> interface, I'd rather see each one as a keyed property so
> that other parts of James can add their own stat buckets.
>
> --- Noel
>
> -----Original Message-----
> From: Jason Webb [mailto:jw@inovem.com]
> Sent: Thursday, August 22, 2002 6:08
> To: 'James Developers List'
> Subject: [PATCH]Simple stats and monitoring for James
>
>
> Please find enclosed my attempt at a simple stats and
> monitoring package for James. There are two main ways for
> getting the stats information out at the
> moment:
> Via. the remote manager,
> Or via a stats log file (mainly used for MRTG style monitoring)
>
> If I've missed anything out, or screwed something up, just
> email me directly, rather than clog up the list with stuff
> that most people couldn't care about.
>
> I've attached a zip file with the patch files as well as the
> new ones. Patch files are zip'd flat, the new files in the
> their correct directories.
>
> -- Jason
>
>
> --
> To unsubscribe, e-mail:
> <mailto:james-dev-> unsubscribe@jakarta.apache.org>
> For
> additional commands,
> e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by "Noel J. Bergman" <no...@devtech.com>.
Jason,
1) You created StatsRepository.ROLE and then replicated the string literal
rather than use the class constant.
2) "Foo" is probably not a good default value.
3) You used System.out.println instead of the logging mechanism.
I didn't look much further than that for this go 'round. I figure that
you'll have a revised version soon enough.
Rather than hardcode the stats the way you did in the interface, I'd rather
see each one as a keyed property so that other parts of James can add their
own stat buckets.
--- Noel
-----Original Message-----
From: Jason Webb [mailto:jw@inovem.com]
Sent: Thursday, August 22, 2002 6:08
To: 'James Developers List'
Subject: [PATCH]Simple stats and monitoring for James
Please find enclosed my attempt at a simple stats and monitoring package
for James.
There are two main ways for getting the stats information out at the
moment:
Via. the remote manager,
Or via a stats log file (mainly used for MRTG style monitoring)
If I've missed anything out, or screwed something up, just email me
directly, rather than clog up the list with stuff that most people
couldn't care about.
I've attached a zip file with the patch files as well as the new ones.
Patch files are zip'd flat, the new files in the their correct
directories.
-- Jason
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>