Hooks for management reporting (was RE:New Hook)

[Ian Holsman <Ia...@cnet.com>]

Guys,

I was wondering if it would be usefull if there was some kind of  'status'  hook
which modules could implement to tell management apps (mod_snmp/mod_status) what is going
on inside of them.

it would return an array name/value pairs, which the management module could convert into 
a fancy table layout on the status page, or a group of SNMP OID's

this would allow a module designer not to mess with mod_status, and maybe a way of getting the
scoreboard code out of mod_status

Re: Hooks for management reporting (was RE:New Hook)

[Bill Stoddard <bi...@wstoddard.com>]

> On Fri, 8 Jun 2001, Greg Stein wrote:
>
> > As Ian pointed out, having mod_status directly read the scoreboard means
> > that we cannot experiment with different scoreboard designs.
>
> i've been saying the same thing since the beginning of MPM.  the analogy
> between mod_status and /bin/ps is really valid.  reading the scoreboard
> directly is like mucking in /dev/kmem, which all modern unixes have
> ditched in favour of /proc.
>
> -dean

I did a bit of work a couple of months back to decouple the portion of the scoreboard that
is used for managing the child processes from the statusinfo.  Never quite got to the
point of figuring out how to collect info for mod_status but this is definitely the right
direction to go in.   I agree with Greg, Dean and Ian.

Bill

Re: Hooks for management reporting (was RE:New Hook)

[dean gaudet <dg...@arctic.org>]

On Fri, 8 Jun 2001, Greg Stein wrote:

> As Ian pointed out, having mod_status directly read the scoreboard means
> that we cannot experiment with different scoreboard designs.

i've been saying the same thing since the beginning of MPM.  the analogy
between mod_status and /bin/ps is really valid.  reading the scoreboard
directly is like mucking in /dev/kmem, which all modern unixes have
ditched in favour of /proc.

-dean

Re: Hooks for management reporting (was RE:New Hook)

[Greg Stein <gs...@lyra.org>]

On Fri, Jun 08, 2001 at 08:32:22AM -0700, rbb@covalent.net wrote:
>...
> > The scoreboard is non-extensible. You may also recall that we were working
> > on a scoreboard design that was name/value pair oriented. Manoj disappeared
> > from httpd development, and the design impetus went with him. We threw out
> > all the partial work a while back and reverted to a 1.3 scoreboard design.
> 
> We threw it out because it was the wrong design.

For the scoreboard, possibly. But my point was that we saw a desire to have
a flexible set of status information.

> > In another note, you mention shifting this from the scoreboard code to the
> > MPM. I don't see that as true. We can have the scoreboard code implement the
> > hook. If/when we ever desire to: we can migrate the hook response from the
> > scoreboard to MPMs. Or individual MPMs can add their own hook.
> 
> Doesn't really matter if the code lives in the MPM or the scoreboard
> itself.  What is being proposed, is having a scoreboard that keeps track
> of the information, a segment of code that converts the scoreboard to an
> internal table, and a segment of code that converts that table to the
> output for the management module.  Just remove the middle piece.

As Ian pointed out, having mod_status directly read the scoreboard means
that we cannot experiment with different scoreboard designs. The scoreboard
implementation can/should be changeable by the MPM. We can't do that because
of the hard-wiring between the MPM/scoreboard/mod_status.

> If you want to make mod_status extensible, there are two options.

I'm not talking about just mod_status. I also suggested the case where we
have a monitoring thread that grabs state and delivers it to a log, out a
custom port, tosses it to another server, ... whatever.

> 1)  Make the scoreboard extensible, literally.  This has been talked about
> for MANY months now.  I started talking about it when I reverted to the
> 1.3 scoreboard, but the shared memory system still isn't in place that
> will allow this to happen.

The scoreboard is related to the MPM functions. I would disagree with this
notion. For my mod_dav status case, it would make no sense to put global
lock database stuff into the scoreboard.

> 2)  Use filters.

Not applicable. Remember: we aren't talking about gathering data in response
to an HTTP request. The data could be gathered for other purposes.

> Modules can allocate their own shared memory, and add
> their data to the bottom of the mod_status output.

This does not provide for a monitoring thread. Or logging the status
information to a file. Or sending the status information to a database.

>...
> The problem is simple in my mind.  The management app is the only thing
> that really knows what it wants to display information about.  The module
> can try to figure out what the management app cares about, but 9 times out
> of 10, it will be wrong.  So, either every module will log too much
> information, or it won't log enough.  If you leave it to the management
> app to figure out what to log and when to log it, you resolve this
> problem.

The management app needs access to the various information that *could* be
logged. That access is provided by Ian's hook. The app gathers the data via
the hook, then decides what to do with what portion.

> > I do see that this can be useful. Consider the SSL certificate cache. How do
> > we get status on that thing? One of these days, I'd like to keep the DAV
> > lock database open (rather than re-open per request) and having status on
> > this global state would be nice, too.
> 
> Both of those are solved by having each module create it's own shared
> memory segment and store the information in there.  To display that
> information, that's a perfect example of why filters are so cool.

Filters won't help us with general status gathering.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

Re: Hooks for management reporting (was RE:New Hook)

[rb...@covalent.net]

> > > Something like ap_get_server_status(char **data, int *data_size);
> > > which calls a server_status hook that modules that care about this
> > > stuff can implement.
> >
> > How is this any different than the scoreboard?  It really sounds to me
> > like we are trying to solve a problem that doesn't exist.  The scoreboard
> > does whbat we need, what is the problem with it?
>
> The scoreboard is non-extensible. You may also recall that we were working
> on a scoreboard design that was name/value pair oriented. Manoj disappeared
> from httpd development, and the design impetus went with him. We threw out
> all the partial work a while back and reverted to a 1.3 scoreboard design.

We threw it out because it was the wrong design.

> In another note, you mention shifting this from the scoreboard code to the
> MPM. I don't see that as true. We can have the scoreboard code implement the
> hook. If/when we ever desire to: we can migrate the hook response from the
> scoreboard to MPMs. Or individual MPMs can add their own hook.

Doesn't really matter if the code lives in the MPM or the scoreboard
itself.  What is being proposed, is having a scoreboard that keeps track
of the information, a segment of code that converts the scoreboard to an
internal table, and a segment of code that converts that table to the
output for the management module.  Just remove the middle piece.

If you want to make mod_status extensible, there are two options.

1)  Make the scoreboard extensible, literally.  This has been talked about
for MANY months now.  I started talking about it when I reverted to the
1.3 scoreboard, but the shared memory system still isn't in place that
will allow this to happen.

2)  Use filters.  Modules can allocate their own shared memory, and add
their data to the bottom of the mod_status output.  Other management apps
do most of their work during the logging phase, and they will by
definition require modifications to those modules.  Those modifications
will be one of two things.  In the current model, logging directly to the
management module's API.  In the proposed hook model, putting the right
information in the table.

The problem is simple in my mind.  The management app is the only thing
that really knows what it wants to display information about.  The module
can try to figure out what the management app cares about, but 9 times out
of 10, it will be wrong.  So, either every module will log too much
information, or it won't log enough.  If you leave it to the management
app to figure out what to log and when to log it, you resolve this
problem.

> I do see that this can be useful. Consider the SSL certificate cache. How do
> we get status on that thing? One of these days, I'd like to keep the DAV
> lock database open (rather than re-open per request) and having status on
> this global state would be nice, too.

Both of those are solved by having each module create it's own shared
memory segment and store the information in there.  To display that
information, that's a perfect example of why filters are so cool.

Ryan
_______________________________________________________________________________
Ryan Bloom                        	rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------

Re: Hooks for management reporting (was RE:New Hook)

[Greg Stein <gs...@lyra.org>]

On Thu, Jun 07, 2001 at 08:44:27PM -0700, rbb@covalent.net wrote:
> On Thu, 7 Jun 2001, Greg Marr wrote:
> > At 02:50 PM 06/07/2001, Greg Stein wrote:
> > >On Thu, Jun 07, 2001 at 09:43:24AM -0700, Ian Holsman wrote:
> > >>I was wondering if it would be usefull if there was some kind of
> > >>'status' hook which modules could implement to tell management apps
> > >>(mod_snmp/mod_status) what is going on inside of them.
> > >>
> > >>it would return an array name/value pairs, which the management
> > >>module could convert into a fancy table layout on the status page,
> > >>or a group of SNMP OID's this would allow a module designer not to
> > >>mess with mod_status, and maybe a way of getting the scoreboard
> > >>code out of mod_status
> > >
> > >The current design is for mod_status to *pull* the information when
> > >it is needed. Considering that pulling is *much* less frequent than
> > >status changes, this is much more efficient for the system.  (a
> > >status hook is a *push*)n
> >
> > It could be implemented as something that mod_status calls when it
> > needs the status.  If it's done in the core, then any management app
> > could use it.

Ah! I saw the hook as a "push status" rather than what the external code
uses to pull it from whatever pieces maintain status.

> > Something like ap_get_server_status(char **data, int *data_size);
> > which calls a server_status hook that modules that care about this
> > stuff can implement.
> 
> How is this any different than the scoreboard?  It really sounds to me
> like we are trying to solve a problem that doesn't exist.  The scoreboard
> does whbat we need, what is the problem with it?

The scoreboard is non-extensible. You may also recall that we were working
on a scoreboard design that was name/value pair oriented. Manoj disappeared
from httpd development, and the design impetus went with him. We threw out
all the partial work a while back and reverted to a 1.3 scoreboard design.

In another note, you mention shifting this from the scoreboard code to the
MPM. I don't see that as true. We can have the scoreboard code implement the
hook. If/when we ever desire to: we can migrate the hook response from the
scoreboard to MPMs. Or individual MPMs can add their own hook.

I do see that this can be useful. Consider the SSL certificate cache. How do
we get status on that thing? One of these days, I'd like to keep the DAV
lock database open (rather than re-open per request) and having status on
this global state would be nice, too.

I could also see this hook as being for *config info* rather than just
runtime state. Currently, mod_info reaches into structures that it probably
shouldn't know anything about. The hook would isolate the two pieces. In
this future scenario, the core would expose the config tree bits via the
info/state hook.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

Re: Hooks for management reporting (was RE:New Hook)

[rb...@covalent.net]

On Thu, 7 Jun 2001, Greg Marr wrote:

> At 02:50 PM 06/07/2001, Greg Stein wrote:
> >On Thu, Jun 07, 2001 at 09:43:24AM -0700, Ian Holsman wrote:
> >>I was wondering if it would be usefull if there was some kind of
> >>'status' hook which modules could implement to tell management apps
> >>(mod_snmp/mod_status) what is going on inside of them.
> >>
> >>it would return an array name/value pairs, which the management
> >>module could convert into a fancy table layout on the status page,
> >>or a group of SNMP OID's this would allow a module designer not to
> >>mess with mod_status, and maybe a way of getting the scoreboard
> >>code out of mod_status
> >
> >The current design is for mod_status to *pull* the information when
> >it is needed. Considering that pulling is *much* less frequent than
> >status changes, this is much more efficient for the system.  (a
> >status hook is a *push*)n
>
> It could be implemented as something that mod_status calls when it
> needs the status.  If it's done in the core, then any management app
> could use it.
>
> Something like ap_get_server_status(char **data, int *data_size);
> which calls a server_status hook that modules that care about this
> stuff can implement.

How is this any different than the scoreboard?  It really sounds to me
like we are trying to solve a problem that doesn't exist.  The scoreboard
does whbat we need, what is the problem with it?

Ryan

_______________________________________________________________________________
Ryan Bloom                        	rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------

Re: Hooks for management reporting (was RE:New Hook)

[Greg Marr <gr...@alum.wpi.edu>]

At 02:50 PM 06/07/2001, Greg Stein wrote:
>On Thu, Jun 07, 2001 at 09:43:24AM -0700, Ian Holsman wrote:
>>I was wondering if it would be usefull if there was some kind of 
>>'status' hook which modules could implement to tell management apps 
>>(mod_snmp/mod_status) what is going on inside of them.
>>
>>it would return an array name/value pairs, which the management 
>>module could convert into a fancy table layout on the status page, 
>>or a group of SNMP OID's this would allow a module designer not to 
>>mess with mod_status, and maybe a way of getting the scoreboard 
>>code out of mod_status
>
>The current design is for mod_status to *pull* the information when 
>it is needed. Considering that pulling is *much* less frequent than 
>status changes, this is much more efficient for the system.  (a 
>status hook is a *push*)n

It could be implemented as something that mod_status calls when it 
needs the status.  If it's done in the core, then any management app 
could use it.

Something like ap_get_server_status(char **data, int *data_size); 
which calls a server_status hook that modules that care about this 
stuff can implement.

-- 
Greg Marr
gregm@alum.wpi.edu
"We thought you were dead."
"I was, but I'm better now." - Sheridan, "The Summoning"

Re: Hooks for management reporting (was RE:New Hook)

[Greg Stein <gs...@lyra.org>]

On Thu, Jun 07, 2001 at 09:43:24AM -0700, Ian Holsman wrote:
> Guys,
> 
> I was wondering if it would be usefull if there was some kind of  'status'  hook
> which modules could implement to tell management apps (mod_snmp/mod_status) what is going
> on inside of them.
> 
> it would return an array name/value pairs, which the management module could convert into 
> a fancy table layout on the status page, or a group of SNMP OID's
> 
> this would allow a module designer not to mess with mod_status, and maybe a way of getting the
> scoreboard code out of mod_status

The current design is for mod_status to *pull* the information when it is
needed. Considering that pulling is *much* less frequent than status
changes, this is much more efficient for the system.

(a status hook is a *push*)n

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

Re: Hooks for management reporting (was RE:New Hook)

[rb...@covalent.net]

On Thu, 7 Jun 2001, Ian Holsman wrote:

> Guys,
>
> I was wondering if it would be usefull if there was some kind of  'status'  hook
> which modules could implement to tell management apps (mod_snmp/mod_status) what is going
> on inside of them.
>
> it would return an array name/value pairs, which the management module could convert into
> a fancy table layout on the status page, or a group of SNMP OID's
>
> this would allow a module designer not to mess with mod_status, and maybe a way of getting the
> scoreboard code out of mod_status

I don't see how this would be implemented.  All I see that this could do,
is move the scoreboard code from mod_status, and put it into the MPM.  I
would personally be against doing that.

Ryan

_______________________________________________________________________________
Ryan Bloom                        	rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------