Changes to what is displayed when handler exception occurs.

[Graham Dumpleton <gr...@dscpl.com.au>]

I've just checked in a lot of code related to:

  http://issues.apache.org/jira/browse/MODPYTHON-193
  http://issues.apache.org/jira/browse/MODPYTHON-196

At the same time I have done this, I have played a bit with the format of what
is displayed back to the browser and in the error log files when an unexpected
exception (internal server error) occurs in a handler.

For example, in the browser one might see:

  MOD_PYTHON ERROR

  PID: 8732
  Interpreter: 'application'
  Phase: 'PythonHeaderParserHandler'

  URI: '/~grahamd/handlers/index.html'
  Location: None
  Directory: '/Users/grahamd/public_html/handlers/'
  Filename: '/Users/grahamd/public_html/handlers/index.html'
  PathInfo: ''

  Handler: './_handlers/dispatch.py'

  Traceback (most recent call last):

    File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/mod_python/importer.py", line 1494, in HandlerDispatch
      default=default_handler, arg=req, silent=hlist.silent)

    File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/mod_python/importer.py", line 1199, in _process_target
      result = _execute_target(config, req, object, arg)

    File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/mod_python/importer.py", line 1098, in _execute_target
      result = object(arg)

    File "/Users/grahamd/public_html/handlers/_handlers/dispatch.py", line 10, in headerparserhandler
      xxx

  NameError: global name 'xxx' is not defined

Although not formatted quite the same, the additional information is similarly
output to the Apache error log file.

The idea is to include at least a bit more basic information which may be of
use in tracking down problems, but not to go overboard.

I know we have talked a bit before about providing a means of allowing people
to return custom error pages and I haven't forgotten that. The cleanup of the
code and working out what the report error function should take in the way of
arguments is a first step to seeing how the ability to override it could be
done.

Anyway, comments most welcome. Is there any other basic information that
should perhaps be included in such a page. For example, the name of server
or virtual host etc. I don't want to include stuff which is too obvious though.

BTW, you may note one of the fields being output is Location. If the handler
is used within a Location directive, this should be the matched value of the
Location directive. There hasn't previously been any way of determining this
before, but MODPYTHON-193 now provides a way.  If anyone is using the
Location directive, would be interested in feedback indicating that this is
always yielding the expected result.

Graham

Re: Changes to what is displayed when handler exception occurs.

[Graham Dumpleton <gr...@dscpl.com.au>]

Have a play with code in subversion now. I have checked in some code
which dumps out information about the per request modules cache when
a 500 error is occurring and details being returned to the browser.

This has been useful as it has helped me to work out a way of  
duplicating
a problem I have seen before whereby the traceback actually shows the
wrong line of code. Now I just have to work out why it is occurring.

As an example of what it is dumping out:

MODULE CACHE DETAILS

Generation: 0
FirstAccess: Sat Oct 28 15:06:55 2006

_mp_26e38bd7e93f6e186203698bcf7c2fa4 {
   Module: '/Users/grahamd/public_html/handlers/_handlers/default.py',
   Instance: 1,
   Generation: 1 [NEW],
   LastModified: Sat Oct 28 15:06:44 2006,
   LastImported: Sat Oct 28 15:06:55 2006,
   LastAccessed: Sat Oct 28 15:06:55 2006,
   DirectHits: 1,
   IndirectHits: 0,
   ModulePath: [],
   Children:
}

_mp_39d7952e4a744b6fa6999b60b940f103 {
   Module: '/Users/grahamd/public_html/handlers/_handlers/dispatch.py',
   Instance: 1,
   Generation: 0 [FAIL],
   LastModified: Sat Oct 28 15:06:49 2006,
   LastImported: Thu Jan  1 10:00:00 1970,
   LastAccessed: Thu Jan  1 10:00:00 1970,
   DirectHits: 0,
   IndirectHits: 0,
   ModulePath: ['/Users/grahamd/public_html/handlers/'],
   Children: _mp_26e38bd7e93f6e186203698bcf7c2fa4
}

This was after a fresh Apache restart and I specifically caused a  
failure in
import of handler. As you see though, it is annotated to indicate what
modules were newly loaded/reloaded on that request and also where
a module load actually failed as well. I may even be able to say when it
was a fresh import or a reload as well as can check last imported time.

I gotta go out for a while, so I might say more about it later. If  
you don't
understand the meaning of something, by all means ask as well.

Graham


On 28/10/2006, at 8:10 AM, Graham Dumpleton wrote:

>
> On 27/10/2006, at 11:08 PM, Dan Eloff wrote:
>
>>> I know we have talked a bit before about providing a means of  
>>> allowing people
>>> to return custom error pages and I haven't forgotten that. The  
>>> cleanup of the
>>> code and working out what the report error function should take  
>>> in the way of
>>> arguments is a first step to seeing how the ability to override  
>>> it could be
>>> done.
>>
>> True
>>
>>> Anyway, comments most welcome. Is there any other basic  
>>> information that
>>> should perhaps be included in such a page. For example, the name  
>>> of server
>>> or virtual host etc. I don't want to include stuff which is too  
>>> obvious though.
>>
>> It looks good, although I wouldn't want to write an exception handler
>> taking that many arguments. It would be best to put them all in a
>> special object and pass the request and that object (or attach one to
>> the other).
>>
>> I can think of one thing that might be helpful to know, but I'm not
>> sure how easy it would be to get (or how valuable it would be.) I'll
>> throw it out there so you can think about it.
>>
>> If PythonAutoReload is on, it would be nice to know how old
>> (days:hours:minutes:seconds) the handler module is (measure by  
>> when it
>> was last reloaded), and the same for the module that raised the
>> exception (if different, or as close to it as possible if it occurred
>> in a module not managed by the importer), excepting modules not  
>> loaded
>> by your importer. Additonally, the names of all modules that failed a
>> call to __mp_clone__ (you would have to save that information  
>> ahead of
>> time, probably use a set)
>>
>> At a glance that would tell you:
>> If your module was reloaded since you changed it
>> If your module was reloaded unexpectedly
>> If any modules failed to pass on their state when reloaded
>>
>> I've encoutered a number of situations where one of those was
>> responsible for the problem, or one of those was suspected and had to
>> be eliminated as a cause of the problem. You will have to weigh the
>> difficulty in implementing it versus the estimated worth to the
>> average user, however.
>
> I'll have a think about it. I certainly keep load times and  
> modifications times in
> the cache, plus a sequence number representing the overall version  
> of the
> full set of modules loaded.
>
> Like I provide a function for generating a DOT graph, it may be  
> nice to have
> another helper function which dumps out a lot of other information  
> out about the
> cache. Ie., all modules loaded, load times, modification times etc  
> etc. This
> could be triggered as a handler in its own right, or by way of an  
> option flag
> included in the bottom of a exception page. The only problem with  
> having it
> as a separate handler on its own URL is that its a separate access  
> so have
> to deal with it going to a different Apache child process etc.  
> Still may be
> useful though.
>
> Not sure about __mp_clone__/__mp_purge__ errors. They are at least
> logged now. :-)
>
> Graham

Re: Changes to what is displayed when handler exception occurs.

[Graham Dumpleton <gr...@dscpl.com.au>]

On 27/10/2006, at 11:08 PM, Dan Eloff wrote:

>> I know we have talked a bit before about providing a means of  
>> allowing people
>> to return custom error pages and I haven't forgotten that. The  
>> cleanup of the
>> code and working out what the report error function should take in  
>> the way of
>> arguments is a first step to seeing how the ability to override it  
>> could be
>> done.
>
> True
>
>> Anyway, comments most welcome. Is there any other basic  
>> information that
>> should perhaps be included in such a page. For example, the name  
>> of server
>> or virtual host etc. I don't want to include stuff which is too  
>> obvious though.
>
> It looks good, although I wouldn't want to write an exception handler
> taking that many arguments. It would be best to put them all in a
> special object and pass the request and that object (or attach one to
> the other).
>
> I can think of one thing that might be helpful to know, but I'm not
> sure how easy it would be to get (or how valuable it would be.) I'll
> throw it out there so you can think about it.
>
> If PythonAutoReload is on, it would be nice to know how old
> (days:hours:minutes:seconds) the handler module is (measure by when it
> was last reloaded), and the same for the module that raised the
> exception (if different, or as close to it as possible if it occurred
> in a module not managed by the importer), excepting modules not loaded
> by your importer. Additonally, the names of all modules that failed a
> call to __mp_clone__ (you would have to save that information ahead of
> time, probably use a set)
>
> At a glance that would tell you:
> If your module was reloaded since you changed it
> If your module was reloaded unexpectedly
> If any modules failed to pass on their state when reloaded
>
> I've encoutered a number of situations where one of those was
> responsible for the problem, or one of those was suspected and had to
> be eliminated as a cause of the problem. You will have to weigh the
> difficulty in implementing it versus the estimated worth to the
> average user, however.

I'll have a think about it. I certainly keep load times and  
modifications times in
the cache, plus a sequence number representing the overall version of  
the
full set of modules loaded.

Like I provide a function for generating a DOT graph, it may be nice  
to have
another helper function which dumps out a lot of other information  
out about the
cache. Ie., all modules loaded, load times, modification times etc  
etc. This
could be triggered as a handler in its own right, or by way of an  
option flag
included in the bottom of a exception page. The only problem with  
having it
as a separate handler on its own URL is that its a separate access so  
have
to deal with it going to a different Apache child process etc. Still  
may be
useful though.

Not sure about __mp_clone__/__mp_purge__ errors. They are at least
logged now. :-)

Graham

Re: Changes to what is displayed when handler exception occurs.

[Dan Eloff <da...@gmail.com>]

> I know we have talked a bit before about providing a means of allowing people
> to return custom error pages and I haven't forgotten that. The cleanup of the
> code and working out what the report error function should take in the way of
> arguments is a first step to seeing how the ability to override it could be
> done.

True

> Anyway, comments most welcome. Is there any other basic information that
> should perhaps be included in such a page. For example, the name of server
> or virtual host etc. I don't want to include stuff which is too obvious though.

It looks good, although I wouldn't want to write an exception handler
taking that many arguments. It would be best to put them all in a
special object and pass the request and that object (or attach one to
the other).

I can think of one thing that might be helpful to know, but I'm not
sure how easy it would be to get (or how valuable it would be.) I'll
throw it out there so you can think about it.

If PythonAutoReload is on, it would be nice to know how old
(days:hours:minutes:seconds) the handler module is (measure by when it
was last reloaded), and the same for the module that raised the
exception (if different, or as close to it as possible if it occurred
in a module not managed by the importer), excepting modules not loaded
by your importer. Additonally, the names of all modules that failed a
call to __mp_clone__ (you would have to save that information ahead of
time, probably use a set)

At a glance that would tell you:
If your module was reloaded since you changed it
If your module was reloaded unexpectedly
If any modules failed to pass on their state when reloaded

I've encoutered a number of situations where one of those was
responsible for the problem, or one of those was suspected and had to
be eliminated as a cause of the problem. You will have to weigh the
difficulty in implementing it versus the estimated worth to the
average user, however.

-Dan