You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by bu...@apache.org on 2012/11/08 05:05:38 UTC

[Bug 54117] New: access violation exception in isapi_redirect.dll

https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

          Priority: P2
            Bug ID: 54117
          Assignee: dev@tomcat.apache.org
           Summary: access violation exception in isapi_redirect.dll
          Severity: major
    Classification: Unclassified
          Reporter: werner.peenz@charteredaccountants.com.au
          Hardware: PC
            Status: NEW
           Version: 1.2.37
         Component: isapi
           Product: Tomcat Connectors

Problem: access violation exception in isapi_redirect.dll.

Environment: Two front end IIS 7.5 web servers (Windows 2008 R2 SP1) in DMZ
redirecting to an application server through isapi_redirect. The website on the
web servers is protected by Siteminder 12.0SP2 (an ISAPI filter).

Description:
The worker processes on the web servers crash multiple times a day, often
within minutes of the previous crash, with either "Faulting module name:
isapi_redirect.dll, version: 1.2.37.0" or "Faulting module name: msvcrt.dll,
version: 7.0.7601.17744."

The stack traces for the crash dumps vary slightly:

1.
isapi_redirect!GetExtensionVersion+7bf2
isapi_redirect+74eb
isapi_redirect+d726
isapi_redirect!GetExtensionVersion+3161
isapi_redirect!HttpExtensionProc+541

2.
msvcrt!_ascii_stricmp+3
msvcrt!stricmp+43
isapi_redirect!GetExtensionVersion+e69
isapi_redirect!GetExtensionVersion+10b8
isapi_redirect!GetExtensionVersion+2d6c
isapi_redirect!HttpExtensionProc+541

3.
isapi_redirect+6dea
isapi_redirect+7420
isapi_redirect+d726
isapi_redirect!GetExtensionVersion+3161
isapi_redirect!HttpExtensionProc+541 

The current log_level of info does not provide any usefull information, and the
information I've found indicate that a log_level of trace will severely impact
performance?

Attached are the minidumps and crash dump analysis generated by the "Debug
Diagnostics Tool".

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #3 from Werner <we...@charteredaccountants.com.au> ---
Created attachment 29569
  --> https://issues.apache.org/bugzilla/attachment.cgi?id=29569&action=edit
Configuration files

Attachment contains configuration (files and registry) and previous crash dump
analysed with symbol file. There are no ini files. Will look at capturing full
dumps.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #13 from Mladen Turk <mt...@apache.org> ---
Created attachment 29594
  --> https://issues.apache.org/bugzilla/attachment.cgi?id=29594&action=edit
isapi_redirect-1.2.38-dev-v2-win64

Please try with this version. It should have more reliable header parser

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #7 from Werner <we...@charteredaccountants.com.au> ---
Changed the message buffer sizes to 65536 (we move application to new servers
in the past week - had same issue on previous servers where we did change the
message buffer size to 65536, without success).

Debug log kills performance, had it on briefly to get:

Number of headers is = 4
Header[0] [Set-Cookie] = [JSESSIONID...
Header[1] [Cache-Control] = [No-Cach...
Header[2] [Content-Type] = [text/htm...
Header[3] [Content-Length] = [21837]

Number of headers is = 5
Header[0] [Accept-Ranges] = [bytes]
Header[1] [ETag] = [W/"13968-1352262....
Header[2] [Last-Modified] = [Wed, 07...
Header[3] [Content-Type] = [text/jav...
Header[4] [Content-Length] = [13968]

Number of headers is = 3
Header[0] [Cache-Control] = [No-Cach...
Header[1] [Content-Type] = [text/htm...
Header[2] [Content-Length] = [24471]

Is it worth getting this data from our UAT environment (where we have not had
these crashes)?

Have deployed 1.2.38-dev (does this require log_level=trace - as mentioned it
killed performance?), will attached results after next crash.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #9 from Mladen Turk <mt...@apache.org> ---
So the latest crash dump is from 1.2.38-dev, correct?
... and the screen shots make no difference. Pls attach crash dump file since
it now matches the sources from trunk.

Inspecting the previous crash data I see that 'Number of headers is = 45' but
that data in not in the log files, so either something gets corrupted or crash
happens so fast that log is not written.

I'll create a new build with some current [debug] messages turned to info so
that your performance doesn't suffer and that we get logged only the
interesting part.
However this won't happen before Monday. Willing to give it a shot?

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #14 from Werner <we...@charteredaccountants.com.au> ---
This version was installed today, and there have been no worker process crashes
since. Need another 2 days without issue (as we've gone without a crash for a
day in the past) to declare victory, but looks promising.

(In reply to comment #13)
> Created attachment 29594 [details]
> isapi_redirect-1.2.38-dev-v2-win64
> 
> Please try with this version. It should have more reliable header parser

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

Mladen Turk <mt...@apache.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #29571|0                           |1
        is obsolete|                            |

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #10 from Werner <we...@charteredaccountants.com.au> ---
Created attachment 29579
  --> https://issues.apache.org/bugzilla/attachment.cgi?id=29579&action=edit
Crash 20121109 1.2.38.dev

Previous upload was still for 1.2.37.0. This dump+analysis attachment is for
1.2.38 dev.
Looks like nothing is written to the logs related to crash, only starting and
initialized messages when the process starts again.
Yes, will give another build with debug messages a shot.

In answer to a previous question, we're connecting to Tomcat 6.0.29.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #4 from Mladen Turk <mt...@apache.org> ---
Great. Thanks. This dump makes a bit more sense.
Actually the best would be to use real debugger which (AFAIK) comes with Visual
Studio, but it seems they can be installed standalone.
http://msdn.microsoft.com/en-us/windows/hardware/gg463009.aspx

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #15 from Werner <we...@charteredaccountants.com.au> ---
We've experienced no crashes since using the 2nd dev version, so this issue
appears to be resolved.

Since this was a targeted dev version, should we upgrade to the general release
of 1.2.38 when it becomes available, and when is that expected to be?

Your assistance to a speedy resolution the past 2 weeks is much appreciated.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #6 from Mladen Turk <mt...@apache.org> ---
Created attachment 29571
  --> https://issues.apache.org/bugzilla/attachment.cgi?id=29571&action=edit
iis 1.2.38-dev

Try with this file, I added a patch that should at least trace the problem for
the given crash dump. It zeroes out the header value pointers in case something
invalid comes in.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #5 from Mladen Turk <mt...@apache.org> ---
With debug log turned on there should a line(s)
... Number of headers is = 45
... Header[number] [name] = [value]


This all looks like either some protocol error or buffer overflow.
To which app server you are connecting (exact version pls.)?

Also could you try with larger message buffer size?
worker.ajp13w.max_packet_size=65536 and packetSize="65536" in server.xml for
AJP connector

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

Mladen Turk <mt...@apache.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 OS|                            |All

--- Comment #2 from Mladen Turk <mt...@apache.org> ---
Could you attach all configuration data, both registry and eventual .ini
file(s)
as well as worker properties and uri mappings?
Please scramble the data but in such way that string lengths remain the same as
in the original in case you have some private data there.

Also there is symbol file (.pdb) for each isapi_redirect we release.
Use the .zip file (x86_64) from 
http://apache.org/dist/tomcat/tomcat-connectors/jk/binaries/windows/symbols/
and put isapi_redirect.pdb from that file along with isapi_redirect.dll.
This should give more detailed crash dump.

You might also check
http://httpd.apache.org/dev/debugging.html
(windows section)

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #11 from Werner <we...@charteredaccountants.com.au> ---
Created attachment 29593
  --> https://issues.apache.org/bugzilla/attachment.cgi?id=29593&action=edit
headers

The attachment shows the 46 header values when the process crashes obtained
from a full crash dump and debugging with Visual Studio using the 1.2.38 pdb,
but 1.2.37 jk_ajp.common.c file. The last of the headers is at address
0x0000000000000000.
Hope this helps.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #1 from Werner <we...@charteredaccountants.com.au> ---
Created attachment 29568
  --> https://issues.apache.org/bugzilla/attachment.cgi?id=29568&action=edit
Minidumps and analysis

THe attachment includes a number of crash analysis reports and one minidump.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #8 from Werner <we...@charteredaccountants.com.au> ---
Created attachment 29574
  --> https://issues.apache.org/bugzilla/attachment.cgi?id=29574&action=edit
Screen shot of Visual Studio debug session

Not sure whether it's usefull but attached screen shots of debug session of
existing crash dumps in Visual Studio.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 54117] access violation exception in isapi_redirect.dll

Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=54117

--- Comment #12 from Mladen Turk <mt...@apache.org> ---
This is huge help, thanks!

Seems your application is faulty, but that shouldn't cause the crash. Let me
try to fix that and I'll attach a new .dll

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org