You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mark Eggers <it...@yahoo.com> on 2012/08/24 22:48:34 UTC
getRealPath returns null
Folks,
This is not strictly a Tomcat problem, but since the application is
running under Tomcat I thought I would ask.
Environment
-----------
Tomcat: 6.0.20
JRE: 1.6.0_16
OS: some UNIX variant
I'm a little light on the environment, since it's not mine. Yes, this is
old. No, I don't have any leverage to get them to update.
Background
----------
An application makes use of a PDF library which generates PDFs from JSP
pages. One of these pages includes a dynamically generated graph.
This library cannot use streaming data to include the graph. The only
two options are:
1. Write the graph within the web application and use relative paths
2. Write the graph somewhere else and:
a. use a fully qualified URL point to a servlet
b. the servlet retrieves the graph by name
Ideally, we write the JPG file to java.io.tmpdir, then reference a
servlet in the JSP to fetch the graph and insert it into the page.
Considerations
--------------
Unfortunately, in certain environments we cannot use solution 2. One of
these environments involves a NAT and hairpinning. In those cases, we
use solution 1.
We use getRealPath in a listener, tag on a directory to write images,
set the path in a singleton, and we're done. We can then use a relative
directory in the JSP to get the image and generate the PDF.
If for any reason getRealPath returns a null, we fall back to using
java.io.tmpdir.
Problem
-------
I ran into a situation where getRealPath with the above environment
returns a null, even when the WAR file is exploded.
This means that we fall back to java.io.tmpdir, and run into the
hairpinning problem.
Question
--------
What other reasons could there be for getRealPath to return a null
rather than the path where the application is run?
Some other Info
---------------
They are not running under a security manager. ps shows no security
manager arguments, and at any rate the application would throw a bunch
of other security violations and fail to start (a few getReourceAsStream
issues in listeners).
I noticed that the server appears to be running from an NFS mount.
However, the exploded WAR file directory is marked as 755, and the
ownership is correct.
About the only thing that comes to mind is NSF3 / NSF4 ACLs. I won't
know more about the server environment until Monday.
If anyone has ideas, I'd be happy to hear about them.
. . . just my (confused) two cents
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Mark Eggers <it...@yahoo.com>.
On 8/25/2012 11:42 AM, Terence M. Bandoian wrote:
> On 8/25/2012 7:01 AM, Konstantin Kolinko wrote:
>> 2012/8/25 Mark Eggers <it...@yahoo.com>:
>>> On 8/24/2012 2:55 PM, Konstantin Kolinko wrote:
>>>> 2012/8/25 Mark Eggers <it...@yahoo.com>:
>>>>> (...)
>>>>>
>>>>> An application makes use of a PDF library which generates PDFs from
>>>>> JSP
>>>>> pages. One of these pages includes a dynamically generated graph.
>>>>>
>>>>> This library cannot use streaming data to include the graph.
>>>>>
>>>>> (...)
>>>>>
>>>> ============
>>>> 3. What is your hairpinning problem?
>>>>
>>> Hairpinning is when a system is sitting behind a NAT, and it attempts to
>>> send itself a packet to the public IP address.
>>>
>>> +----------------<
>>> | NAT <----------------> Server
>>> +---------------->
>>> public IP private IP
>>> ww.xx.yy.zz aa.bb.cc.dd
>>>
>>> In this environment it occurs with a host name. The server sends a
>>> packet to
>>> itself using the publicly accessible name, which resolves to the
>>> external IP
>>> address (ww.xx.yy.zz), which then gets sent right back to the server.
>>>
>>> Unless you specifically configure a NAT device to allow this, the packet
>>> will be dropped.
>>>
>>>
>>>> Is it because you use "a fully qualified URL point to a servlet" and
>>>> you cannot build that URL correctly?
>>>>
>>>> Do you really need an absolute URL? You mean that you cannot call your
>>>> servlet otherwise?
>>>
>>> This is correct. The library in use will only take relative URLs, or
>>> fully
>>> qualified URLs (including protocol, port, and host name).
>>>
>>> Relative URLs means I write back into the web application, which is
>>> where
>>> getRealPath() comes into play.
>>>
>>> I cannot use absolute URLs (/context/servlet-name?imagename=foo).
>>> This gets
>>> interpreted by the library as an absolute file name (and no, using
>>> /java.io.tmpdir/foo does not work).
>>>
>>> I would love to rewrite my URL to be
>>>
>>> http://localhost:80/context/servlet-name?imagename=foo
>>>
>>> That will only work if there are no virtual hosts in any front end
>>> proxy and
>>> no additional hosts in server.xml.
>>>
>>> I could write a separate application that serves the image files
>>> internally.
>>> It would be quite simple to do, but it would potentially require server
>>> modifications.
>>>
>>> 1. Listens to requests on a separate port (server.xml modification)
>>> 2. Application resides in the default host (for multi-host configs)
>>>
>>> Then the path would look something like the following.
>>>
>>> A. Request
>>>
>>> Browser ------------> NAT ------------> Server
>>> public IP private IP
>>> ww.xx.yy.zz aa.bb.cc.dd
>>>
>>> B. Image
>>>
>>> +-----< Server
>>> | ^
>>> | | localhost - some arbitrary port
>>> +---------+
>>>
>>> C. Resulting PDF
>>>
>>> Browser <------------ NAT <------------ Server
>>> public IP private IP
>>> ww.xx.yy.zz aa.bb.cc.dd
>>>
>> That PDF-generating library - where it sits? Is it on Server?
>>
>> How does the library get its source data (an XSL-FO document?) that is
>> converted to PDF? Which of the following:
>>
>> a) It uses HTTP to request a JSP page. Then it uses HTTP to request
>> the image.
>>
>> b) You prepare the data and feed it to the library programmatically
>> (e.g. as an InputStream). The library performs a request to get the
>> image.
>>
>> If it is a) you could use UrlRewriteFilter + forward to serve the
>> image from any arbitrary URL.
>>
>> If it is b),
>> - How the library knows the base URL of the document, to resolve
>> relative links?
>> - Could you use a "file:///" URL for the image?
>> - Does the library use some resolver before looking for the image
>> (e.g. javax.xml.stream.XMLResolver, org.xml.sax.EntityResolver),
>> so that you can intercept the lookup.
>> - In theory, it is possible to use custom URL schemes, like Tomcat's
>> internal jndi:// one.
>>
>> Best regards,
>> Konstantin Kolinko
>
>
> If you're embedding the image in the PDF, a file: URL might be the best
> way to go. As a workaround, you might consider configuring the absolute
> pathname of a work directory as a system property and using that to
> store the image and create the file: URL.
>
> -Terence Bandoian
I've tried an absolute file name, which doesn't seem to work with the
PDF library. This is quite odd, since part of their documentation on the
prohibition of using absolute URLs states:
Image should be referenced with relative path. Absolute URLs, like
src="http://myserver:80/path/to/img.gif" are allowed as well, but
src="/path/to/img.gif" not.
I'll try a file URL, but without knowing the mechanism that is being
used to resolve the reference, it'll just be trial and error.
While I appreciate all of the help being given in this discussion, we've
wandered quite far afield from Tomcat stuff. I'm thinking of marking
future replies as [OT] just so that people won't get bogged down in the
minutia of a third party library issue.
Again, thanks lots.
I now have four games to play:
1. file URL - I suspect this will fail
2. UrlRewriteFilter - rewrite to localhost
a. This will work if everything is on one host and
b. I'm allowed to connect to it via HTTP
c. It's the only host or request headers contain the host name
3. UrlRewriteFilter - rewrite to IP address (see above for issues)
4. File(System.getProperty("catalina.base")).getCanonicalPath()
The last is a Tomcat-specific solution. This makes the web application a
bit less portable, but in this case that's not an issue.
In ease of testing, 1 and 4 seem to be simple to try. Items 2 and 3 will
require a bit of learning on my part (and adding another library to the
soup).
. . . just my two cents.
/mde/
Wow, again thanks for all the feedback. I have to say that this list is
probably one of the most productive / useful mailing lists I've ever
been on.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by "Terence M. Bandoian" <te...@tmbsw.com>.
On 8/25/2012 7:01 AM, Konstantin Kolinko wrote:
> 2012/8/25 Mark Eggers <it...@yahoo.com>:
>> On 8/24/2012 2:55 PM, Konstantin Kolinko wrote:
>>> 2012/8/25 Mark Eggers <it...@yahoo.com>:
>>>> (...)
>>>>
>>>> An application makes use of a PDF library which generates PDFs from JSP
>>>> pages. One of these pages includes a dynamically generated graph.
>>>>
>>>> This library cannot use streaming data to include the graph.
>>>>
>>>> (...)
>>>>
>>> ============
>>> 3. What is your hairpinning problem?
>>>
>> Hairpinning is when a system is sitting behind a NAT, and it attempts to
>> send itself a packet to the public IP address.
>>
>> +----------------<
>> | NAT <----------------> Server
>> +---------------->
>> public IP private IP
>> ww.xx.yy.zz aa.bb.cc.dd
>>
>> In this environment it occurs with a host name. The server sends a packet to
>> itself using the publicly accessible name, which resolves to the external IP
>> address (ww.xx.yy.zz), which then gets sent right back to the server.
>>
>> Unless you specifically configure a NAT device to allow this, the packet
>> will be dropped.
>>
>>
>>> Is it because you use "a fully qualified URL point to a servlet" and
>>> you cannot build that URL correctly?
>>>
>>> Do you really need an absolute URL? You mean that you cannot call your
>>> servlet otherwise?
>>
>> This is correct. The library in use will only take relative URLs, or fully
>> qualified URLs (including protocol, port, and host name).
>>
>> Relative URLs means I write back into the web application, which is where
>> getRealPath() comes into play.
>>
>> I cannot use absolute URLs (/context/servlet-name?imagename=foo). This gets
>> interpreted by the library as an absolute file name (and no, using
>> /java.io.tmpdir/foo does not work).
>>
>> I would love to rewrite my URL to be
>>
>> http://localhost:80/context/servlet-name?imagename=foo
>>
>> That will only work if there are no virtual hosts in any front end proxy and
>> no additional hosts in server.xml.
>>
>> I could write a separate application that serves the image files internally.
>> It would be quite simple to do, but it would potentially require server
>> modifications.
>>
>> 1. Listens to requests on a separate port (server.xml modification)
>> 2. Application resides in the default host (for multi-host configs)
>>
>> Then the path would look something like the following.
>>
>> A. Request
>>
>> Browser ------------> NAT ------------> Server
>> public IP private IP
>> ww.xx.yy.zz aa.bb.cc.dd
>>
>> B. Image
>>
>> +-----< Server
>> | ^
>> | | localhost - some arbitrary port
>> +---------+
>>
>> C. Resulting PDF
>>
>> Browser <------------ NAT <------------ Server
>> public IP private IP
>> ww.xx.yy.zz aa.bb.cc.dd
>>
> That PDF-generating library - where it sits? Is it on Server?
>
> How does the library get its source data (an XSL-FO document?) that is
> converted to PDF? Which of the following:
>
> a) It uses HTTP to request a JSP page. Then it uses HTTP to request the image.
>
> b) You prepare the data and feed it to the library programmatically
> (e.g. as an InputStream). The library performs a request to get the
> image.
>
> If it is a) you could use UrlRewriteFilter + forward to serve the
> image from any arbitrary URL.
>
> If it is b),
> - How the library knows the base URL of the document, to resolve relative links?
> - Could you use a "file:///" URL for the image?
> - Does the library use some resolver before looking for the image
> (e.g. javax.xml.stream.XMLResolver, org.xml.sax.EntityResolver),
> so that you can intercept the lookup.
> - In theory, it is possible to use custom URL schemes, like Tomcat's
> internal jndi:// one.
>
> Best regards,
> Konstantin Kolinko
If you're embedding the image in the PDF, a file: URL might be the best
way to go. As a workaround, you might consider configuring the absolute
pathname of a work directory as a system property and using that to
store the image and create the file: URL.
-Terence Bandoian
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Mark Eggers <it...@yahoo.com>.
On 8/25/2012 12:19 PM, Konstantin Kolinko wrote:
> 2012/8/25 Mark Eggers <it...@yahoo.com>:
>> On 8/25/2012 5:01 AM, Konstantin Kolinko wrote:
>>>
>>> That PDF-generating library - where it sits? Is it on Server?
>>>
>>
>> It sits in the application. It's actually two jar files - one that does the
>> PDF translation proper and the other is a tag library.
>>
>>
>>> How does the library get its source data (an XSL-FO document?) that is
>>> converted to PDF? Which of the following:
>>>
>>
>> This I don't know, but I suspect not. The library is obfuscated, but there
>> doesn't appear to be the standard XSL-FO translation.
>>
>>
>>> a) It uses HTTP to request a JSP page. Then it uses HTTP to request the
>>> image.
>>>
>>
>> This is the way I think it operates when you use a tag library. Why the
>> library requires a relative URL or a fully qualified URL but does not work
>> with an absolute URL I do not know.
>>
>
> A tag can easily get its content as a String or as a Reader.
>
> E.g. see attributes in <jsp:doBody/> and <jsp:invoke/> standard
> actions, JspFragment.invoke(Writer), BodyContent.getReader(),
> BodyContent.getString().
>
> A tag also has access to the request object and to the servlet
> context, so it can use those to create an URL, or it uses
> ServletContext.getResource[AsStream]() to get the file.
>
> A debugger + a breakpoint in URL.openConnection() or in
> ServletContext.getResource(), or in some java.io classes can tell how
> the data are loaded. (Or just ask the library authors).
>
I'd love to ask the library authors. Sadly, they seem less than
responsive. I sent them a small web application that creates the permgen
leak (happens any time an image is included) several weeks ago, and no
response so far.
I'll ping them again today.
>>> (...)
>>
>>
>> Hmm, I think I tried UrlRewriteFilter before, but maybe not in quite the
>> right way. You are thinking along the lines of rewriting
>>
>> http://some-host:80/context/servlet?name=imagename
>>
>> to
>>
>> ../servlet-location/servlet?name=imagename
>>
>> and letting the servlet dig it out of java.io.tmpdir?
>>
>
> I was thinking about naming the images in some specific way and using
> that to forward requests to the servlet, regardless of actual path.
>
I actually have the file name. I can pass it as a parameter to the
servlet getting the image, send that off to a POJO to figure out where
the image actually is, return a BufferedImage, and send that out.
> On second thought, it might be even easier to just use suffix matching
> in web.xml.
> E.g. if the images are named *.dynamic.png and the servlet (or filter)
> is mapped like
>
> <servlet-mapping>
> <url-pattern>*.dynamic.png</url-pattern>
>
> and then the servlet can ignore all the path except the file name.
>
Ah, and I wish there was a regular expression I could map to file names.
The file names are (currently) some arbitrary number followed by an
underscore, followed by the session id (?!) followed by .jpg.
I didn't write this . . . I'm just the poor systems architect who gets
to make stuff work. I'm not really a Java developer. I do enough
development to get myself in trouble, and that's about it.
>
> BTW, if it were Tomcat 7 (and not 6.0.20) you would also have an
> option of using <Context aliases="..." /> to map the temporary folder
> into some subdirectory of your web application.
>
Lots of nice things in Tomcat 7. I doubt that the target environment
will move to that any time soon.
Per your and Terence's suggestion, I just tried a file:/// URL on my
local platform.
I replaced the img src="servlet-url" with:
file:///c:/Users/mdeggers/Apache/apache-tomcat-6.0.35/temp/gf-title-sm.jpg
since I'm doing this on a Windows 7 machine right now.
I got images embedded in the PDF.
I'm guessing (as you apparently did, Konstantin) that the library
creates a URL from the string, then does an openConnection() (or maybe
an openStream()).
I am sure glad this isn't running under a security manager.
I'll make some changes in my copy of the application, and run it on the
test server. It's easy enough to construct a string, stuff it into a
singleton via a listener, then have that resource available everywhere
it's needed in the application.
I won't know if this is the final solution until at least Monday, but I
can reproduce most of this (hairpinning at least) on a test environment.
Thanks to all - I'll post a followup when I get feedback from the people
running the target environment.
. . . . thanks again
/mde/
>
> Best regards,
> Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Konstantin Kolinko <kn...@gmail.com>.
2012/8/25 Mark Eggers <it...@yahoo.com>:
> On 8/25/2012 5:01 AM, Konstantin Kolinko wrote:
>>
>> That PDF-generating library - where it sits? Is it on Server?
>>
>
> It sits in the application. It's actually two jar files - one that does the
> PDF translation proper and the other is a tag library.
>
>
>> How does the library get its source data (an XSL-FO document?) that is
>> converted to PDF? Which of the following:
>>
>
> This I don't know, but I suspect not. The library is obfuscated, but there
> doesn't appear to be the standard XSL-FO translation.
>
>
>> a) It uses HTTP to request a JSP page. Then it uses HTTP to request the
>> image.
>>
>
> This is the way I think it operates when you use a tag library. Why the
> library requires a relative URL or a fully qualified URL but does not work
> with an absolute URL I do not know.
>
A tag can easily get its content as a String or as a Reader.
E.g. see attributes in <jsp:doBody/> and <jsp:invoke/> standard
actions, JspFragment.invoke(Writer), BodyContent.getReader(),
BodyContent.getString().
A tag also has access to the request object and to the servlet
context, so it can use those to create an URL, or it uses
ServletContext.getResource[AsStream]() to get the file.
A debugger + a breakpoint in URL.openConnection() or in
ServletContext.getResource(), or in some java.io classes can tell how
the data are loaded. (Or just ask the library authors).
>> (...)
>
>
> Hmm, I think I tried UrlRewriteFilter before, but maybe not in quite the
> right way. You are thinking along the lines of rewriting
>
> http://some-host:80/context/servlet?name=imagename
>
> to
>
> ../servlet-location/servlet?name=imagename
>
> and letting the servlet dig it out of java.io.tmpdir?
>
I was thinking about naming the images in some specific way and using
that to forward requests to the servlet, regardless of actual path.
On second thought, it might be even easier to just use suffix matching
in web.xml.
E.g. if the images are named *.dynamic.png and the servlet (or filter)
is mapped like
<servlet-mapping>
<url-pattern>*.dynamic.png</url-pattern>
and then the servlet can ignore all the path except the file name.
BTW, if it were Tomcat 7 (and not 6.0.20) you would also have an
option of using <Context aliases="..." /> to map the temporary folder
into some subdirectory of your web application.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Mark Eggers <it...@yahoo.com>.
On 8/28/2012 11:39 AM, Mark Eggers wrote:
> ==== COMMENT ====
>
> Well, fantastic isn't quite the word I'd use . . . .
>
> I wonder what would happen if I used the classesToInitialize attribute
> of the JreMemoryLeakPreventionListner, and gave it java.awt.MediaTracker?
>
Just to update - same memory leak as shown by VisualVM. In retrospect
this is reasonable, since concrete implementations of the Image class
retain references as to how they're loaded.
Only an Image.flush() releases these resources.
Since I don't have access to those resources, I cannot execute the method.
. . . . rats
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Mark Eggers <it...@yahoo.com>.
Comments inline with ==== COMMENT ====
On 8/28/2012 8:19 AM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Mark,
>
> On 8/27/12 6:07 PM, Mark Eggers wrote:
>> >On 8/27/2012 1:24 PM, Christopher Schultz wrote:
>>> >>
>>> >>Hmm... that's ... not supposed to happen. The AppContext
>>> >>leak-prevention has been in Tomcat since back in 2009. Can you
>>> >>confirm that the JreMemoryLeakProtectionListener is actually
>>> >>being loaded?
>> >
>> >A heap dump (jvisualvm) on the following environments show one
>> >instance of the class being loaded.
> That's good!
>
>> >1. Windows 7, JRE 1.7.0_05, Tomcat 6.0.35 2. Windows 7, JRE
>> >1.7.0_05, Tomcat 7.0.27 3. Windows 7, JRE 1.6.0_32, Tomcat 6.0.35
>> >4. Windows 7, JRE 1.6.0_32, Tomcat 7.0.27 5. CentOS 6.3, JRE
>> >1.6.0_32, Tomcat 6.0.35, -Djava.awt.headless=true this is actually
>> >running X Windows though 6. CentOS 5.8, JRE 1.6.0_32, Tomcat
>> >6.0.35, -Djava.awt.headless=true not running X Windows (dump via
>> >jmap)
>> >
>> >Hmm - looking at Tomcat 6.0.35 on the CentOS 5.8 system, I see the
>> >following when looking at the instance of
>> >JreMemoryProtectionListener.
>> >
>> >appContextProtection (Z) : true awtThreadProtection (Z) : true
> appContextProtection=true is the default and awtThreadProtection=true
> is*not* the default. I would say that this is the proper
> configuration for what you are trying to solve. In that case, can you
> check the ClassLoader used for the AppContext and AWT threads?
==== COMMENT ====
Interesting thought. So you suspect that even with the
JreMemoryLeakPreventionListener loaded, it's not taking care of the
AppContext?
I just took a look at one of my heap dumps.
AppContext has one instance loaded. The contextClassLoader field is set
to Launcher$AppClassLoader, which is what you would expect.
EventQueue has one instance loaded. The contextClassLoader field is set
to Launcher$AppClassLoader, which is what you would expect.
==== COMMENT ====
>
>> >And I do have the attribute set to manage the AWT thread (at least
>> >on the Tomcat I dumped).
>> >
>> >On the Windows (Tomcat 6.0.35, JRE 1.6.0_32), I see the following:
>> >
>> >appContextProtection (Z) : true awtThreadProtection (Z) : false
>> >classesToInitialize (L) : <null> driverManagerProtection (Z) :
>> >true gcDaemonProtection (Z) : true ldapPoolProtection (Z) : true
>> >securityLoginConfigurationProtection (Z) : true
>> >securityPolicyProtection (Z) : true tokenPollerProtection (Z) :
>> >true urlCacheProtection (Z) : true xmlParsingProtection (Z) : true
>> >
>> >which is what I expect from server.xml.
> So the server.xml has awtThreadProtection=false (or just uses the
> default)?
>
>> >Commenting out the listener (actually it doesn't matter either way
>> >since I'm running in a windowed environment) and adding the AWT
>> >Event thread protection produces on Windows:
>> >
>> >appContextProtection (Z) : true awtThreadProtection (Z) : true
>> >classesToInitialize (L) : <null> driverManagerProtection (Z) :
>> >true gcDaemonProtection (Z) : true ldapPoolProtection (Z) : true
>> >securityLoginConfigurationProtection (Z) : true
>> >securityPolicyProtection (Z) : true tokenPollerProtection (Z) :
>> >true urlCacheProtection (Z) : true xmlParsingProtection (Z) : true
> If you comment-out the listener, they why is the class loaded at all?
> Are you sure everything in your environment is sane?
>
==== COMMENT ====
Sorry, this is probably one of the few times I was less than careful
with my writing. I meant to say that I removed the listener with the
attribute (AWTThreadProtection="true") and used the default configuration.
I actually have both in server.xml. I just comment out whichever one I'm
not testing at the moment.
Hopefully that's a bit clearer :-p.
==== COMMENT ====
>> >And I still see the exact same permgen memory leak upon unloading
>> >the application.
>> >
>> >The problem looks to be in an internal use of MediaTracker. I
>> >suspect that the library uses this to speed image recovery should
>> >there be repeated calls.
> IIRC MediaTracker is a concurrent component that loads stuff (like
> images) in the background and sends update-events to interested
> components. Perhaps there is yet another thread that needs to be
> managed on top of the AWT thread. Can you take a thread dump and
> possibly indicate the ClassLoader for each thread?
>
==== COMMENT ====
I'll go poke around the heap dump and see what I find. Possibly look at
each thread and find the classloader ? Sigh, that's a pain.
I'll also do a thread dump while the application is running to see if
any strange threads appear.
My immediate guess (from reading the AWT model) is that there will not
be another thread.
==== COMMENT ====
>> >Unfortunately, these images are dynamically generated, so any use
>> >of MediaTracker is pretty worthless in this use case.
> I've found that use of the MediaTracker is entirely worthless except
> when using GUIs that need to display images within components (like
> buttons, etc.). Anyone using MediaTracker in a client-side library is
> making a mistake (that is, your 3rd-party library is either not
> targeted towards a server-side application, or the devs don't know
> what they are doing).
>
>> >There's a way to purge the library caches, and I've experimented
>> >with that as well. Unfortunately, it doesn't unregister the images
>> >it stored in the MediaTracker. In short, flushing the library's
>> >cache in this use case does not address the problem.
> Fantastic.
==== COMMENT ====
Well, fantastic isn't quite the word I'd use . . . .
I wonder what would happen if I used the classesToInitialize attribute
of the JreMemoryLeakPreventionListner, and gave it java.awt.MediaTracker?
Still, the images would be loaded (although they're small). However, a
quick glance at the Image class doesn't reveal why that class would
retain any references to the classes that added them to the
MediaTracker. I'd just get a bunch of images stuffed into memory, but
not the rest of the web application.
In short, I'd still get a slow (or fast - depending on application
usage) memory leak. It would just leave a bit less around when I
unloaded the application (possibly).
This is also a hack, in that I may or may not be able to do this on the
target server. However, I could look at the listener code and duplicate
that in my listener that starts up the AWT thread.
==== COMMENT ====
>
> - -chris
. . . . rabbit holes, don't tumble down them.
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 8/27/12 6:07 PM, Mark Eggers wrote:
> On 8/27/2012 1:24 PM, Christopher Schultz wrote:
>>
>> Hmm... that's ... not supposed to happen. The AppContext
>> leak-prevention has been in Tomcat since back in 2009. Can you
>> confirm that the JreMemoryLeakProtectionListener is actually
>> being loaded?
>
> A heap dump (jvisualvm) on the following environments show one
> instance of the class being loaded.
That's good!
> 1. Windows 7, JRE 1.7.0_05, Tomcat 6.0.35 2. Windows 7, JRE
> 1.7.0_05, Tomcat 7.0.27 3. Windows 7, JRE 1.6.0_32, Tomcat 6.0.35
> 4. Windows 7, JRE 1.6.0_32, Tomcat 7.0.27 5. CentOS 6.3, JRE
> 1.6.0_32, Tomcat 6.0.35, -Djava.awt.headless=true this is actually
> running X Windows though 6. CentOS 5.8, JRE 1.6.0_32, Tomcat
> 6.0.35, -Djava.awt.headless=true not running X Windows (dump via
> jmap)
>
> Hmm - looking at Tomcat 6.0.35 on the CentOS 5.8 system, I see the
> following when looking at the instance of
> JreMemoryProtectionListener.
>
> appContextProtection (Z) : true awtThreadProtection (Z) : true
appContextProtection=true is the default and awtThreadProtection=true
is *not* the default. I would say that this is the proper
configuration for what you are trying to solve. In that case, can you
check the ClassLoader used for the AppContext and AWT threads?
> And I do have the attribute set to manage the AWT thread (at least
> on the Tomcat I dumped).
>
> On the Windows (Tomcat 6.0.35, JRE 1.6.0_32), I see the following:
>
> appContextProtection (Z) : true awtThreadProtection (Z) : false
> classesToInitialize (L) : <null> driverManagerProtection (Z) :
> true gcDaemonProtection (Z) : true ldapPoolProtection (Z) : true
> securityLoginConfigurationProtection (Z) : true
> securityPolicyProtection (Z) : true tokenPollerProtection (Z) :
> true urlCacheProtection (Z) : true xmlParsingProtection (Z) : true
>
> which is what I expect from server.xml.
So the server.xml has awtThreadProtection=false (or just uses the
default)?
> Commenting out the listener (actually it doesn't matter either way
> since I'm running in a windowed environment) and adding the AWT
> Event thread protection produces on Windows:
>
> appContextProtection (Z) : true awtThreadProtection (Z) : true
> classesToInitialize (L) : <null> driverManagerProtection (Z) :
> true gcDaemonProtection (Z) : true ldapPoolProtection (Z) : true
> securityLoginConfigurationProtection (Z) : true
> securityPolicyProtection (Z) : true tokenPollerProtection (Z) :
> true urlCacheProtection (Z) : true xmlParsingProtection (Z) : true
If you comment-out the listener, they why is the class loaded at all?
Are you sure everything in your environment is sane?
> And I still see the exact same permgen memory leak upon unloading
> the application.
>
> The problem looks to be in an internal use of MediaTracker. I
> suspect that the library uses this to speed image recovery should
> there be repeated calls.
IIRC MediaTracker is a concurrent component that loads stuff (like
images) in the background and sends update-events to interested
components. Perhaps there is yet another thread that needs to be
managed on top of the AWT thread. Can you take a thread dump and
possibly indicate the ClassLoader for each thread?
> Unfortunately, these images are dynamically generated, so any use
> of MediaTracker is pretty worthless in this use case.
I've found that use of the MediaTracker is entirely worthless except
when using GUIs that need to display images within components (like
buttons, etc.). Anyone using MediaTracker in a client-side library is
making a mistake (that is, your 3rd-party library is either not
targeted towards a server-side application, or the devs don't know
what they are doing).
> There's a way to purge the library caches, and I've experimented
> with that as well. Unfortunately, it doesn't unregister the images
> it stored in the MediaTracker. In short, flushing the library's
> cache in this use case does not address the problem.
Fantastic.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAlA84XEACgkQ9CaO5/Lv0PDHoACeLrDXPbfkqeLBoaIy8My8gmGA
ZgwAniWtkEP3WSk47Mxjg36flndbbJ0U
=Abus
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Mark Eggers <it...@yahoo.com>.
Edited for sanity.
Comments surrounded by ==== COMMENT ====
On 8/27/2012 1:24 PM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Mark,
>
> On 8/27/12 3:39 PM, Mark Eggers wrote:
>> This is long and mostly off-topic. Read if you wish to follow the
>> fun and games of tracking down permgen leaks, especially with a
>> third party obfuscated library.
>
> :)
>
>> JMLPL documentation?
>
> http://tomcat.apache.org/tomcat-6.0-doc/config/listeners.html#JRE_Memory_Leak_Prevention_Listener_-_org.apache.catalina.core.JreMemoryLeakPreventionListener
==== COMMENT ====
Roger that - JMLPL - java memory leak prevention listener - I should
have known.
==== COMMENT ====
>
>> I'm also stuck with 6.0.x (where x is 35 in development, and 20 in
>> production). I do not think that upgrading production is a
>> possibility.
>>
>> However, I'm seeing this in 6.0.35.
>
> I was referring to this, which was added in 6.0.34:
> https://issues.apache.org/bugzilla/show_bug.cgi?id=51688
==== COMMENT ====
Ah, then it's a good thing I do this in the application, since the
target platform is 6.0.20.
==== COMMENT ====
>
> It's the same thing you do in your own listener (unless
> getSystemEventQueue() does something even worse than simply calling
> getDefaultToolkit()).
>
==== COMMENT ====
All this really does is start the AWT Event Thread. This seems to work
fine for both Windows and various flavors of Linux (Fedora, CentOS). It
appears to work both with a windowing environment, and without a
windowing environment running java.awt.headless=true. At least the
thread shows up in thread dumps.
==== COMMENT ====
>> This was done since the target server was not known (at the time).
>> Also, this doesn't require changes to a Tomcat server that is not
>> in my control.
>>
>> Using the following leak prevention listener configuration didn't
>> change the fact that there was a leak on unloading the
>> application.
>>
>> <Listener
>> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>>
>>
> AWTThreadProtection="true"/>
>
> Hmm... does that happen in 6.0.35 as well, or only 6.0.20? It was only
> added in 6.0.34 but it should be working. If not, please re-open the
> aforementioned bug and explain.
==== COMMENT ====
This does happen in both 6.0.35 and 7.0.27. I know, I know, I'll upgrade
to 7.0.29 soon.
==== COMMENT ====
>
>> Note that by default I have this configured in my environment.
>>
>> <Listener
>> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"/>
>>
>> According to the documentation, this takes care of many leak
>> issues, including AppContext.
>
> Right.
==== COMMENT ====
Stack traces deleted to keep everyone sane
==== COMMENT ====
>
> Hmm... that's ... not supposed to happen. The AppContext
> leak-prevention has been in Tomcat since back in 2009. Can you confirm
> that the JreMemoryLeakProtectionListener is actually being loaded?
==== COMMENT ====
A heap dump (jvisualvm) on the following environments show one instance
of the class being loaded.
1. Windows 7, JRE 1.7.0_05, Tomcat 6.0.35
2. Windows 7, JRE 1.7.0_05, Tomcat 7.0.27
3. Windows 7, JRE 1.6.0_32, Tomcat 6.0.35
4. Windows 7, JRE 1.6.0_32, Tomcat 7.0.27
5. CentOS 6.3, JRE 1.6.0_32, Tomcat 6.0.35, -Djava.awt.headless=true
this is actually running X Windows though
6. CentOS 5.8, JRE 1.6.0_32, Tomcat 6.0.35, -Djava.awt.headless=true
not running X Windows (dump via jmap)
Hmm - looking at Tomcat 6.0.35 on the CentOS 5.8 system, I see the
following when looking at the instance of JreMemoryProtectionListener.
appContextProtection (Z) : true
awtThreadProtection (Z) : true
classesToInitialize (L) : <null>
driverManagerProtection (Z) : true
gcDaemonProtection (Z) : true
ldapPoolProtection (Z) : true
securityLoginConfigurationProtection (Z) : true
securityPolicyProtection (Z) : true
tokenPollerProtection (Z) : true
urlCacheProtection (Z) : true
xmlParsingProtection (Z) : true
And I do have the attribute set to manage the AWT thread (at least on
the Tomcat I dumped).
On the Windows (Tomcat 6.0.35, JRE 1.6.0_32), I see the following:
appContextProtection (Z) : true
awtThreadProtection (Z) : false
classesToInitialize (L) : <null>
driverManagerProtection (Z) : true
gcDaemonProtection (Z) : true
ldapPoolProtection (Z) : true
securityLoginConfigurationProtection (Z) : true
securityPolicyProtection (Z) : true
tokenPollerProtection (Z) : true
urlCacheProtection (Z) : true
xmlParsingProtection (Z) : true
which is what I expect from server.xml.
Commenting out the listener (actually it doesn't matter either way since
I'm running in a windowed environment) and adding the AWT Event thread
protection produces on Windows:
appContextProtection (Z) : true
awtThreadProtection (Z) : true
classesToInitialize (L) : <null>
driverManagerProtection (Z) : true
gcDaemonProtection (Z) : true
ldapPoolProtection (Z) : true
securityLoginConfigurationProtection (Z) : true
securityPolicyProtection (Z) : true
tokenPollerProtection (Z) : true
urlCacheProtection (Z) : true
xmlParsingProtection (Z) : true
And I still see the exact same permgen memory leak upon unloading the
application.
The problem looks to be in an internal use of MediaTracker. I suspect
that the library uses this to speed image recovery should there be
repeated calls.
Unfortunately, these images are dynamically generated, so any use of
MediaTracker is pretty worthless in this use case.
There's a way to purge the library caches, and I've experimented with
that as well. Unfortunately, it doesn't unregister the images it stored
in the MediaTracker. In short, flushing the library's cache in this use
case does not address the problem.
. . . thanks for your feedback Chris.
/mde/
==== COMMENT ====
>
> - -chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 8/27/12 3:39 PM, Mark Eggers wrote:
> This is long and mostly off-topic. Read if you wish to follow the
> fun and games of tracking down permgen leaks, especially with a
> third party obfuscated library.
:)
> JMLPL documentation?
http://tomcat.apache.org/tomcat-6.0-doc/config/listeners.html#JRE_Memory_Leak_Prevention_Listener_-_org.apache.catalina.core.JreMemoryLeakPreventionListener
> I'm also stuck with 6.0.x (where x is 35 in development, and 20 in
> production). I do not think that upgrading production is a
> possibility.
>
> However, I'm seeing this in 6.0.35.
I was referring to this, which was added in 6.0.34:
https://issues.apache.org/bugzilla/show_bug.cgi?id=51688
It's the same thing you do in your own listener (unless
getSystemEventQueue() does something even worse than simply calling
getDefaultToolkit()).
> This was done since the target server was not known (at the time).
> Also, this doesn't require changes to a Tomcat server that is not
> in my control.
>
> Using the following leak prevention listener configuration didn't
> change the fact that there was a leak on unloading the
> application.
>
> <Listener
> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>
>
AWTThreadProtection="true"/>
Hmm... does that happen in 6.0.35 as well, or only 6.0.20? It was only
added in 6.0.34 but it should be working. If not, please re-open the
aforementioned bug and explain.
> Note that by default I have this configured in my environment.
>
> <Listener
> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"/>
>
> According to the documentation, this takes care of many leak
> issues, including AppContext.
Right.
> Now the fun begins. I first run this under Tomcat 6.0.35 and JRE
> 1.6.0_32 on Windows 7 Home Premium (64 bit).
>
> Tracking down one of the leaked library classes, I get this.
>
> --> class sun.awt.AppContext (160 bytes) (static field
> mainAppContext:) --> sun.awt.AppContext@0xa0126130 (81 bytes)
> (field table:) --> java.util.HashMap@0xa0126168 (64 bytes) (field
> table:) --> [Ljava.util.HashMap$Entry;@0xa0126198 (144 bytes)
> (Element 1 of [Ljava.util.HashMap$Entry;@0xa0126198:) -->
> java.util.HashMap$Entry@0xa04cc760 (44 bytes) (field value:) -->
> java.awt.MediaTracker@0xa04cc790 (32 bytes) (field target:) -->
> javax.swing.ImageIcon$3@0xa04cc7a8 (379 bytes) (field acc:) -->
> java.security.AccessControlContext@0xa04cc888 (41 bytes) (field
> context:) --> [Ljava.security.ProtectionDomain;@0xa04cc8a8 (80
> bytes) (Element 3 of
> [Ljava.security.ProtectionDomain;@0xa04cc8a8:) -->
> java.security.ProtectionDomain@0xa0424dc8 (58 bytes) (field
> classloader:) -->
> org.apache.catalina.loader.WebappClassLoader@0xa04205d0 (309
> bytes)
>
> Now, I run this under Tomcat 7.0.27 and JRE 1.7.0._05 on Windows 7
> Home Premium (64 bit).
>
> Tracking down one of the leaked library classes, I get this.
>
> --> class sun.awt.AppContext (168 bytes) (static field
> mainAppContext:) --> sun.awt.AppContext@0xa0255698 (81 bytes)
> (field table:) --> java.util.HashMap@0xa02556d0 (64 bytes) (field
> table:) --> [Ljava.util.HashMap$Entry;@0xa0255700 (144 bytes)
> (Element 6 of [Ljava.util.HashMap$Entry;@0xa0255700:) -->
> java.util.HashMap$Entry@0xa0d81ca8 (44 bytes) (field value:) -->
> java.awt.MediaTracker@0xa0d81d28 (32 bytes) (field target:) -->
> javax.swing.ImageIcon$3@0xa0d81d40 (364 bytes) (field acc:) -->
> java.security.AccessControlContext@0xa0d81e18 (41 bytes) (field
> context:) --> [Ljava.security.ProtectionDomain;@0xa0d81e38 (80
> bytes) (Element 3 of
> [Ljava.security.ProtectionDomain;@0xa0d81e38:) -->
> java.security.ProtectionDomain@0xa0c14c58 (58 bytes) (field
> classloader:) -->
> org.apache.catalina.loader.WebappClassLoader@0xa0b135c8 (325
> bytes)
>
> Looks pretty much the same.
Hmm... that's ... not supposed to happen. The AppContext
leak-prevention has been in Tomcat since back in 2009. Can you confirm
that the JreMemoryLeakProtectionListener is actually being loaded?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAlA714IACgkQ9CaO5/Lv0PDOTwCff0Gj5/aerx6TyyA/BlcoTQWA
JMoAn0Pc9+2N12uc3642XwanFUuy9SmI
=6gJi
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Mark Eggers <it...@yahoo.com>.
This is long and mostly off-topic. Read if you wish to follow the fun
and games of tracking down permgen leaks, especially with a third party
obfuscated library.
On 8/27/2012 8:15 AM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Mark,
>
> On 8/25/12 2:15 PM, Mark Eggers wrote:
>> I could even work on convincing people to move away from this PDF
>> library (creates permgen leaks when images are included) to
>> something a bit more sane. Maybe even something that read stream
>> data and didn't cause a permgen memory leak . . . .
>
> There have been some recent updates to the
> JreMemoryLeakPreventionListener that may help alleviate these. Search
> for "AWT" in the JMLPL documentation.
>
> Also, if you've found something that you can trace back to the JVM,
> please let us know and we can update the JMLPL to help avoid PermGen
> leaks.
>
> - -chris
Chris,
JMLPL documentation?
I'm also stuck with 6.0.x (where x is 35 in development, and 20 in
production). I do not think that upgrading production is a possibility.
However, I'm seeing this in 6.0.35.
A custom context listener has been used to pin the AWT thread to the
system class loader. This was based on comments made in the Tomcat
mailing list a while back. The code is as follows:
public class GraphicsListener implements ServletContextListener {
private Log log =
LogFactory.getLog(org.mdeggers.listeners.GraphicsListener.class);
public void contextInitialized(ServletContextEvent sce) {
if (log.isInfoEnabled()) {
log.info("Pinning AWT Event Thread to parent classloader.");
}
// get the current classloader for the thread and store it
Thread myThread = Thread.currentThread();
ClassLoader ccl = myThread.getContextClassLoader();
// switch to the system classloader and start up the AWT thread
myThread.setContextClassLoader(ClassLoader.getSystemClassLoader());
EventQueue eq = Toolkit.getDefaultToolkit().getSystemEventQueue();
// switch back to the current classloader
myThread.setContextClassLoader(ccl);
}
public void contextDestroyed(ServletContextEvent sce) {
}
}
This was done since the target server was not known (at the time). Also,
this doesn't require changes to a Tomcat server that is not in my control.
Using the following leak prevention listener configuration didn't change
the fact that there was a leak on unloading the application.
<Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
AWTThreadProtection="true"/>
Note that by default I have this configured in my environment.
<Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener"/>
According to the documentation, this takes care of many leak issues,
including AppContext.
Now the fun begins. I first run this under Tomcat 6.0.35 and JRE
1.6.0_32 on Windows 7 Home Premium (64 bit).
Tracking down one of the leaked library classes, I get this.
--> class sun.awt.AppContext (160 bytes) (static field mainAppContext:)
--> sun.awt.AppContext@0xa0126130 (81 bytes) (field table:)
--> java.util.HashMap@0xa0126168 (64 bytes) (field table:)
--> [Ljava.util.HashMap$Entry;@0xa0126198 (144 bytes)
(Element 1 of [Ljava.util.HashMap$Entry;@0xa0126198:)
--> java.util.HashMap$Entry@0xa04cc760 (44 bytes) (field value:)
--> java.awt.MediaTracker@0xa04cc790 (32 bytes) (field target:)
--> javax.swing.ImageIcon$3@0xa04cc7a8 (379 bytes) (field acc:)
--> java.security.AccessControlContext@0xa04cc888 (41 bytes)
(field context:)
--> [Ljava.security.ProtectionDomain;@0xa04cc8a8 (80 bytes)
(Element 3 of [Ljava.security.ProtectionDomain;@0xa04cc8a8:)
--> java.security.ProtectionDomain@0xa0424dc8 (58 bytes)
(field classloader:)
--> org.apache.catalina.loader.WebappClassLoader@0xa04205d0 (309 bytes)
Now, I run this under Tomcat 7.0.27 and JRE 1.7.0._05 on Windows 7 Home
Premium (64 bit).
Tracking down one of the leaked library classes, I get this.
--> class sun.awt.AppContext (168 bytes) (static field mainAppContext:)
--> sun.awt.AppContext@0xa0255698 (81 bytes) (field table:)
--> java.util.HashMap@0xa02556d0 (64 bytes) (field table:)
--> [Ljava.util.HashMap$Entry;@0xa0255700 (144 bytes)
(Element 6 of [Ljava.util.HashMap$Entry;@0xa0255700:)
--> java.util.HashMap$Entry@0xa0d81ca8 (44 bytes) (field value:)
--> java.awt.MediaTracker@0xa0d81d28 (32 bytes) (field target:)
--> javax.swing.ImageIcon$3@0xa0d81d40 (364 bytes) (field acc:)
--> java.security.AccessControlContext@0xa0d81e18 (41 bytes)
(field context:)
--> [Ljava.security.ProtectionDomain;@0xa0d81e38 (80 bytes)
(Element 3 of [Ljava.security.ProtectionDomain;@0xa0d81e38:)
--> java.security.ProtectionDomain@0xa0c14c58 (58 bytes)
(field classloader:)
--> org.apache.catalina.loader.WebappClassLoader@0xa0b135c8 (325 bytes)
Looks pretty much the same.
These are all chains present in the library (not the test application
that I wrote).
Here's how I arrived at the call chain.
0. Run the application and hit a PDF page that includes an image
1. Unload the application
2. Force a garbage collection with VisualVM
3. Do a heap dump with VisualVM
4. Save the heap dump
5. Use jhat to look at the heap dump
a. Find a leaked class
b. Click on its classloader
c. Reference chains from rootset - exclude weak references
d. scroll down to system class references
However, there are now clues in catalina.out (Tomcat 7).
Relative URL
------------
SEVERE: Servlet.service() for servlet default threw exception
java.io.FileNotFoundException:
The requested resource
(/PTest/pjsp/C:/Users/mdeggers/Apache/apache-tomcat-7.0.27/webapps/PTest/images/gf-title-sm.jpg)
is not available
Absolute (file:///) URL
-----------------------
SEVERE: Servlet.service() for servlet default threw exception
java.io.FileNotFoundException:
The requested resource
(/PTest/c:/Users/mdeggers/Apache/apache-tomcat-7.0.27/temp/gf-title-sm.jpg)
is not available
I apologize for the line wrapping.
In both cases, the physical location of the image file is shown
correctly. However, in the first case (relative URL) the context (PTest)
and the path to the JSP (pjsp) are added to the front. In the second
case (file:/// url), only the context (PTest) is added.
I've not posted the entire stack trace so I don't incriminate the makers
of the third party library. The stack trace is indeed different between
the relative and absolute cases.
I will post the entire stack trace on their forum and help mailing list.
I think the library does some brute force work in order to load images.
The first is the file path shown above. The second appears to be a
getResourceAsStream. Since there are icons in the JAR files supplied by
the library, these appear to be loaded. The next method (whatever that
is), appears to load the images.
The impact is slight with the test application. With the production
application there is about 20 MB of memory consumed per application reload.
. . . . what a mess.
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 8/25/12 2:15 PM, Mark Eggers wrote:
> I could even work on convincing people to move away from this PDF
> library (creates permgen leaks when images are included) to
> something a bit more sane. Maybe even something that read stream
> data and didn't cause a permgen memory leak . . . .
There have been some recent updates to the
JreMemoryLeakPreventionListener that may help alleviate these. Search
for "AWT" in the JMLPL documentation.
Also, if you've found something that you can trace back to the JVM,
please let us know and we can update the JMLPL to help avoid PermGen
leaks.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAlA7jx0ACgkQ9CaO5/Lv0PCfhwCgnOszPv1sOa3B5i+S5j0fLkIz
vZcAoLs6E4XvKZe53LmWOAAIQA1O0ACh
=QfKb
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Mark Eggers <it...@yahoo.com>.
On 8/25/2012 5:01 AM, Konstantin Kolinko wrote:
> 2012/8/25 Mark Eggers <it...@yahoo.com>:
>> On 8/24/2012 2:55 PM, Konstantin Kolinko wrote:
>>> 2012/8/25 Mark Eggers <it...@yahoo.com>:
>>>>
>>>> (...)
>>>>
>>>> An application makes use of a PDF library which generates PDFs from JSP
>>>> pages. One of these pages includes a dynamically generated graph.
>>>>
>>>> This library cannot use streaming data to include the graph.
>>>>
>>>> (...)
>>>>
>>
>>> ============
>>> 3. What is your hairpinning problem?
>>>
>>
>> Hairpinning is when a system is sitting behind a NAT, and it attempts to
>> send itself a packet to the public IP address.
>>
>> +----------------<
>> | NAT <----------------> Server
>> +---------------->
>> public IP private IP
>> ww.xx.yy.zz aa.bb.cc.dd
>>
>> In this environment it occurs with a host name. The server sends a packet to
>> itself using the publicly accessible name, which resolves to the external IP
>> address (ww.xx.yy.zz), which then gets sent right back to the server.
>>
>> Unless you specifically configure a NAT device to allow this, the packet
>> will be dropped.
>>
>>
>>> Is it because you use "a fully qualified URL point to a servlet" and
>>> you cannot build that URL correctly?
>>>
>>> Do you really need an absolute URL? You mean that you cannot call your
>>> servlet otherwise?
>>
>>
>> This is correct. The library in use will only take relative URLs, or fully
>> qualified URLs (including protocol, port, and host name).
>>
>> Relative URLs means I write back into the web application, which is where
>> getRealPath() comes into play.
>>
>> I cannot use absolute URLs (/context/servlet-name?imagename=foo). This gets
>> interpreted by the library as an absolute file name (and no, using
>> /java.io.tmpdir/foo does not work).
>>
>> I would love to rewrite my URL to be
>>
>> http://localhost:80/context/servlet-name?imagename=foo
>>
>> That will only work if there are no virtual hosts in any front end proxy and
>> no additional hosts in server.xml.
>>
>> I could write a separate application that serves the image files internally.
>> It would be quite simple to do, but it would potentially require server
>> modifications.
>>
>> 1. Listens to requests on a separate port (server.xml modification)
>> 2. Application resides in the default host (for multi-host configs)
>>
>> Then the path would look something like the following.
>>
>> A. Request
>>
>> Browser ------------> NAT ------------> Server
>> public IP private IP
>> ww.xx.yy.zz aa.bb.cc.dd
>>
>> B. Image
>>
>> +-----< Server
>> | ^
>> | | localhost - some arbitrary port
>> +---------+
>>
>> C. Resulting PDF
>>
>> Browser <------------ NAT <------------ Server
>> public IP private IP
>> ww.xx.yy.zz aa.bb.cc.dd
>>
>
> That PDF-generating library - where it sits? Is it on Server?
>
It sits in the application. It's actually two jar files - one that does
the PDF translation proper and the other is a tag library.
> How does the library get its source data (an XSL-FO document?) that is
> converted to PDF? Which of the following:
>
This I don't know, but I suspect not. The library is obfuscated, but
there doesn't appear to be the standard XSL-FO translation.
> a) It uses HTTP to request a JSP page. Then it uses HTTP to request the image.
>
This is the way I think it operates when you use a tag library. Why the
library requires a relative URL or a fully qualified URL but does not
work with an absolute URL I do not know.
> b) You prepare the data and feed it to the library programmatically
> (e.g. as an InputStream). The library performs a request to get the
> image.
>
It can work this way. The advantage of using a JSP (at least from the
programmer's standpoint) is that the library understands a lot of (but
not all) CSS. This means that if you're careful, what the user sees on
the page with a JSP will be how the PDF looks.
> If it is a) you could use UrlRewriteFilter + forward to serve the
> image from any arbitrary URL.
Hmm, I think I tried UrlRewriteFilter before, but maybe not in quite the
right way. You are thinking along the lines of rewriting
http://some-host:80/context/servlet?name=imagename
to
../servlet-location/servlet?name=imagename
and letting the servlet dig it out of java.io.tmpdir?
Ah, as I write this maybe there's another way. I wonder if the filter
has a way of getting the local (ie., private) IP address. Then the
rewrite could be:
http://internal-ip:80/context/servlet?name=imagename
I know that when the tester used the internal IP address, everything
worked as planned (the image appeared) and that it was using
java.io.tmpdir (per my logging info).
This actually tells me two things.
1. Such a rewrite has the chance of working
2. In the staging system they are either
a. Not using multiple hosts or
b. Deploying the war file in the default host
This will only work if either Tomcat is running on port 80, or any front
end server is on the same machine.
If Tomcat is connected via mod_jk to a front end machine, I'll have to
figure out how to get the front end IP address and rewrite appropriately.
There are lots of other permutations, but I'll not bore the list with those.
>
> If it is b),
> - How the library knows the base URL of the document, to resolve relative links?
> - Could you use a "file:///" URL for the image?
> - Does the library use some resolver before looking for the image
> (e.g. javax.xml.stream.XMLResolver, org.xml.sax.EntityResolver),
> so that you can intercept the lookup.
> - In theory, it is possible to use custom URL schemes, like Tomcat's
> internal jndi:// one.
Actually, I wish it was b). Managing this environment in a POJO would be
much nicer. I could possibly play games with memory-based files and
eliminate writing to disk (the images are small, not reused).
I could even work on convincing people to move away from this PDF
library (creates permgen leaks when images are included) to something a
bit more sane. Maybe even something that read stream data and didn't
cause a permgen memory leak . . . .
However, that's not going to happen in the time frame for this
deployment. The filter might solve the problem.
> Best regards,
> Konstantin Kolinko
Thanks again for your insite Konstantin . . .
. . . just my (now busy weekend) two cents.
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Konstantin Kolinko <kn...@gmail.com>.
2012/8/25 Mark Eggers <it...@yahoo.com>:
> On 8/24/2012 2:55 PM, Konstantin Kolinko wrote:
>> 2012/8/25 Mark Eggers <it...@yahoo.com>:
>>>
>>>(...)
>>>
>>> An application makes use of a PDF library which generates PDFs from JSP
>>> pages. One of these pages includes a dynamically generated graph.
>>>
>>> This library cannot use streaming data to include the graph.
>>>
>>> (...)
>>>
>
>> ============
>> 3. What is your hairpinning problem?
>>
>
> Hairpinning is when a system is sitting behind a NAT, and it attempts to
> send itself a packet to the public IP address.
>
> +----------------<
> | NAT <----------------> Server
> +---------------->
> public IP private IP
> ww.xx.yy.zz aa.bb.cc.dd
>
> In this environment it occurs with a host name. The server sends a packet to
> itself using the publicly accessible name, which resolves to the external IP
> address (ww.xx.yy.zz), which then gets sent right back to the server.
>
> Unless you specifically configure a NAT device to allow this, the packet
> will be dropped.
>
>
>> Is it because you use "a fully qualified URL point to a servlet" and
>> you cannot build that URL correctly?
>>
>> Do you really need an absolute URL? You mean that you cannot call your
>> servlet otherwise?
>
>
> This is correct. The library in use will only take relative URLs, or fully
> qualified URLs (including protocol, port, and host name).
>
> Relative URLs means I write back into the web application, which is where
> getRealPath() comes into play.
>
> I cannot use absolute URLs (/context/servlet-name?imagename=foo). This gets
> interpreted by the library as an absolute file name (and no, using
> /java.io.tmpdir/foo does not work).
>
> I would love to rewrite my URL to be
>
> http://localhost:80/context/servlet-name?imagename=foo
>
> That will only work if there are no virtual hosts in any front end proxy and
> no additional hosts in server.xml.
>
> I could write a separate application that serves the image files internally.
> It would be quite simple to do, but it would potentially require server
> modifications.
>
> 1. Listens to requests on a separate port (server.xml modification)
> 2. Application resides in the default host (for multi-host configs)
>
> Then the path would look something like the following.
>
> A. Request
>
> Browser ------------> NAT ------------> Server
> public IP private IP
> ww.xx.yy.zz aa.bb.cc.dd
>
> B. Image
>
> +-----< Server
> | ^
> | | localhost - some arbitrary port
> +---------+
>
> C. Resulting PDF
>
> Browser <------------ NAT <------------ Server
> public IP private IP
> ww.xx.yy.zz aa.bb.cc.dd
>
That PDF-generating library - where it sits? Is it on Server?
How does the library get its source data (an XSL-FO document?) that is
converted to PDF? Which of the following:
a) It uses HTTP to request a JSP page. Then it uses HTTP to request the image.
b) You prepare the data and feed it to the library programmatically
(e.g. as an InputStream). The library performs a request to get the
image.
If it is a) you could use UrlRewriteFilter + forward to serve the
image from any arbitrary URL.
If it is b),
- How the library knows the base URL of the document, to resolve relative links?
- Could you use a "file:///" URL for the image?
- Does the library use some resolver before looking for the image
(e.g. javax.xml.stream.XMLResolver, org.xml.sax.EntityResolver),
so that you can intercept the lookup.
- In theory, it is possible to use custom URL schemes, like Tomcat's
internal jndi:// one.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Mark Eggers <it...@yahoo.com>.
Konstantin,
Thanks for the comments. I'll make mine inline below.
On 8/24/2012 2:55 PM, Konstantin Kolinko wrote:
> 2012/8/25 Mark Eggers <it...@yahoo.com>:
>> Folks,
>>
>> This is not strictly a Tomcat problem, but since the application is running
>> under Tomcat I thought I would ask.
>>
>> Environment
>> -----------
>> Tomcat: 6.0.20
>> JRE: 1.6.0_16
>> OS: some UNIX variant
>>
>> I'm a little light on the environment, since it's not mine. Yes, this is
>> old. No, I don't have any leverage to get them to update.
>>
>> Background
>> ----------
>>
>> An application makes use of a PDF library which generates PDFs from JSP
>> pages. One of these pages includes a dynamically generated graph.
>>
>> This library cannot use streaming data to include the graph. The only two
>> options are:
>>
>> 1. Write the graph within the web application and use relative paths
>> 2. Write the graph somewhere else and:
>> a. use a fully qualified URL point to a servlet
>> b. the servlet retrieves the graph by name
>>
>> Ideally, we write the JPG file to java.io.tmpdir, then reference a servlet
>> in the JSP to fetch the graph and insert it into the page.
>>
>> Considerations
>> --------------
>>
>> Unfortunately, in certain environments we cannot use solution 2. One of
>> these environments involves a NAT and hairpinning. In those cases, we use
>> solution 1.
>>
>> We use getRealPath in a listener, tag on a directory to write images, set
>> the path in a singleton, and we're done. We can then use a relative
>> directory in the JSP to get the image and generate the PDF.
>>
>> If for any reason getRealPath returns a null, we fall back to using
>> java.io.tmpdir.
>>
>>
>> Problem
>> -------
>>
>> I ran into a situation where getRealPath with the above environment returns
>> a null, even when the WAR file is exploded.
>>
>> This means that we fall back to java.io.tmpdir, and run into the hairpinning
>> problem.
>>
>> Question
>> --------
>>
>> What other reasons could there be for getRealPath to return a null rather
>> than the path where the application is run?
>>
>> Some other Info
>> ---------------
>>
>> They are not running under a security manager. ps shows no security manager
>> arguments, and at any rate the application would throw a bunch of other
>> security violations and fail to start (a few getReourceAsStream issues in
>> listeners).
>>
>> I noticed that the server appears to be running from an NFS mount. However,
>> the exploded WAR file directory is marked as 755, and the ownership is
>> correct.
>>
>> About the only thing that comes to mind is NSF3 / NSF4 ACLs. I won't know
>> more about the server environment until Monday.
>>
>> If anyone has ideas, I'd be happy to hear about them.
>>
>
> 1. One known case is that getRealPath() will return null if the path
> does not actually exist.
>
> (In that case getResource() will return null as well).
>
> Does getRealPath("/") work?
I'll put together a WAR file and test on Monday. Unfortunately they're
East Coast USA time, which means it's 6 PM as I write this.
>
> 2. If you are guessing on ACL... can you try whether
> new File(System.getProperty("catalina.base")).getCanonicalPath()
> call succeeds?
>
> I think that I heard that getCanonicalPath() can fail if some parent
> directories do not have enough permissions on them.
>
I'll try this as well. I'll stuff it in the same listener, and log the
results. Fortunately I log my listener stuff in a separate file, so it's
pretty easy to see what failed and why.
> ============
> 3. What is your hairpinning problem?
>
Hairpinning is when a system is sitting behind a NAT, and it attempts to
send itself a packet to the public IP address.
+----------------<
| NAT <----------------> Server
+---------------->
public IP private IP
ww.xx.yy.zz aa.bb.cc.dd
In this environment it occurs with a host name. The server sends a
packet to itself using the publicly accessible name, which resolves to
the external IP address (ww.xx.yy.zz), which then gets sent right back
to the server.
Unless you specifically configure a NAT device to allow this, the packet
will be dropped.
> Is it because you use "a fully qualified URL point to a servlet" and
> you cannot build that URL correctly?
>
> Do you really need an absolute URL? You mean that you cannot call your
> servlet otherwise?
This is correct. The library in use will only take relative URLs, or
fully qualified URLs (including protocol, port, and host name).
Relative URLs means I write back into the web application, which is
where getRealPath() comes into play.
I cannot use absolute URLs (/context/servlet-name?imagename=foo). This
gets interpreted by the library as an absolute file name (and no, using
/java.io.tmpdir/foo does not work).
I would love to rewrite my URL to be
http://localhost:80/context/servlet-name?imagename=foo
That will only work if there are no virtual hosts in any front end proxy
and no additional hosts in server.xml.
I could write a separate application that serves the image files
internally. It would be quite simple to do, but it would potentially
require server modifications.
1. Listens to requests on a separate port (server.xml modification)
2. Application resides in the default host (for multi-host configs)
Then the path would look something like the following.
A. Request
Browser ------------> NAT ------------> Server
public IP private IP
ww.xx.yy.zz aa.bb.cc.dd
B. Image
+-----< Server
| ^
| | localhost - some arbitrary port
+---------+
C. Resulting PDF
Browser <------------ NAT <------------ Server
public IP private IP
ww.xx.yy.zz aa.bb.cc.dd
>
> Best regards,
> Konstantin Kolinko
Thanks for the thoughts. I'll construct a test WAR file and see how fast
I can get them to deploy it.
I like my separate application solution as the most generic. It also
keeps me from writing back into the web application (something I hate to
do). I just don't know how flexible they'll be in adding this complexity
to their environment.
. . . . just my two cents.
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: getRealPath returns null
Posted by Konstantin Kolinko <kn...@gmail.com>.
2012/8/25 Mark Eggers <it...@yahoo.com>:
> Folks,
>
> This is not strictly a Tomcat problem, but since the application is running
> under Tomcat I thought I would ask.
>
> Environment
> -----------
> Tomcat: 6.0.20
> JRE: 1.6.0_16
> OS: some UNIX variant
>
> I'm a little light on the environment, since it's not mine. Yes, this is
> old. No, I don't have any leverage to get them to update.
>
> Background
> ----------
>
> An application makes use of a PDF library which generates PDFs from JSP
> pages. One of these pages includes a dynamically generated graph.
>
> This library cannot use streaming data to include the graph. The only two
> options are:
>
> 1. Write the graph within the web application and use relative paths
> 2. Write the graph somewhere else and:
> a. use a fully qualified URL point to a servlet
> b. the servlet retrieves the graph by name
>
> Ideally, we write the JPG file to java.io.tmpdir, then reference a servlet
> in the JSP to fetch the graph and insert it into the page.
>
> Considerations
> --------------
>
> Unfortunately, in certain environments we cannot use solution 2. One of
> these environments involves a NAT and hairpinning. In those cases, we use
> solution 1.
>
> We use getRealPath in a listener, tag on a directory to write images, set
> the path in a singleton, and we're done. We can then use a relative
> directory in the JSP to get the image and generate the PDF.
>
> If for any reason getRealPath returns a null, we fall back to using
> java.io.tmpdir.
>
>
> Problem
> -------
>
> I ran into a situation where getRealPath with the above environment returns
> a null, even when the WAR file is exploded.
>
> This means that we fall back to java.io.tmpdir, and run into the hairpinning
> problem.
>
> Question
> --------
>
> What other reasons could there be for getRealPath to return a null rather
> than the path where the application is run?
>
> Some other Info
> ---------------
>
> They are not running under a security manager. ps shows no security manager
> arguments, and at any rate the application would throw a bunch of other
> security violations and fail to start (a few getReourceAsStream issues in
> listeners).
>
> I noticed that the server appears to be running from an NFS mount. However,
> the exploded WAR file directory is marked as 755, and the ownership is
> correct.
>
> About the only thing that comes to mind is NSF3 / NSF4 ACLs. I won't know
> more about the server environment until Monday.
>
> If anyone has ideas, I'd be happy to hear about them.
>
1. One known case is that getRealPath() will return null if the path
does not actually exist.
(In that case getResource() will return null as well).
Does getRealPath("/") work?
2. If you are guessing on ACL... can you try whether
new File(System.getProperty("catalina.base")).getCanonicalPath()
call succeeds?
I think that I heard that getCanonicalPath() can fail if some parent
directories do not have enough permissions on them.
============
3. What is your hairpinning problem?
Is it because you use "a fully qualified URL point to a servlet" and
you cannot build that URL correctly?
Do you really need an absolute URL? You mean that you cannot call your
servlet otherwise?
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org