You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Christopher Zinn <ch...@gmail.com> on 2017/01/08 22:04:10 UTC
Tomcat 8, 8.5 and 9 returning another apps environment with
context.lookup() from a .parallelStream()
Hello all,
This is my first time posting to a mailing list so hopefully I'm doing
this correctly.
We are running into an issue where we have multiple copies of the same
WAR loaded on a Tomcat instance each with its own context.xml.
The initial problem I was trying to diagnose was A JNDI lookup to a
Connection Pool in one of the WARs was returning the connection pool of one
of the others.
The problem only happens when the JNDI lookup is performed within a
parallelStream().
I was able to produce a very simple WAR with a single servlet and a
context XML to reproduce the problem I'm having (See below).
If you run the servlet from the first application, it works correctly.
You will see that it only ever looks up 'Test 1'.
When I run the servlet from the second application it will only return Test
1 in the first part (stream()) but a mix of 'Test 1' and 'Test 2' in the
parallelStream() part.
Thanks in advance for any advice or comments to our issue.
-----------------------------------------------
My Environment: (The problem happens to us on our CENTOS Linux
environments as well)
Server version: Apache Tomcat/8.5.9
Server built: Dec 5 2016 20:18:12 UTC
Server number: 8.5.9.0
OS Name: Windows 10
OS Version: 10.0
Architecture: amd64
JVM Version: 1.8.0_111-b14
JVM Vendor: Oracle Corporation
Runtime.getRuntime().availableProcessors() returns 12 on my machine.
CPU: Intel i7-5820K
My Test Servlet:
@WebServlet(urlPatterns = "/test")
public final class MyServlet extends HttpServlet
{
private static List<Integer> someNumbers = generateNonsense();
@Override
protected void doGet(HttpServletRequest request, HttpServletResponse
response) throws ServletException, IOException
{
StringBuilder answerBuilder = new StringBuilder();
// This should be okay.
someNumbers.stream().forEach( number ->
{
answerBuilder.append(".stream() Looked up ")
.append(lookupEnvironmentValue())
.append("<br/>");
});
// This is most likely bad
someNumbers.parallelStream().forEach( number ->
{
synchronized (MyServlet.class)
{
answerBuilder.append(".parallelStream() Looked up ")
.append(lookupEnvironmentValue())
.append("<br/>");
}
});
response.setContentType("text/html");
response.getWriter().write(answerBuilder.toString());
}
private String lookupEnvironmentValue()
{
try
{
Context context = new InitialContext();
return (String) context.lookup("java:comp/env/testName");
}
catch(NamingException e)
{
e.printStackTrace();
return e.getMessage();
}
}
private static List<Integer> generateNonsense()
{
List<Integer> nonsense = new ArrayList<>();
for(int n=0; n<1000; n++)
nonsense.add(n);
return nonsense;
}
}
My test context.xml
<Context path="/test1">
<Environment name="testName" value="Test 1" type="java.lang.String"
override="false"/>
</Context>
Preparing Tomcat:
Step 1: Download the core.zip of Tomcat 8, 8.5 or 9
Step 2: create a apache-tomcat-8.5.9/conf/Catalina/localhost directory
Step 3: create two files in this directory with the contents of my
context.xml: test1.xml and test2.xml and change the value and path in the
second one to '2'
Step 4: Create a WAR with just the sample servlet I have here. and then
drop them in apache-tomcat-8.5.9/webapps directory / call them test1.war
and test2.war
Step 5: Go into apache-tomcat-8.5.9/bin and catalina start to start it up.
The urls to test for me are:
http://localhost:8080/test1/test
http://localhost:8080/test2/test
Re: Tomcat 8, 8.5 and 9 returning another apps environment with
context.lookup() from a .parallelStream()
Posted by Mark Thomas <ma...@apache.org>.
On 09/01/2017 19:57, Christopher Zinn wrote:
> Thanks for the information. I was not expecting such a great response so
> quickly.
>
> We are going to test the workaround that sets the java.util.concurrent.
> ForkJoinPool.common.threadFactory property as that will be the easiest for
> us to configure and deploy to our Tomcat instances.
The correct value for that property is:
java.util.concurrent.ForkJoinPool$ForkJoinTaskFactory
As I dig into this further, I realise it isn't that simple.
1. There will still be a memory leak.
2. The threads will have the wrong thread context class loader and your
JNDI lookups will fail.
3. ForEach is hard-coded to use the common ForkJoin pool.
As far as I can see, you only option is to correctly set the thread
context class before you do the JNDI lookup. Because of the way the
ForkJoinPool is written that is going to prevent the code ever running
under a SecurityManager (which may not be an issue for you) because
InnocuousForkJoinWorkerThread is hard-coded to throw an Exception if you
call setContextClassLoader().
It looks like ForkJoin was not designed with a multi-class loader
environment in mind. We should be able to get most of this fixed but
without adding significant complexity, the end result is going to be
that ForkJoinWorkerThreads always use the system class loader. That
means if you want to use JNDI, you are going to have to set the thread
context class loader yourself.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 8, 8.5 and 9 returning another apps environment with
context.lookup() from a .parallelStream()
Posted by Christopher Zinn <ch...@gmail.com>.
Thanks for the information. I was not expecting such a great response so
quickly.
We are going to test the workaround that sets the java.util.concurrent.
ForkJoinPool.common.threadFactory property as that will be the easiest for
us to configure and deploy to our Tomcat instances.
- Chris
On Mon, Jan 9, 2017 at 11:57 AM, Mark Thomas <ma...@apache.org> wrote:
> On 09/01/2017 14:29, Mark Thomas wrote:
> > On 08/01/2017 22:04, Christopher Zinn wrote:
>
> <snip/>
>
> >> We are running into an issue where we have multiple copies of the
> same
> >> WAR loaded on a Tomcat instance each with its own context.xml.
> >> The initial problem I was trying to diagnose was A JNDI lookup to a
> >> Connection Pool in one of the WARs was returning the connection pool of
> one
> >> of the others.
> >> The problem only happens when the JNDI lookup is performed within a
> >> parallelStream().
> >>
> >> I was able to produce a very simple WAR with a single servlet and a
> >> context XML to reproduce the problem I'm having (See below).
> >>
> >> If you run the servlet from the first application, it works
> correctly.
> >> You will see that it only ever looks up 'Test 1'.
> >> When I run the servlet from the second application it will only return
> Test
> >> 1 in the first part (stream()) but a mix of 'Test 1' and 'Test 2' in the
> >> parallelStream() part.
>
> <snip/>
>
> > A quick look at the JRE source code suggests this is handled correctly
> > but the code isn't the easiest to trace through. I'll set up the
> > provided test case and take a closer look. That may take an hour or two.
>
> I can confirm that the root cause is a JRE bug. Unless you are running
> under a SecurityManager the default ForkJoinWorkerThreadFactory does not
> take any steps to ensure correct operation in a multi-class loader
> environment.
>
> You have a couple of work-arounds available
> - Run under a SecurityManager
> - Set the java.util.concurrent.ForkJoinPool.common.threadFactory system
> property to
> java.util.concurrent.ForkJoinPool.InnocuousForkJoinWorkerThreadFactory
>
> Meanwhile, I'll be doing the following:
>
> - Updating Tomcat's JreMemoryLeakPreventionListener to include
> protection for this memory leak.
> - Adding this leak to https://github.com/markt-asf/memory-leaks
> - Raising a JRE bug
> - Pinging our friendly Oracle contact to get this fixed (we've been
> pretty successful at getting this fixed recently)
>
> Cheers,
>
> Mark
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat 8, 8.5 and 9 returning another apps environment with
context.lookup() from a .parallelStream()
Posted by Mark Thomas <ma...@apache.org>.
On 09/01/2017 14:29, Mark Thomas wrote:
> On 08/01/2017 22:04, Christopher Zinn wrote:
<snip/>
>> We are running into an issue where we have multiple copies of the same
>> WAR loaded on a Tomcat instance each with its own context.xml.
>> The initial problem I was trying to diagnose was A JNDI lookup to a
>> Connection Pool in one of the WARs was returning the connection pool of one
>> of the others.
>> The problem only happens when the JNDI lookup is performed within a
>> parallelStream().
>>
>> I was able to produce a very simple WAR with a single servlet and a
>> context XML to reproduce the problem I'm having (See below).
>>
>> If you run the servlet from the first application, it works correctly.
>> You will see that it only ever looks up 'Test 1'.
>> When I run the servlet from the second application it will only return Test
>> 1 in the first part (stream()) but a mix of 'Test 1' and 'Test 2' in the
>> parallelStream() part.
<snip/>
> A quick look at the JRE source code suggests this is handled correctly
> but the code isn't the easiest to trace through. I'll set up the
> provided test case and take a closer look. That may take an hour or two.
I can confirm that the root cause is a JRE bug. Unless you are running
under a SecurityManager the default ForkJoinWorkerThreadFactory does not
take any steps to ensure correct operation in a multi-class loader
environment.
You have a couple of work-arounds available
- Run under a SecurityManager
- Set the java.util.concurrent.ForkJoinPool.common.threadFactory system
property to
java.util.concurrent.ForkJoinPool.InnocuousForkJoinWorkerThreadFactory
Meanwhile, I'll be doing the following:
- Updating Tomcat's JreMemoryLeakPreventionListener to include
protection for this memory leak.
- Adding this leak to https://github.com/markt-asf/memory-leaks
- Raising a JRE bug
- Pinging our friendly Oracle contact to get this fixed (we've been
pretty successful at getting this fixed recently)
Cheers,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 8, 8.5 and 9 returning another apps environment with
context.lookup() from a .parallelStream()
Posted by Mark Thomas <ma...@apache.org>.
On 08/01/2017 22:04, Christopher Zinn wrote:
> Hello all,
> This is my first time posting to a mailing list so hopefully I'm doing
> this correctly.
Welcome to the Apache Tomcat community.
As problem reports go, this is pretty much perfect. And it includes a
simple test case. Many, many thanks.
> We are running into an issue where we have multiple copies of the same
> WAR loaded on a Tomcat instance each with its own context.xml.
> The initial problem I was trying to diagnose was A JNDI lookup to a
> Connection Pool in one of the WARs was returning the connection pool of one
> of the others.
> The problem only happens when the JNDI lookup is performed within a
> parallelStream().
>
> I was able to produce a very simple WAR with a single servlet and a
> context XML to reproduce the problem I'm having (See below).
>
> If you run the servlet from the first application, it works correctly.
> You will see that it only ever looks up 'Test 1'.
> When I run the servlet from the second application it will only return Test
> 1 in the first part (stream()) but a mix of 'Test 1' and 'Test 2' in the
> parallelStream() part.
>
> Thanks in advance for any advice or comments to our issue.
My initial comments are that this is rather odd. The correct JNDI
context is looked up via the thread context class loader. The class
loader from one web application should not be visible to another. I
wonder if when parallelStream() is first called some form of shared
thread-pool is set up that retains a reference to the current thread
context class loader?
A quick look at the JRE source code suggests this is handled correctly
but the code isn't the easiest to trace through. I'll set up the
provided test case and take a closer look. That may take an hour or two.
Mark
>
> -----------------------------------------------
> My Environment: (The problem happens to us on our CENTOS Linux
> environments as well)
> Server version: Apache Tomcat/8.5.9
> Server built: Dec 5 2016 20:18:12 UTC
> Server number: 8.5.9.0
> OS Name: Windows 10
> OS Version: 10.0
> Architecture: amd64
> JVM Version: 1.8.0_111-b14
> JVM Vendor: Oracle Corporation
>
> Runtime.getRuntime().availableProcessors() returns 12 on my machine.
> CPU: Intel i7-5820K
>
>
> My Test Servlet:
> @WebServlet(urlPatterns = "/test")
> public final class MyServlet extends HttpServlet
> {
> private static List<Integer> someNumbers = generateNonsense();
>
> @Override
> protected void doGet(HttpServletRequest request, HttpServletResponse
> response) throws ServletException, IOException
> {
> StringBuilder answerBuilder = new StringBuilder();
>
> // This should be okay.
> someNumbers.stream().forEach( number ->
> {
> answerBuilder.append(".stream() Looked up ")
> .append(lookupEnvironmentValue())
> .append("<br/>");
> });
>
> // This is most likely bad
> someNumbers.parallelStream().forEach( number ->
> {
> synchronized (MyServlet.class)
> {
> answerBuilder.append(".parallelStream() Looked up ")
> .append(lookupEnvironmentValue())
> .append("<br/>");
> }
> });
>
> response.setContentType("text/html");
> response.getWriter().write(answerBuilder.toString());
> }
>
> private String lookupEnvironmentValue()
> {
> try
> {
> Context context = new InitialContext();
> return (String) context.lookup("java:comp/env/testName");
> }
> catch(NamingException e)
> {
> e.printStackTrace();
> return e.getMessage();
> }
> }
>
> private static List<Integer> generateNonsense()
> {
> List<Integer> nonsense = new ArrayList<>();
> for(int n=0; n<1000; n++)
> nonsense.add(n);
>
> return nonsense;
> }
> }
>
>
> My test context.xml
> <Context path="/test1">
> <Environment name="testName" value="Test 1" type="java.lang.String"
> override="false"/>
> </Context>
>
>
> Preparing Tomcat:
> Step 1: Download the core.zip of Tomcat 8, 8.5 or 9
> Step 2: create a apache-tomcat-8.5.9/conf/Catalina/localhost directory
> Step 3: create two files in this directory with the contents of my
> context.xml: test1.xml and test2.xml and change the value and path in the
> second one to '2'
> Step 4: Create a WAR with just the sample servlet I have here. and then
> drop them in apache-tomcat-8.5.9/webapps directory / call them test1.war
> and test2.war
> Step 5: Go into apache-tomcat-8.5.9/bin and catalina start to start it up.
>
> The urls to test for me are:
> http://localhost:8080/test1/test
> http://localhost:8080/test2/test
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org