[Bug 56871] New: Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

            Bug ID: 56871
           Summary: Apparent corruption of Servlet InputStreams
           Product: Tomcat 7
           Version: 7.0.55
          Hardware: PC
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Connectors
          Assignee: dev@tomcat.apache.org
          Reporter: ciaranj@gmail.com

Created attachment 31930
  --> https://issues.apache.org/bugzilla/attachment.cgi?id=31930&action=edit
Test servlet (inc. test client node.js script)

Initially noted on a Tomcat 7.0.23 installation, running against J2SE7u4, but
also repeated against java 7u65 and Tomcat 7.0.55 and tomcat 8.0.9.

On my Windows Server 2012 installation it appears as though my InputStream is
being corrupted.

I traced it as best I could, and it *seems* that the
InternalNioInputBuffer$SocketInputBuffer is reading the data from the socket
'out of order' (very, very rarely)

My expectaction is that if I POST the exact same content to the server
repeatedly the server will always respond with the expected result.  (I accept
that response times may vary depending on system loads, GCing etc.)

My experience on a production application however is that if I continuously
load the server with POST requests (this is legacy code, so technically this is
SOAP requests being services by an xfire servlet) occassionally the data that
XFire attempts to process is corrupt.

I've attached a test servlet and node.js client that 'reliably' reproduce the
fault, this fault does occur with a concurrency of '1' but the rate of
occurrence appears to increase if I increase the concurrency value in
'client.js'.

As it may matter, this server has a Hyper-V installation on it, so there is a
'Hyper-V Virtual Ethernet Adapter' present and a physical 'Broadcom NetXTreme
57xx Gigabit Controller'.

I also tested running tomcat with a '-Djava.net.preferIPv4Stack=true' and the
issue remained.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

--- Comment #1 from Mark Thomas <ma...@apache.org> ---
I'm starting to look at this now. How reliably does this script reproduce the
issue. To put it another way, how long / how many requests before you start
seeing errors?

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

--- Comment #6 from Konstantin Kolinko <kn...@gmail.com> ---
> My text editor shows that "Lorem ipsum dolor" string is 3678 characters, not 3668.

My miscalculation. The difference in values is due to \r\n characters.

If I change the content-length value in client.js script, I get plenty of
SocketTimeoutExceptions on console and in localhost.2014-09-19.log,  so it
confirms that those exceptions are rather visible.

I am unable to reproduce this issue in my environment
- Tomcat 7.0.55, configured Nio connector, removed AJP connector and
AccessLogValve from configuration. Removed all standard applications except
ROOT one.
- Java 7u67 32-bit
- nodejs 0.10.32 64-bit
All running on the same Windows 7 machine.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

--- Comment #5 from Konstantin Kolinko <kn...@gmail.com> ---

> I traced it as best I could, and it *seems* that the
> InternalNioInputBuffer$SocketInputBuffer is reading the data
> from the socket 'out of order' (very, very rarely)

>From EchoServlet.java
>            if( !sb.toString().equals("a246722a276771371e1807a92f3bc86f") ) {
>              System.err.println("Unexpected data value: \n"+ new String(bytes));
>            }

1. I see that your code detects broken data by their MD5 digest and prints them
to System.err. Can you provide samples of such output? If data are read out of
order, it will be visible in the printed text.

> while( (len=sis.read(buffer, 0, 8*1024)) >0 ) {

2. Just as a note: My first impression was that I want ">=0" in the above line.
The end of data is signaled by value of "-1", not by 0.

On further reading of javadoc of InputStream.read(byte[]) I see that this
method has to block until at least one byte is read and is not allowed to
return "0".

3. ServletInputStream.read() is allowed to throw an IOException. E.g. it may
throw SocketTimeoutException.

>From client.js:
> var options = {
>  ...
>  headers: { "Content-Length": 3668}
>};
> ...
>  req.write('Lorem ipsum dolor sit amet [...]

4. So I guess that it uses explicit Content-Length length and not chunked
transfer-encoding. 
My text editor shows that "Lorem ipsum dolor" string is 3678 characters, not
3668.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

Ciaran <ci...@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ciaranj@gmail.com
                 OS|                            |All

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

Violeta Georgieva <vi...@apache.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |NEEDINFO

--- Comment #9 from Violeta Georgieva <vi...@apache.org> ---
Hi,

Please provide reproducible scenario.

Regards
Violeta

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

--- Comment #4 from Ciaran <ci...@gmail.com> ---
Sounds good to me.  One thing I'd also note is that when I was debugging the
paths through the code were 'different' for certain (again exactly the same
HTTP) requests and I couldn't replicate the fault if I added too much logging
(classic race!)

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

Mark Thomas <ma...@apache.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEEDINFO                    |RESOLVED
         Resolution|---                         |WORKSFORME

--- Comment #11 from Mark Thomas <ma...@apache.org> ---
After multiple extended tests (~60 mins) on multiple platforms both physical
and virtual no Tomcat committer has been able to repeat this issue. I am
therefore closing this as WORKSFORME.

If you are able to repeat this issue, please feel free to re-open and provide
the necessary steps to repeat this from a clean install of the the latest
stable Tomcat 7 (or later) release.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

--- Comment #3 from Mark Thomas <ma...@apache.org> ---
At this point I'm not that happy chalking this up to "someone else's problem".
What you have decribed could just as easily be a very rare timing error in
Tomcat's NIO code.

What I propose to do is run your test case on a varity of different systems and
see if I can reproduce it on any of them. If I can reproduce it then obviously
the next step is to track down the root cause (easy said than done) and then
fix it. If I can't reproduce this I'll close this as WORKSFORME but we can
always re-open it if someone else hits the same issue and can provide some new
information.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

--- Comment #7 from Mark Thomas <ma...@apache.org> ---
I can't repeat this on OSX either.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

--- Comment #12 from Ciaran <ci...@gmail.com> ---
I appreciate the effort, thank you.  I wish I could preserve my physical
environment in some manner to isolate this further, but my testing in
virtualised environments has had the same results as you :/

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

--- Comment #2 from Ciaran <ci...@gmail.com> ---
On my system the test exercised (and continues to) the fault reliably within a
couple of minutes.  However I've since been unable to replicate the failure on
clean virtual machines.

I'm happy to chalk it up to some local machine config issue (drivers perhaps)
if you are!

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

--- Comment #10 from Mark Thomas <ma...@apache.org> ---
I tested this yesterday on a Windows Server 2012 VM and could not repeat it
either.

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 56871] Apparent corruption of Servlet InputStreams

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=56871

--- Comment #8 from Mark Thomas <ma...@apache.org> ---
I also can't repeat this on:

- Physical Server 2008 R2, 64-bit Java 1.7.0_67

- Virtual (VMware Fusion) Ubuntu 14.04.1 LTS, 3.13.0-15, 64-bit Java 1.7.0_60

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org