You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by bu...@apache.org on 2017/03/24 17:43:14 UTC

[Bug 60918] New: Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

            Bug ID: 60918
           Summary: Process Send File releases the same Http11Processor in
                    two threads, leading to
                    java.lang.IllegalStateException: Unexpected state:
                    headers already parsed. Buffer not recycled?
           Product: Tomcat 8
           Version: 8.5.11
          Hardware: PC
            Status: NEW
          Severity: major
          Priority: P2
         Component: Catalina
          Assignee: dev@tomcat.apache.org
          Reporter: andrew.garland@sunbirddcim.com
  Target Milestone: ----

Created attachment 34876
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=34876&action=edit
Some pieces of the full trace showing the issue

This is a similar report to
https://bz.apache.org/bugzilla/show_bug.cgi?id=58646 but whatever fix was done
in that case does not prevent this issue.

Basic outline:

1. thread http-nio-8080-exec-1 processes a request that will result in send
file.  It pops Http11Processor@8dc1458 from recycledProcessors
2. The send file action is completed in thread http-nio-8080-ClientPoller-0. 
Then that thread is done, it pushes Http11Processor@8dc1458 onto
recycledProcessors
3. A few milliseconds later, http-nio-8080-exec-1 also pushes
Http11Processor@8dc1458 onto recycledProcessors, so now that one processor is
in the list of recycledProcessors twice
4. Eventually, two nio exec threads both pop that one instance off of
recycledProcessors

4A.  The second thread to use it throws

24-Mar-2017 12:08:08.093 INFO [http-nio-8080-exec-1]
org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request
header
 Note: further occurrences of HTTP header parsing errors will be logged at
DEBUG level.
 java.lang.IllegalStateException: Unexpected state: headers already parsed.
Buffer not recycled?
        at
org.apache.coyote.http11.Http11InputBuffer.parseHeaders(Http11InputBuffer.java:554)
        at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:683)


4B.  The first thread, which completes after the second one, throws

24-Mar-2017 12:08:08.833 SEVERE [http-nio-8080-exec-7]
org.apache.coyote.http11.Http11Processor.service Error processing request
 java.lang.NullPointerException
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:389)
        at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:783)

and then

24-Mar-2017 12:08:08.849 SEVERE [http-nio-8080-exec-7]
org.apache.coyote.http11.Http11Processor.endRequest Error finishing response
 java.lang.NullPointerException
        at
org.apache.coyote.http11.Http11OutputBuffer.commit(Http11OutputBuffer.java:351)
        at
org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1288)
        at
org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:254)
        at
org.apache.coyote.http11.Http11Processor.endRequest(Http11Processor.java:1457)
        at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:823)

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #10 from Mark Thomas <ma...@apache.org> ---
We'll probably start the release process next week. It is normally started
around the beginning of the every month but the first week of April is looking
busy for me. We could delay it but there is enough in the changelog to justify
a release.

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #5 from Andrew Garland <an...@sunbirddcim.com> ---
thanks for the info.

Can you point me to the commit diff?  Just curious.

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #14 from Jennifer Hinnant <ev...@gmail.com> ---
Branded event tech app by Eventeca.com will drive your sales sky- high & give
your audience the engagement they want with our customized & knowledgeable
event application solutions! Connect with us via call @ 1 888 649 5285

https://eventeca.com/check-in-app

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #6 from Mark Thomas <ma...@apache.org> ---
r1788544 (for 9.0.x)

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #8 from Mark Thomas <ma...@apache.org> ---
8.0.x and earlier are not affected. This bug was introduced by the refactoring
in 8.5.x onwards.

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

Mark Thomas <ma...@apache.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #12 from Mark Thomas <ma...@apache.org> ---
Fixed in:
- trunk for 9.0.0.M19 onwards
- 8.5.x for 8.5.13 onwards

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #2 from Mark Thomas <ma...@apache.org> ---
Thanks for the report and especially the debug logs. I was able to pin-point
the root cause very quickly from those.

Are you able to build Tomcat from svn in order to confirm a fix?

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #4 from Mark Thomas <ma...@apache.org> ---
These are the instructions for 9.0.x
https://svn.apache.org/viewvc/tomcat/trunk/BUILDING.txt?view=markup

8.5.x is very similar, just a different svn path and a different minimum Java
version.

Both 9.0.x and 8.5.x have the fix. I'm currently working on a back-port to
8.0.x.

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

Re: [Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

All,

This looks like spam to me.

Can someone with karma remove it?

- -chris

On 6/19/19 06:20, bugzilla@apache.org wrote:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=60918
> 
> --- Comment #13 from Castro B <ca...@gmx.com> --- I had
> also prevented the problem by re-coding the RecycledProcessors
> class to check for existence before pushing onto the stack.
> 
> Castro B, https://sparpedia.ch
> 
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl0KVkwACgkQHPApP6U8
pFhMCRAAt2sK9Dzxat7w7qgM5gO72UtBmlXDC6+9scHcnB8rvHQ0pjuIXFhhV/Mq
NiVLI7dGxJ3359iQacis7l9jOb6HKlB8mykslyuxd3kirXyhDaVutaoq15V62xT+
Z6+wWyK1CSYiDfC+j1FQy2NWdGcOa6YVFoPdl6lzMsAIHenOw/jkYd0MZd/2YeVX
EiTCIHAvAIfqnRiYVP/nTEnmChVvbfJGDF8lnfBFl+9cRjddxJ5U/SvcNNr+hl/F
4anEFyT74d5oX2FlXUoklxPRtAaluhs3UDBFEVooFw+lA8fMtawluxUHgiqG4TbQ
TM2ijZTtDC0JZyZTjxzqjln5tnguda88tRZc+doYbOoJQfckfs78fFsYzjsOmi0t
riFC0UN+KIHeVuNIYdmpmLW0aA2DZRGK6AYhSxm6YtJ7nKBIQ2HrWS4ErAOBOZHp
SF+w9dRHrxjmA7c8+Shhh/IrdoGqR1cOwEP0/qDPaZnwWU/avZ1K9JZSww27kvpn
RoflEMv1QOOLB9FDIjS2ga5i/olrmuIiXjZfwDCU6V9sUichS3ah3Efh+1CM0IxD
UObHuLIabGmSoMB9pHQdHa1dksmQK5OYj/J+iXQu0pTFNKk7l6pehrYT+/LhXU2y
pN2FuYOh4haSqMA9C/HZKx3MJoZXaiI3zuiz6YT4Es85spCyhlE=
=eDMZ
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #13 from Castro B <ca...@gmx.com> ---
I had also prevented the problem by re-coding the RecycledProcessors class to
check for existence before pushing onto the stack.

Castro B,
https://sparpedia.ch

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #9 from Andrew Garland <an...@sunbirddcim.com> ---
Created attachment 34877
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=34877&action=edit
Recoded RecycledProcessors

The problem does not recur using the version built from trunk!  Thanks for the
rapid turnaround.  I guess it will be a month or two before this is part of an
official release?

FYI, I had also prevented the problem by re-coding the RecycledProcessors class
to check for existence before pushing onto the stack.  This was quick and dirty
and would not scale, but food for thought.

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #11 from Remy Maucherat <re...@apache.org> ---
(In reply to Andrew Garland from comment #9)
> Created attachment 34877 [details]
> Recoded RecycledProcessors
> 
> The problem does not recur using the version built from trunk!  Thanks for
> the rapid turnaround.  I guess it will be a month or two before this is part
> of an official release?
> 
> FYI, I had also prevented the problem by re-coding the RecycledProcessors
> class to check for existence before pushing onto the stack.  This was quick
> and dirty and would not scale, but food for thought.

And we're not going to do that :)

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #7 from Andrew Garland <an...@sunbirddcim.com> ---
FYI, minor typo in changelog

subsequent requests experiencing and <code>IllegalStateException</code>.

"and" -> "an"

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #13 from yangshj <ya...@gmail.com> ---
Who can provide the test code for the recurrence of this bug

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

--- Comment #3 from Andrew Garland <an...@sunbirddcim.com> ---
i haven't before, but I am sure I could :)

Please point me at BUILD instructions.

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

[Bug 60918] Process Send File releases the same Http11Processor in two threads, leading to java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60918

Andrew Garland <an...@sunbirddcim.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 OS|                            |All

--- Comment #1 from Andrew Garland <an...@sunbirddcim.com> ---
To me, there are two separate issues here:

1.  The logic for pushing the processor back in send file case has some flaw.

2.  RecycledProcessors should protect against the same instance being in the
stack more than once, as a defense against this flaw or any future flaws that
result in mistakenly putting the same processor in twice.

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org