You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Sander Hoentjen <sa...@hoentjen.eu> on 2017/03/24 13:10:05 UTC
mod_http2 file uploads slow/broken?
Hi,
I am running Apache 2.4.25 with mod_http2, and I notice that sometimes
file uploads are broken.
Receiving end is a php script, and it logs something like:
Internal error on sending request(POST /upload/upload.php HTTP/2.0);
uri(/upload/upload.php) content-length(931728): SendRequest: prepare():
user_get_body(bodyLocalBuf, 36865): read from client failed
With HTTP/1.1 this problem does not occur. With HTTP/2 this problems
occurs sometimes, but not always. What I do notice though is that
uploads via HTTP/2 are *much* slower, about 20-30s on HTTP/2 vs 4-5s on
HTTP/1.1 for the exact same file (about 15MB)
Is this a known issue? If not, anything I can do to help?
Regards,
Sander
Re: mod_http2 file uploads slow/broken?
Posted by Sander Hoentjen <sa...@hoentjen.eu>.
Hi Stefan,
With 1.9.3 file upload seems to complete every time just fine, so looks
like graceful restarts were indeed the issue. Thanks!
As to the speed, yes it needs to be improved but there is also some
weird thing:
For my testfile of about 14M,
uploads over http take about 20s
uploads over https take about 15s
(with http1.1 both take about 5s)
In my setup SSL is terminated by HAProxy in TCP mode, so Apache gets
non-SSL in both case, only in the SSL case prepended py proxy-protocol info.
Regards,
Sander
On 03/24/2017 08:14 PM, Stefan Eissing wrote:
> Hi Sander,
>
> from the logs, I think you are running into PR 59348 which was fixed in v1.8.11.
>
> The bug was that during graceful restart, ongoing streams were not finished. This restart could also happen, I think, if you have configured a max number of connections or requests per process, for example with MaxConnectionsPerChild. If that is the case, you can set this to 0 and see if the upload still fails.
>
> More interesting would be to know if the current github version has the problem still. With the fix, I hope it finishes the ongoing h2 requests before the process exits.
>
> As to the speed: it is as I described earlier, the data is written in too small chunks to the request processing thread. That needs to be improved for sure.
>
> Thanks for testing, awaiting your results. If you need help with building the github version, please let me know.
>
> Cheers,
>
> Stefan
>
>> Am 24.03.2017 um 16:56 schrieb Sander Hoentjen <sa...@hoentjen.eu>:
>>
>> Hi Stefan,
>>
>> So far I can't reproduce the breaking of the upload on a testing
>> machine. I send you logs from a production machine off-list. In the
>> meanwhile I will try to build mod_http2 from
>> https://github.com/icing/mod_h2.
>>
>> Thanks for your help!
>>
>> Sander
>>
>>
>>
>> On 03/24/2017 03:38 PM, Stefan Eissing wrote:
>>> Hi Sander,
>>>
>>> the uploads sometimes break is new to me and I sure would like
>>> to find out what is going wrong in your setup. One obvious reason
>>> for the php script failing to read its input is a timeout or
>>> abort of the main connection. mod_http2 uses the general Timeout
>>> settings for its requests as well. If you can reproduce this,
>>> a log with "LogLevel http2:trace1" or even trace2 would help.
>>>
>>> I know that input streaming is not very optimal right now. Each
>>> arriving DATA frame is send directly onward to the request thread
>>> and that should be buffered and flushed at proper times. Again,
>>> a log at trace1/2 level should show exactly what is going on.
>>>
>>> If you can, I'd be interested to hear how the current version,
>>> available at https://github.com/icing/mod_h2 fares in you 2.4.25
>>> server. If you are on Windows, apachelounge has also builds with
>>> the latest version packaged in.
>>>
>>> Cheers,
>>>
>>> Stefan
>>>
>>>> Am 24.03.2017 um 14:10 schrieb Sander Hoentjen <sa...@hoentjen.eu>:
>>>>
>>>> Hi,
>>>>
>>>> I am running Apache 2.4.25 with mod_http2, and I notice that sometimes
>>>> file uploads are broken.
>>>>
>>>> Receiving end is a php script, and it logs something like:
>>>> Internal error on sending request(POST /upload/upload.php HTTP/2.0);
>>>> uri(/upload/upload.php) content-length(931728): SendRequest: prepare():
>>>> user_get_body(bodyLocalBuf, 36865): read from client failed
>>>>
>>>> With HTTP/1.1 this problem does not occur. With HTTP/2 this problems
>>>> occurs sometimes, but not always. What I do notice though is that
>>>> uploads via HTTP/2 are *much* slower, about 20-30s on HTTP/2 vs 4-5s on
>>>> HTTP/1.1 for the exact same file (about 15MB)
>>>>
>>>> Is this a known issue? If not, anything I can do to help?
>>>>
>>>> Regards,
>>>> Sander
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 file uploads slow/broken?
Posted by Stefan Eissing <st...@greenbytes.de>.
Hi Sander,
from the logs, I think you are running into PR 59348 which was fixed in v1.8.11.
The bug was that during graceful restart, ongoing streams were not finished. This restart could also happen, I think, if you have configured a max number of connections or requests per process, for example with MaxConnectionsPerChild. If that is the case, you can set this to 0 and see if the upload still fails.
More interesting would be to know if the current github version has the problem still. With the fix, I hope it finishes the ongoing h2 requests before the process exits.
As to the speed: it is as I described earlier, the data is written in too small chunks to the request processing thread. That needs to be improved for sure.
Thanks for testing, awaiting your results. If you need help with building the github version, please let me know.
Cheers,
Stefan
> Am 24.03.2017 um 16:56 schrieb Sander Hoentjen <sa...@hoentjen.eu>:
>
> Hi Stefan,
>
> So far I can't reproduce the breaking of the upload on a testing
> machine. I send you logs from a production machine off-list. In the
> meanwhile I will try to build mod_http2 from
> https://github.com/icing/mod_h2.
>
> Thanks for your help!
>
> Sander
>
>
>
> On 03/24/2017 03:38 PM, Stefan Eissing wrote:
>> Hi Sander,
>>
>> the uploads sometimes break is new to me and I sure would like
>> to find out what is going wrong in your setup. One obvious reason
>> for the php script failing to read its input is a timeout or
>> abort of the main connection. mod_http2 uses the general Timeout
>> settings for its requests as well. If you can reproduce this,
>> a log with "LogLevel http2:trace1" or even trace2 would help.
>>
>> I know that input streaming is not very optimal right now. Each
>> arriving DATA frame is send directly onward to the request thread
>> and that should be buffered and flushed at proper times. Again,
>> a log at trace1/2 level should show exactly what is going on.
>>
>> If you can, I'd be interested to hear how the current version,
>> available at https://github.com/icing/mod_h2 fares in you 2.4.25
>> server. If you are on Windows, apachelounge has also builds with
>> the latest version packaged in.
>>
>> Cheers,
>>
>> Stefan
>>
>>> Am 24.03.2017 um 14:10 schrieb Sander Hoentjen <sa...@hoentjen.eu>:
>>>
>>> Hi,
>>>
>>> I am running Apache 2.4.25 with mod_http2, and I notice that sometimes
>>> file uploads are broken.
>>>
>>> Receiving end is a php script, and it logs something like:
>>> Internal error on sending request(POST /upload/upload.php HTTP/2.0);
>>> uri(/upload/upload.php) content-length(931728): SendRequest: prepare():
>>> user_get_body(bodyLocalBuf, 36865): read from client failed
>>>
>>> With HTTP/1.1 this problem does not occur. With HTTP/2 this problems
>>> occurs sometimes, but not always. What I do notice though is that
>>> uploads via HTTP/2 are *much* slower, about 20-30s on HTTP/2 vs 4-5s on
>>> HTTP/1.1 for the exact same file (about 15MB)
>>>
>>> Is this a known issue? If not, anything I can do to help?
>>>
>>> Regards,
>>> Sander
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>
>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 file uploads slow/broken?
Posted by Sander Hoentjen <sa...@hoentjen.eu>.
Hi Stefan,
So far I can't reproduce the breaking of the upload on a testing
machine. I send you logs from a production machine off-list. In the
meanwhile I will try to build mod_http2 from
https://github.com/icing/mod_h2.
Thanks for your help!
Sander
On 03/24/2017 03:38 PM, Stefan Eissing wrote:
> Hi Sander,
>
> the uploads sometimes break is new to me and I sure would like
> to find out what is going wrong in your setup. One obvious reason
> for the php script failing to read its input is a timeout or
> abort of the main connection. mod_http2 uses the general Timeout
> settings for its requests as well. If you can reproduce this,
> a log with "LogLevel http2:trace1" or even trace2 would help.
>
> I know that input streaming is not very optimal right now. Each
> arriving DATA frame is send directly onward to the request thread
> and that should be buffered and flushed at proper times. Again,
> a log at trace1/2 level should show exactly what is going on.
>
> If you can, I'd be interested to hear how the current version,
> available at https://github.com/icing/mod_h2 fares in you 2.4.25
> server. If you are on Windows, apachelounge has also builds with
> the latest version packaged in.
>
> Cheers,
>
> Stefan
>
>> Am 24.03.2017 um 14:10 schrieb Sander Hoentjen <sa...@hoentjen.eu>:
>>
>> Hi,
>>
>> I am running Apache 2.4.25 with mod_http2, and I notice that sometimes
>> file uploads are broken.
>>
>> Receiving end is a php script, and it logs something like:
>> Internal error on sending request(POST /upload/upload.php HTTP/2.0);
>> uri(/upload/upload.php) content-length(931728): SendRequest: prepare():
>> user_get_body(bodyLocalBuf, 36865): read from client failed
>>
>> With HTTP/1.1 this problem does not occur. With HTTP/2 this problems
>> occurs sometimes, but not always. What I do notice though is that
>> uploads via HTTP/2 are *much* slower, about 20-30s on HTTP/2 vs 4-5s on
>> HTTP/1.1 for the exact same file (about 15MB)
>>
>> Is this a known issue? If not, anything I can do to help?
>>
>> Regards,
>> Sander
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 file uploads slow/broken?
Posted by Stefan Eissing <st...@greenbytes.de>.
Hi Sander,
the uploads sometimes break is new to me and I sure would like
to find out what is going wrong in your setup. One obvious reason
for the php script failing to read its input is a timeout or
abort of the main connection. mod_http2 uses the general Timeout
settings for its requests as well. If you can reproduce this,
a log with "LogLevel http2:trace1" or even trace2 would help.
I know that input streaming is not very optimal right now. Each
arriving DATA frame is send directly onward to the request thread
and that should be buffered and flushed at proper times. Again,
a log at trace1/2 level should show exactly what is going on.
If you can, I'd be interested to hear how the current version,
available at https://github.com/icing/mod_h2 fares in you 2.4.25
server. If you are on Windows, apachelounge has also builds with
the latest version packaged in.
Cheers,
Stefan
> Am 24.03.2017 um 14:10 schrieb Sander Hoentjen <sa...@hoentjen.eu>:
>
> Hi,
>
> I am running Apache 2.4.25 with mod_http2, and I notice that sometimes
> file uploads are broken.
>
> Receiving end is a php script, and it logs something like:
> Internal error on sending request(POST /upload/upload.php HTTP/2.0);
> uri(/upload/upload.php) content-length(931728): SendRequest: prepare():
> user_get_body(bodyLocalBuf, 36865): read from client failed
>
> With HTTP/1.1 this problem does not occur. With HTTP/2 this problems
> occurs sometimes, but not always. What I do notice though is that
> uploads via HTTP/2 are *much* slower, about 20-30s on HTTP/2 vs 4-5s on
> HTTP/1.1 for the exact same file (about 15MB)
>
> Is this a known issue? If not, anything I can do to help?
>
> Regards,
> Sander
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de