You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Adam Retter <ad...@exist-db.org> on 2018/10/10 17:41:27 UTC
Broken pipe (Write failed) when making Unauthorized request
I am using the Maven httpclient 4.5.6 artifact and trying to make a PUT
request to a server. The server for some URLs requires authentication, it
is impossible to know which URLs do and don't ahead of time. I only want to
send the credentials if the server replies 401 Unauthorized.
I am trying to do a HTTP PUT to the server. If the server replies HTTP 401
Unauthorized, then I plan to retry with the authorization credentials.
I have found rather a strange issue, and I am not sure if the fault is with
the Apache HTTP Client or the server. However, I get different behaviour in
the Apache HTTP Client depending on how fast I can stream the data to the
server.
What I am seeing looks like this:
1. The Client opens a HTTP connection to the server
2. The Client starts sending data to the server
3. The Server starts receiving the data from the client, and determines
that the client is not authorized.
4. The Server responds HTTP 401 Unauthorized to the client
5. The server closes the connection.
At the client end when I only have a small amount of data to PUT, or use a
fast mechanism to write the data, all is fine and Apache HTTP Client gives
me the HTTP Response 401 from the server. All is good!
However, at the client end when I have a lot of data to PUT, or use a slow
mechanism to write the data, the Apache HTTP Client is still writting data
to the server when (5) occurs. The operating system then raises a "Broken
pipe (Write failed)" error, which Java relays to the Apache HTTP Client as
a SocketException. I then see errors like the following in my application:
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing
request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing
request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing
request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Exception in thread "main" java.net.SocketException: Broken pipe (Write
failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at
org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
at
org.apache.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:136)
at
org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:167)
at
org.apache.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:122)
at
org.apache.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:179)
at
org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:89)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
at java.io.Writer.write(Writer.java:157)
at Issue2186.copySlow(Issue2186.java:44)
at Issue2186.lambda$0(Issue2186.java:26)
at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:73)
at
org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
at
org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160)
at
org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at Issue2186.main(Issue2186.java:55)
So what is going on here? Is there something I am doing wrong on the client
side, or should the server be waiting to receive all the data from the
client before replying with the 401 Unauthorized response?
My simple code to reproduce the issue looks like:
public class Issue2186 {
private static final Path file = Paths.get("/tmp/mediumfile.xml");
private static HttpEntity getRequestBody() {
final ContentProducer producer = os -> {
// copyFast(file, os);
// NOTE: shows the broken pipe error
copySlow(file, os);
};
final EntityTemplate entityTemplate = new EntityTemplate(producer);
entityTemplate.setContentType("application/xml");
entityTemplate.setChunked(true);
return entityTemplate;
}
private static void copyFast(final Path file, final OutputStream os)
throws IOException {
Files.copy(file, os);
}
private static void copySlow(final Path file, final OutputStream os)
throws IOException {
try(final LineNumberReader reader = new
LineNumberReader(Files.newBufferedReader(file));
final OutputStreamWriter writer = new
OutputStreamWriter(new CloseShieldOutputStream(os), UTF_8)) {
String line;
while((line = reader.readLine()) != null) {
writer.write(line + "\n");
}
}
}
public static void main(final String args[]) throws IOException {
final CloseableHttpClient httpClient = HttpClients.createDefault();
try {
final HttpPut httpPut = new HttpPut("
http://localhost:8080/exist/rest/db/mediumfile.xml");
httpPut.setEntity(getRequestBody());
final CloseableHttpResponse httpResponse =
httpClient.execute(httpPut);
try {
System.out.println(httpResponse.getStatusLine());
} finally {
httpResponse.close();
}
} finally {
httpClient.close();
}
}
}
Thanks for your time.
--Adam Retter
eXist Core Developer
{ United Kingdom / United States }
adam@exist-db.org
Re: Broken pipe (Write failed) when making Unauthorized request
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2018-10-11 at 15:41 +0500, Adam Retter wrote:
> >
> > 1. Turn on 'expect-continue' handshake. It is intended precisely
> > for
> > such situations.
> >
>
> So I assume that for this I just set the HTTP Header, like so:
>
> httpPut.setHeader("Expect", "100-continue");
>
>
There is a RequestConfig option for that.
> If so, unfortunately that is not helping the request from the client
> now
> looks like:
>
> PUT /exist/rest/db/mediumfile.xml HTTP/1.1
> Expect: 100-continue
> Transfer-Encoding: chunked
> Content-Type: application/xml
> Host: localhost:8080
> Connection: Keep-Alive
> User-Agent: Apache-HttpClient/4.5.6 (Java/1.8.0_162)
> Accept-Encoding: gzip,deflate
>
> The server almost immediately responds with
>
> HTTP/1.1 100 Continue
>
> The client then starts sending data, during which, the server
> responds with:
>
> HTTP/1.1 401 Unauthorized
> Date: Thu, 11 Oct 2018 10:37:39 GMT
> WWW-Authenticate: Basic realm="exist"
> Content-Length: 0
> Server: Jetty(9.4.10.v20180503)
>
> The server then closes the connection. I then get the same Broken
> pipe
> (Write failed) exceptions back from the Apache HTTP Client.
>
> Does this mean that the server is also not behaving correctly? i.e.
> because
> it returns 100 before it checks the authorization stuff? or is that
> also
> allowed by the spec?
>
Unfortunately this is legal from the HTTP spec standpoint even though
this behavior is just plain silly.
I have pointed it out to the Jetty developers a long time ago but they
said they liked it that way.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Broken pipe (Write failed) when making Unauthorized request
Posted by Adam Retter <ad...@exist-db.org>.
>
> 1. Turn on 'expect-continue' handshake. It is intended precisely for
> such situations.
>
So I assume that for this I just set the HTTP Header, like so:
httpPut.setHeader("Expect", "100-continue");
If so, unfortunately that is not helping the request from the client now
looks like:
PUT /exist/rest/db/mediumfile.xml HTTP/1.1
Expect: 100-continue
Transfer-Encoding: chunked
Content-Type: application/xml
Host: localhost:8080
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.6 (Java/1.8.0_162)
Accept-Encoding: gzip,deflate
The server almost immediately responds with
HTTP/1.1 100 Continue
The client then starts sending data, during which, the server responds with:
HTTP/1.1 401 Unauthorized
Date: Thu, 11 Oct 2018 10:37:39 GMT
WWW-Authenticate: Basic realm="exist"
Content-Length: 0
Server: Jetty(9.4.10.v20180503)
The server then closes the connection. I then get the same Broken pipe
(Write failed) exceptions back from the Apache HTTP Client.
Does this mean that the server is also not behaving correctly? i.e. because
it returns 100 before it checks the authorization stuff? or is that also
allowed by the spec?
2. Migrate to a non-blocking HTTP client such as Apache HttpAsyncClient
> 4.1 or 5.0 that can handle out of sequence responses.
>
I will give this some thought...
--
Adam Retter
eXist Core Developer
{ United Kingdom / United States }
adam@exist-db.org
Re: Broken pipe (Write failed) when making Unauthorized request
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2018-10-10 at 22:41 +0500, Adam Retter wrote:
> I am using the Maven httpclient 4.5.6 artifact and trying to make a
> PUT
> request to a server. The server for some URLs requires
> authentication, it
> is impossible to know which URLs do and don't ahead of time. I only
> want to
> send the credentials if the server replies 401 Unauthorized.
>
> I am trying to do a HTTP PUT to the server. If the server replies
> HTTP 401
> Unauthorized, then I plan to retry with the authorization
> credentials.
>
> I have found rather a strange issue, and I am not sure if the fault
> is with
> the Apache HTTP Client or the server. However, I get different
> behaviour in
> the Apache HTTP Client depending on how fast I can stream the data to
> the
> server.
>
> What I am seeing looks like this:
>
> 1. The Client opens a HTTP connection to the server
> 2. The Client starts sending data to the server
> 3. The Server starts receiving the data from the client, and
> determines
> that the client is not authorized.
> 4. The Server responds HTTP 401 Unauthorized to the client
> 5. The server closes the connection.
>
> At the client end when I only have a small amount of data to PUT, or
> use a
> fast mechanism to write the data, all is fine and Apache HTTP Client
> gives
> me the HTTP Response 401 from the server. All is good!
>
> However, at the client end when I have a lot of data to PUT, or use a
> slow
> mechanism to write the data, the Apache HTTP Client is still writting
> data
> to the server when (5) occurs. The operating system then raises a
> "Broken
> pipe (Write failed)" error, which Java relays to the Apache HTTP
> Client as
> a SocketException. I then see errors like the following in my
> application:
>
This issue is known as out of sequence response. Such behavior is valid
according to the HTTP spec but is difficult to accommodate for when
using the classic (blocking) i/o model as blocking connections are not
able to write data and react to input events at the same time.
You have two options
1. Turn on 'expect-continue' handshake. It is intended precisely for
such situations.
2. Migrate to a non-blocking HTTP client such as Apache HttpAsyncClient
4.1 or 5.0 that can handle out of sequence responses.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org