You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Thomas Andrews (JIRA)" <ji...@apache.org> on 2012/04/27 18:52:48 UTC
[jira] [Commented] (AVRO-1069) HttpTransceiver never closes its
OutputStream
[ https://issues.apache.org/jira/browse/AVRO-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263787#comment-13263787 ]
Thomas Andrews commented on AVRO-1069:
--------------------------------------
When I rolled my own transceiver which was a copy of HttpTransceiver with close() calls to the InputStream and OutputStream added, I got much better connection re-use in Java. Specifically, in load test with four streams sending requests to the server, using Avro's HttpTransceiver, I used "netstat" to see that there were anywhere from 3-4 sockets open to handle the requests, but when I used my custom transceiver, it used only one socket for all four streams.
Since Java (by default) has a maximum of 5 cached open connections, this means at slightly higher load, HttpTransceiver is going to be opening lots of one-shot sockets.
> HttpTransceiver never closes its OutputStream
> ---------------------------------------------
>
> Key: AVRO-1069
> URL: https://issues.apache.org/jira/browse/AVRO-1069
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Thomas Andrews
>
> I'm not sure if this is the cause of my underlying problem, but the class org.apache.avro.ipc.HttpTransceiver opens an OutputStream and never explicitly closes it. That seems like very bad behavior, especially depending on whether the HttpURLConnection class holds onto the OutputStream or not - if it does, then potentially the Transceiver will never close the OutputStream.
> The specific oddity I am seeing is related to Flume. I have a Flume agent listening on port 60000 and a client using a FlumeEventAvroServer with the Avro HttpTransceiver class.
> Occasionally, when the Avro agent gets refreshed or otherwise reset, it fails with a BindException error.
> Even rarer, but still often enough to cause concern, the client software holds onto something which blocks the Flume agent from rebinding to the port even when the Flume agent is restarted. Essentially, something is now "occupying" the port. It is my guess that it is the transceiver which has failed to close the OutputStream, but I'm not 100% sure.
> I think you should also be closing the InputStream, as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira