You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Tom May (JIRA)" <ji...@apache.org> on 2011/06/01 02:38:48 UTC
[jira] [Created] (THRIFT-1190) readBufferBytesAllocated
TNonblockingServer.java should be AtomicLong to fix FD leakage and general
server malfunction
readBufferBytesAllocated TNonblockingServer.java should be AtomicLong to fix FD leakage and general server malfunction
----------------------------------------------------------------------------------------------------------------------
Key: THRIFT-1190
URL: https://issues.apache.org/jira/browse/THRIFT-1190
Project: Thrift
Issue Type: Bug
Components: Java - Library
Affects Versions: 0.6
Environment: Any
Reporter: Tom May
We're having a problem using THsHaServer where the server doesn't suddenly stops closing close connections and runs out of file descriptors.
It looks like the problem is in TNonblockingServer.java. The variable "long readBufferBytesAllocated" is shared between threads but isn't synchronized. Intermittently, its value can get out of whack due to races and cause this if statement to always execute:
// if this frame will push us over the memory limit, then return.
// with luck, more memory will free up the next time around.
if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
return true;
}
We started having this problem when we set MAX_READ_BUFFER_BYTES to a somewhat small size, which unmasked the issue.
I don't see anywhere here to attach a patch so here it is:
Index: lib/java/src/org/apache/thrift/server/TNonblockingServer.java
===================================================================
--- lib/java/src/org/apache/thrift/server/TNonblockingServer.java (revision 1129978)
+++ lib/java/src/org/apache/thrift/server/TNonblockingServer.java (working copy)
@@ -28,6 +28,7 @@
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
import org.apache.thrift.TByteArrayOutputStream;
import org.apache.thrift.TException;
@@ -87,7 +88,7 @@
/**
* How many bytes are currently allocated to read buffers.
*/
- private long readBufferBytesAllocated = 0;
+ private final AtomicLong readBufferBytesAllocated = new AtomicLong(0);
public TNonblockingServer(AbstractNonblockingServerArgs args) {
super(args);
@@ -479,12 +480,12 @@
// if this frame will push us over the memory limit, then return.
// with luck, more memory will free up the next time around.
- if (readBufferBytesAllocated + frameSize > MAX_READ_BUFFER_BYTES) {
+ if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
return true;
}
// increment the amount of memory allocated to read buffers
- readBufferBytesAllocated += frameSize;
+ readBufferBytesAllocated.addAndGet(frameSize);
// reallocate the readbuffer as a frame-sized buffer
buffer_ = ByteBuffer.allocate(frameSize);
@@ -576,7 +577,7 @@
// if we're being closed due to an error, we might have allocated a
// buffer that we need to subtract for our memory accounting.
if (state_ == READING_FRAME || state_ == READ_FRAME_COMPLETE) {
- readBufferBytesAllocated -= buffer_.array().length;
+ readBufferBytesAllocated.addAndGet(-buffer_.array().length);
}
trans_.close();
}
@@ -600,7 +601,7 @@
// our read buffer count. we do this here as well as in close because
// we'd like to free this read memory up as quickly as possible for other
// clients.
- readBufferBytesAllocated -= buffer_.array().length;
+ readBufferBytesAllocated.addAndGet(-buffer_.array().length);
if (response_.len() == 0) {
// go straight to reading again. this was probably an oneway method
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1190) readBufferBytesAllocated in
TNonblockingServer.java should be AtomicLong to fix FD leakage and general
server malfunction
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042318#comment-13042318 ]
Hudson commented on THRIFT-1190:
--------------------------------
Integrated in Thrift #151 (See [https://builds.apache.org/hudson/job/Thrift/151/])
THRIFT-1190. java: readBufferBytesAllocated in TNonblockingServer.java should be AtomicLong to fix FD leakage and general server malfunction
There was a race condition in the use of the memory limiting feature that would lead to memory loss.
Patch: Tom May
bryanduxbury : http://svn.apache.org/viewvc/?view=rev&rev=1130231
Files :
* /thrift/trunk/lib/java/src/org/apache/thrift/server/TNonblockingServer.java
> readBufferBytesAllocated in TNonblockingServer.java should be AtomicLong to fix FD leakage and general server malfunction
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: THRIFT-1190
> URL: https://issues.apache.org/jira/browse/THRIFT-1190
> Project: Thrift
> Issue Type: Bug
> Components: Java - Library
> Affects Versions: 0.6
> Environment: Any
> Reporter: Tom May
> Assignee: Tom May
> Fix For: 0.7
>
> Attachments: thrift.patch
>
>
> We're having a problem using THsHaServer where the server suddenly stops closing connections and runs out of file descriptors.
> It looks like the problem is in TNonblockingServer.java. The variable "long readBufferBytesAllocated" is shared between threads but isn't synchronized. Intermittently, its value can get out of whack due to races and cause this if statement to always execute:
> // if this frame will push us over the memory limit, then return.
> // with luck, more memory will free up the next time around.
> if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
> return true;
> }
> We started having this problem when we set MAX_READ_BUFFER_BYTES to a somewhat small size, which unmasked the issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (THRIFT-1190) readBufferBytesAllocated in
TNonblockingServer.java should be AtomicLong to fix FD leakage and general
server malfunction
Posted by "Tom May (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom May updated THRIFT-1190:
----------------------------
Description:
We're having a problem using THsHaServer where the server suddenly stops closing connections and runs out of file descriptors.
It looks like the problem is in TNonblockingServer.java. The variable "long readBufferBytesAllocated" is shared between threads but isn't synchronized. Intermittently, its value can get out of whack due to races and cause this if statement to always execute:
// if this frame will push us over the memory limit, then return.
// with luck, more memory will free up the next time around.
if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
return true;
}
We started having this problem when we set MAX_READ_BUFFER_BYTES to a somewhat small size, which unmasked the issue.
I don't see anywhere here to attach a patch so here it is:
Index: lib/java/src/org/apache/thrift/server/TNonblockingServer.java
===================================================================
--- lib/java/src/org/apache/thrift/server/TNonblockingServer.java (revision 1129978)
+++ lib/java/src/org/apache/thrift/server/TNonblockingServer.java (working copy)
@@ -28,6 +28,7 @@
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
import org.apache.thrift.TByteArrayOutputStream;
import org.apache.thrift.TException;
@@ -87,7 +88,7 @@
/**
* How many bytes are currently allocated to read buffers.
*/
- private long readBufferBytesAllocated = 0;
+ private final AtomicLong readBufferBytesAllocated = new AtomicLong(0);
public TNonblockingServer(AbstractNonblockingServerArgs args) {
super(args);
@@ -479,12 +480,12 @@
// if this frame will push us over the memory limit, then return.
// with luck, more memory will free up the next time around.
- if (readBufferBytesAllocated + frameSize > MAX_READ_BUFFER_BYTES) {
+ if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
return true;
}
// increment the amount of memory allocated to read buffers
- readBufferBytesAllocated += frameSize;
+ readBufferBytesAllocated.addAndGet(frameSize);
// reallocate the readbuffer as a frame-sized buffer
buffer_ = ByteBuffer.allocate(frameSize);
@@ -576,7 +577,7 @@
// if we're being closed due to an error, we might have allocated a
// buffer that we need to subtract for our memory accounting.
if (state_ == READING_FRAME || state_ == READ_FRAME_COMPLETE) {
- readBufferBytesAllocated -= buffer_.array().length;
+ readBufferBytesAllocated.addAndGet(-buffer_.array().length);
}
trans_.close();
}
@@ -600,7 +601,7 @@
// our read buffer count. we do this here as well as in close because
// we'd like to free this read memory up as quickly as possible for other
// clients.
- readBufferBytesAllocated -= buffer_.array().length;
+ readBufferBytesAllocated.addAndGet(-buffer_.array().length);
if (response_.len() == 0) {
// go straight to reading again. this was probably an oneway method
was:
We're having a problem using THsHaServer where the server doesn't suddenly stops closing close connections and runs out of file descriptors.
It looks like the problem is in TNonblockingServer.java. The variable "long readBufferBytesAllocated" is shared between threads but isn't synchronized. Intermittently, its value can get out of whack due to races and cause this if statement to always execute:
// if this frame will push us over the memory limit, then return.
// with luck, more memory will free up the next time around.
if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
return true;
}
We started having this problem when we set MAX_READ_BUFFER_BYTES to a somewhat small size, which unmasked the issue.
I don't see anywhere here to attach a patch so here it is:
Index: lib/java/src/org/apache/thrift/server/TNonblockingServer.java
===================================================================
--- lib/java/src/org/apache/thrift/server/TNonblockingServer.java (revision 1129978)
+++ lib/java/src/org/apache/thrift/server/TNonblockingServer.java (working copy)
@@ -28,6 +28,7 @@
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
import org.apache.thrift.TByteArrayOutputStream;
import org.apache.thrift.TException;
@@ -87,7 +88,7 @@
/**
* How many bytes are currently allocated to read buffers.
*/
- private long readBufferBytesAllocated = 0;
+ private final AtomicLong readBufferBytesAllocated = new AtomicLong(0);
public TNonblockingServer(AbstractNonblockingServerArgs args) {
super(args);
@@ -479,12 +480,12 @@
// if this frame will push us over the memory limit, then return.
// with luck, more memory will free up the next time around.
- if (readBufferBytesAllocated + frameSize > MAX_READ_BUFFER_BYTES) {
+ if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
return true;
}
// increment the amount of memory allocated to read buffers
- readBufferBytesAllocated += frameSize;
+ readBufferBytesAllocated.addAndGet(frameSize);
// reallocate the readbuffer as a frame-sized buffer
buffer_ = ByteBuffer.allocate(frameSize);
@@ -576,7 +577,7 @@
// if we're being closed due to an error, we might have allocated a
// buffer that we need to subtract for our memory accounting.
if (state_ == READING_FRAME || state_ == READ_FRAME_COMPLETE) {
- readBufferBytesAllocated -= buffer_.array().length;
+ readBufferBytesAllocated.addAndGet(-buffer_.array().length);
}
trans_.close();
}
@@ -600,7 +601,7 @@
// our read buffer count. we do this here as well as in close because
// we'd like to free this read memory up as quickly as possible for other
// clients.
- readBufferBytesAllocated -= buffer_.array().length;
+ readBufferBytesAllocated.addAndGet(-buffer_.array().length);
if (response_.len() == 0) {
// go straight to reading again. this was probably an oneway method
> readBufferBytesAllocated in TNonblockingServer.java should be AtomicLong to fix FD leakage and general server malfunction
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: THRIFT-1190
> URL: https://issues.apache.org/jira/browse/THRIFT-1190
> Project: Thrift
> Issue Type: Bug
> Components: Java - Library
> Affects Versions: 0.6
> Environment: Any
> Reporter: Tom May
> Attachments: thrift.patch
>
>
> We're having a problem using THsHaServer where the server suddenly stops closing connections and runs out of file descriptors.
> It looks like the problem is in TNonblockingServer.java. The variable "long readBufferBytesAllocated" is shared between threads but isn't synchronized. Intermittently, its value can get out of whack due to races and cause this if statement to always execute:
> // if this frame will push us over the memory limit, then return.
> // with luck, more memory will free up the next time around.
> if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
> return true;
> }
> We started having this problem when we set MAX_READ_BUFFER_BYTES to a somewhat small size, which unmasked the issue.
> I don't see anywhere here to attach a patch so here it is:
> Index: lib/java/src/org/apache/thrift/server/TNonblockingServer.java
> ===================================================================
> --- lib/java/src/org/apache/thrift/server/TNonblockingServer.java (revision 1129978)
> +++ lib/java/src/org/apache/thrift/server/TNonblockingServer.java (working copy)
> @@ -28,6 +28,7 @@
> import java.util.HashSet;
> import java.util.Iterator;
> import java.util.Set;
> +import java.util.concurrent.atomic.AtomicLong;
>
> import org.apache.thrift.TByteArrayOutputStream;
> import org.apache.thrift.TException;
> @@ -87,7 +88,7 @@
> /**
> * How many bytes are currently allocated to read buffers.
> */
> - private long readBufferBytesAllocated = 0;
> + private final AtomicLong readBufferBytesAllocated = new AtomicLong(0);
>
> public TNonblockingServer(AbstractNonblockingServerArgs args) {
> super(args);
> @@ -479,12 +480,12 @@
>
> // if this frame will push us over the memory limit, then return.
> // with luck, more memory will free up the next time around.
> - if (readBufferBytesAllocated + frameSize > MAX_READ_BUFFER_BYTES) {
> + if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
> return true;
> }
>
> // increment the amount of memory allocated to read buffers
> - readBufferBytesAllocated += frameSize;
> + readBufferBytesAllocated.addAndGet(frameSize);
>
> // reallocate the readbuffer as a frame-sized buffer
> buffer_ = ByteBuffer.allocate(frameSize);
> @@ -576,7 +577,7 @@
> // if we're being closed due to an error, we might have allocated a
> // buffer that we need to subtract for our memory accounting.
> if (state_ == READING_FRAME || state_ == READ_FRAME_COMPLETE) {
> - readBufferBytesAllocated -= buffer_.array().length;
> + readBufferBytesAllocated.addAndGet(-buffer_.array().length);
> }
> trans_.close();
> }
> @@ -600,7 +601,7 @@
> // our read buffer count. we do this here as well as in close because
> // we'd like to free this read memory up as quickly as possible for other
> // clients.
> - readBufferBytesAllocated -= buffer_.array().length;
> + readBufferBytesAllocated.addAndGet(-buffer_.array().length);
>
> if (response_.len() == 0) {
> // go straight to reading again. this was probably an oneway method
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (THRIFT-1190) readBufferBytesAllocated in
TNonblockingServer.java should be AtomicLong to fix FD leakage and general
server malfunction
Posted by "Tom May (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom May updated THRIFT-1190:
----------------------------
Description:
We're having a problem using THsHaServer where the server suddenly stops closing connections and runs out of file descriptors.
It looks like the problem is in TNonblockingServer.java. The variable "long readBufferBytesAllocated" is shared between threads but isn't synchronized. Intermittently, its value can get out of whack due to races and cause this if statement to always execute:
// if this frame will push us over the memory limit, then return.
// with luck, more memory will free up the next time around.
if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
return true;
}
We started having this problem when we set MAX_READ_BUFFER_BYTES to a somewhat small size, which unmasked the issue.
was:
We're having a problem using THsHaServer where the server suddenly stops closing connections and runs out of file descriptors.
It looks like the problem is in TNonblockingServer.java. The variable "long readBufferBytesAllocated" is shared between threads but isn't synchronized. Intermittently, its value can get out of whack due to races and cause this if statement to always execute:
// if this frame will push us over the memory limit, then return.
// with luck, more memory will free up the next time around.
if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
return true;
}
We started having this problem when we set MAX_READ_BUFFER_BYTES to a somewhat small size, which unmasked the issue.
I don't see anywhere here to attach a patch so here it is:
Index: lib/java/src/org/apache/thrift/server/TNonblockingServer.java
===================================================================
--- lib/java/src/org/apache/thrift/server/TNonblockingServer.java (revision 1129978)
+++ lib/java/src/org/apache/thrift/server/TNonblockingServer.java (working copy)
@@ -28,6 +28,7 @@
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
import org.apache.thrift.TByteArrayOutputStream;
import org.apache.thrift.TException;
@@ -87,7 +88,7 @@
/**
* How many bytes are currently allocated to read buffers.
*/
- private long readBufferBytesAllocated = 0;
+ private final AtomicLong readBufferBytesAllocated = new AtomicLong(0);
public TNonblockingServer(AbstractNonblockingServerArgs args) {
super(args);
@@ -479,12 +480,12 @@
// if this frame will push us over the memory limit, then return.
// with luck, more memory will free up the next time around.
- if (readBufferBytesAllocated + frameSize > MAX_READ_BUFFER_BYTES) {
+ if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
return true;
}
// increment the amount of memory allocated to read buffers
- readBufferBytesAllocated += frameSize;
+ readBufferBytesAllocated.addAndGet(frameSize);
// reallocate the readbuffer as a frame-sized buffer
buffer_ = ByteBuffer.allocate(frameSize);
@@ -576,7 +577,7 @@
// if we're being closed due to an error, we might have allocated a
// buffer that we need to subtract for our memory accounting.
if (state_ == READING_FRAME || state_ == READ_FRAME_COMPLETE) {
- readBufferBytesAllocated -= buffer_.array().length;
+ readBufferBytesAllocated.addAndGet(-buffer_.array().length);
}
trans_.close();
}
@@ -600,7 +601,7 @@
// our read buffer count. we do this here as well as in close because
// we'd like to free this read memory up as quickly as possible for other
// clients.
- readBufferBytesAllocated -= buffer_.array().length;
+ readBufferBytesAllocated.addAndGet(-buffer_.array().length);
if (response_.len() == 0) {
// go straight to reading again. this was probably an oneway method
> readBufferBytesAllocated in TNonblockingServer.java should be AtomicLong to fix FD leakage and general server malfunction
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: THRIFT-1190
> URL: https://issues.apache.org/jira/browse/THRIFT-1190
> Project: Thrift
> Issue Type: Bug
> Components: Java - Library
> Affects Versions: 0.6
> Environment: Any
> Reporter: Tom May
> Attachments: thrift.patch
>
>
> We're having a problem using THsHaServer where the server suddenly stops closing connections and runs out of file descriptors.
> It looks like the problem is in TNonblockingServer.java. The variable "long readBufferBytesAllocated" is shared between threads but isn't synchronized. Intermittently, its value can get out of whack due to races and cause this if statement to always execute:
> // if this frame will push us over the memory limit, then return.
> // with luck, more memory will free up the next time around.
> if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
> return true;
> }
> We started having this problem when we set MAX_READ_BUFFER_BYTES to a somewhat small size, which unmasked the issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (THRIFT-1190) readBufferBytesAllocated
TNonblockingServer.java should be AtomicLong to fix FD leakage and general
server malfunction
Posted by "Tom May (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom May updated THRIFT-1190:
----------------------------
Attachment: thrift.patch
Here's the patch.
> readBufferBytesAllocated TNonblockingServer.java should be AtomicLong to fix FD leakage and general server malfunction
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: THRIFT-1190
> URL: https://issues.apache.org/jira/browse/THRIFT-1190
> Project: Thrift
> Issue Type: Bug
> Components: Java - Library
> Affects Versions: 0.6
> Environment: Any
> Reporter: Tom May
> Attachments: thrift.patch
>
>
> We're having a problem using THsHaServer where the server doesn't suddenly stops closing close connections and runs out of file descriptors.
> It looks like the problem is in TNonblockingServer.java. The variable "long readBufferBytesAllocated" is shared between threads but isn't synchronized. Intermittently, its value can get out of whack due to races and cause this if statement to always execute:
> // if this frame will push us over the memory limit, then return.
> // with luck, more memory will free up the next time around.
> if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
> return true;
> }
> We started having this problem when we set MAX_READ_BUFFER_BYTES to a somewhat small size, which unmasked the issue.
> I don't see anywhere here to attach a patch so here it is:
> Index: lib/java/src/org/apache/thrift/server/TNonblockingServer.java
> ===================================================================
> --- lib/java/src/org/apache/thrift/server/TNonblockingServer.java (revision 1129978)
> +++ lib/java/src/org/apache/thrift/server/TNonblockingServer.java (working copy)
> @@ -28,6 +28,7 @@
> import java.util.HashSet;
> import java.util.Iterator;
> import java.util.Set;
> +import java.util.concurrent.atomic.AtomicLong;
>
> import org.apache.thrift.TByteArrayOutputStream;
> import org.apache.thrift.TException;
> @@ -87,7 +88,7 @@
> /**
> * How many bytes are currently allocated to read buffers.
> */
> - private long readBufferBytesAllocated = 0;
> + private final AtomicLong readBufferBytesAllocated = new AtomicLong(0);
>
> public TNonblockingServer(AbstractNonblockingServerArgs args) {
> super(args);
> @@ -479,12 +480,12 @@
>
> // if this frame will push us over the memory limit, then return.
> // with luck, more memory will free up the next time around.
> - if (readBufferBytesAllocated + frameSize > MAX_READ_BUFFER_BYTES) {
> + if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
> return true;
> }
>
> // increment the amount of memory allocated to read buffers
> - readBufferBytesAllocated += frameSize;
> + readBufferBytesAllocated.addAndGet(frameSize);
>
> // reallocate the readbuffer as a frame-sized buffer
> buffer_ = ByteBuffer.allocate(frameSize);
> @@ -576,7 +577,7 @@
> // if we're being closed due to an error, we might have allocated a
> // buffer that we need to subtract for our memory accounting.
> if (state_ == READING_FRAME || state_ == READ_FRAME_COMPLETE) {
> - readBufferBytesAllocated -= buffer_.array().length;
> + readBufferBytesAllocated.addAndGet(-buffer_.array().length);
> }
> trans_.close();
> }
> @@ -600,7 +601,7 @@
> // our read buffer count. we do this here as well as in close because
> // we'd like to free this read memory up as quickly as possible for other
> // clients.
> - readBufferBytesAllocated -= buffer_.array().length;
> + readBufferBytesAllocated.addAndGet(-buffer_.array().length);
>
> if (response_.len() == 0) {
> // go straight to reading again. this was probably an oneway method
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (THRIFT-1190) readBufferBytesAllocated in
TNonblockingServer.java should be AtomicLong to fix FD leakage and general
server malfunction
Posted by "Tom May (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom May updated THRIFT-1190:
----------------------------
Summary: readBufferBytesAllocated in TNonblockingServer.java should be AtomicLong to fix FD leakage and general server malfunction (was: readBufferBytesAllocated TNonblockingServer.java should be AtomicLong to fix FD leakage and general server malfunction)
> readBufferBytesAllocated in TNonblockingServer.java should be AtomicLong to fix FD leakage and general server malfunction
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: THRIFT-1190
> URL: https://issues.apache.org/jira/browse/THRIFT-1190
> Project: Thrift
> Issue Type: Bug
> Components: Java - Library
> Affects Versions: 0.6
> Environment: Any
> Reporter: Tom May
> Attachments: thrift.patch
>
>
> We're having a problem using THsHaServer where the server doesn't suddenly stops closing close connections and runs out of file descriptors.
> It looks like the problem is in TNonblockingServer.java. The variable "long readBufferBytesAllocated" is shared between threads but isn't synchronized. Intermittently, its value can get out of whack due to races and cause this if statement to always execute:
> // if this frame will push us over the memory limit, then return.
> // with luck, more memory will free up the next time around.
> if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
> return true;
> }
> We started having this problem when we set MAX_READ_BUFFER_BYTES to a somewhat small size, which unmasked the issue.
> I don't see anywhere here to attach a patch so here it is:
> Index: lib/java/src/org/apache/thrift/server/TNonblockingServer.java
> ===================================================================
> --- lib/java/src/org/apache/thrift/server/TNonblockingServer.java (revision 1129978)
> +++ lib/java/src/org/apache/thrift/server/TNonblockingServer.java (working copy)
> @@ -28,6 +28,7 @@
> import java.util.HashSet;
> import java.util.Iterator;
> import java.util.Set;
> +import java.util.concurrent.atomic.AtomicLong;
>
> import org.apache.thrift.TByteArrayOutputStream;
> import org.apache.thrift.TException;
> @@ -87,7 +88,7 @@
> /**
> * How many bytes are currently allocated to read buffers.
> */
> - private long readBufferBytesAllocated = 0;
> + private final AtomicLong readBufferBytesAllocated = new AtomicLong(0);
>
> public TNonblockingServer(AbstractNonblockingServerArgs args) {
> super(args);
> @@ -479,12 +480,12 @@
>
> // if this frame will push us over the memory limit, then return.
> // with luck, more memory will free up the next time around.
> - if (readBufferBytesAllocated + frameSize > MAX_READ_BUFFER_BYTES) {
> + if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
> return true;
> }
>
> // increment the amount of memory allocated to read buffers
> - readBufferBytesAllocated += frameSize;
> + readBufferBytesAllocated.addAndGet(frameSize);
>
> // reallocate the readbuffer as a frame-sized buffer
> buffer_ = ByteBuffer.allocate(frameSize);
> @@ -576,7 +577,7 @@
> // if we're being closed due to an error, we might have allocated a
> // buffer that we need to subtract for our memory accounting.
> if (state_ == READING_FRAME || state_ == READ_FRAME_COMPLETE) {
> - readBufferBytesAllocated -= buffer_.array().length;
> + readBufferBytesAllocated.addAndGet(-buffer_.array().length);
> }
> trans_.close();
> }
> @@ -600,7 +601,7 @@
> // our read buffer count. we do this here as well as in close because
> // we'd like to free this read memory up as quickly as possible for other
> // clients.
> - readBufferBytesAllocated -= buffer_.array().length;
> + readBufferBytesAllocated.addAndGet(-buffer_.array().length);
>
> if (response_.len() == 0) {
> // go straight to reading again. this was probably an oneway method
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (THRIFT-1190) readBufferBytesAllocated in
TNonblockingServer.java should be AtomicLong to fix FD leakage and general
server malfunction
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury closed THRIFT-1190.
---------------------------------
Resolution: Fixed
Fix Version/s: 0.7
Assignee: Tom May
Nice catch! I just committed this patch.
> readBufferBytesAllocated in TNonblockingServer.java should be AtomicLong to fix FD leakage and general server malfunction
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: THRIFT-1190
> URL: https://issues.apache.org/jira/browse/THRIFT-1190
> Project: Thrift
> Issue Type: Bug
> Components: Java - Library
> Affects Versions: 0.6
> Environment: Any
> Reporter: Tom May
> Assignee: Tom May
> Fix For: 0.7
>
> Attachments: thrift.patch
>
>
> We're having a problem using THsHaServer where the server suddenly stops closing connections and runs out of file descriptors.
> It looks like the problem is in TNonblockingServer.java. The variable "long readBufferBytesAllocated" is shared between threads but isn't synchronized. Intermittently, its value can get out of whack due to races and cause this if statement to always execute:
> // if this frame will push us over the memory limit, then return.
> // with luck, more memory will free up the next time around.
> if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
> return true;
> }
> We started having this problem when we set MAX_READ_BUFFER_BYTES to a somewhat small size, which unmasked the issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira