You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by WireBaron <gi...@git.apache.org> on 2017/07/25 23:00:45 UTC
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
GitHub user WireBaron opened a pull request:
https://github.com/apache/geode/pull/657
GEODE-3286: Failing to cleanup connections from ConnectionTable recei…
…ver table
@kohlmu-pivotal @galen-pivotal @pivotal-amurmann @bschuchardt @hiteshk25
- prevent adding a closed connection to the connection table's receivers
- add a new unit test for connection table
- adding connection factory object for creating receiving connections
- have the idle connection timeout ensure connections are removed from connection
table receivers
- modify tcpConduit stat accesses to allow easier mocking
Signed-off-by: Hitesh Khamesra <hi...@yahoo.com>
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
- [x] Has your PR been rebased against the latest commit within the target branch (typically `develop`)?
- [x] Is your initial contribution a single, squashed commit?
- [x] Does `gradlew build` run cleanly?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/WireBaron/geode feature/GEODE-3286
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/geode/pull/657.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #657
----
commit 8aed26846de6e9ff1c123acae98a7b5ce6d82a83
Author: Brian Rowe <br...@pivotal.io>
Date: 2017-07-25T22:43:35Z
GEODE-3286: Failing to cleanup connections from ConnectionTable receiver table
- prevent adding a closed connection to the connection table's receivers
- add a new unit test for connection table
- adding connection factory object for creating receiving connections
- have the idle connection timeout ensure connections are removed from connection
table receivers
- modify tcpConduit stat accesses to allow easier mocking
Signed-off-by: Hitesh Khamesra <hi...@yahoo.com>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/geode/pull/657
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by WireBaron <gi...@git.apache.org>.
Github user WireBaron commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129961225
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/ConnectionTable.java ---
@@ -279,26 +280,29 @@ protected void acceptConnection(Socket sock) throws IOException, ConnectionExcep
// in our caller.
// no need to log error here since caller will log warning
- if (conn != null && !finishedConnecting) {
+ if (connection != null && !finishedConnecting) {
// we must be throwing from checkCancelInProgress so close the connection
- closeCon(LocalizedStrings.ConnectionTable_CANCEL_AFTER_ACCEPT.toLocalizedString(), conn);
- conn = null;
+ closeCon(LocalizedStrings.ConnectionTable_CANCEL_AFTER_ACCEPT.toLocalizedString(),
+ connection);
+ connection = null;
}
}
- if (conn != null) {
+ if (connection != null) {
synchronized (this.receivers) {
- this.owner.stats.incReceivers();
+ this.owner.getStats().incReceivers();
if (this.closed) {
closeCon(LocalizedStrings.ConnectionTable_CONNECTION_TABLE_NO_LONGER_IN_USE
- .toLocalizedString(), conn);
+ .toLocalizedString(), connection);
return;
}
- this.receivers.add(conn);
+ if (!connection.isSocketClosed()) {
--- End diff --
Looks like there may be an issue here since the socket close is done asynchronously from the connection close(). I've changed the check to also check connection.stopped, which is set to false inside the close call, before the connection cleans up it's state.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by WireBaron <gi...@git.apache.org>.
Github user WireBaron commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129683975
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/Connection.java ---
@@ -1322,6 +1328,14 @@ private void createBatchSendBuffer() {
this.batchFlusher.start();
}
+ public void onIdleCancel() {
--- End diff --
We'll change this to cleanUpOnIdleTaskCancel. The connection will always be closed when we reach this code. We debated adding an assert here, but decided against it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by WireBaron <gi...@git.apache.org>.
Github user WireBaron commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129961462
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/Connection.java ---
@@ -1322,6 +1328,14 @@ private void createBatchSendBuffer() {
this.batchFlusher.start();
}
+ public void onIdleCancel() {
--- End diff --
This was changed in the second commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode issue #657: GEODE-3286: Failing to cleanup connections from Connection...
Posted by WireBaron <gi...@git.apache.org>.
Github user WireBaron commented on the issue:
https://github.com/apache/geode/pull/657
Pushed a new version with some method renames.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by galen-pivotal <gi...@git.apache.org>.
Github user galen-pivotal commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129645767
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/Connection.java ---
@@ -1322,6 +1328,14 @@ private void createBatchSendBuffer() {
this.batchFlusher.start();
}
+ public void onIdleCancel() {
--- End diff --
Should this be closing and cleaning up the connection as well?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by WireBaron <gi...@git.apache.org>.
Github user WireBaron commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129684574
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/ConnectionTable.java ---
@@ -279,26 +280,29 @@ protected void acceptConnection(Socket sock) throws IOException, ConnectionExcep
// in our caller.
// no need to log error here since caller will log warning
- if (conn != null && !finishedConnecting) {
+ if (connection != null && !finishedConnecting) {
// we must be throwing from checkCancelInProgress so close the connection
- closeCon(LocalizedStrings.ConnectionTable_CANCEL_AFTER_ACCEPT.toLocalizedString(), conn);
- conn = null;
+ closeCon(LocalizedStrings.ConnectionTable_CANCEL_AFTER_ACCEPT.toLocalizedString(),
+ connection);
+ connection = null;
}
}
- if (conn != null) {
+ if (connection != null) {
synchronized (this.receivers) {
- this.owner.stats.incReceivers();
+ this.owner.getStats().incReceivers();
if (this.closed) {
closeCon(LocalizedStrings.ConnectionTable_CONNECTION_TABLE_NO_LONGER_IN_USE
- .toLocalizedString(), conn);
+ .toLocalizedString(), connection);
return;
}
- this.receivers.add(conn);
+ if (!connection.isSocketClosed()) {
--- End diff --
The connection removing code is actually spread all over and is done different ways depending on how and when the connection got closed. Tracking all of that down is actually where we've spent the bulk of the four days we were working on this. The only place we found that was obviously wrong was this section of code, which wasn't properly dealing with connections that closed immediately in another thread.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by WireBaron <gi...@git.apache.org>.
Github user WireBaron commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129683563
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/Connection.java ---
@@ -568,6 +568,12 @@ protected Connection(ConnectionTable t, Socket socket) throws IOException, Conne
}
}
+ protected void initRecevier() {
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode issue #657: GEODE-3286: Failing to cleanup connections from Connection...
Posted by WireBaron <gi...@git.apache.org>.
Github user WireBaron commented on the issue:
https://github.com/apache/geode/pull/657
Pushed a new revision tightening the check before adding a connection to the connectionTable's receivers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by galen-pivotal <gi...@git.apache.org>.
Github user galen-pivotal commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129892735
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/ConnectionTable.java ---
@@ -279,26 +280,29 @@ protected void acceptConnection(Socket sock) throws IOException, ConnectionExcep
// in our caller.
// no need to log error here since caller will log warning
- if (conn != null && !finishedConnecting) {
+ if (connection != null && !finishedConnecting) {
// we must be throwing from checkCancelInProgress so close the connection
- closeCon(LocalizedStrings.ConnectionTable_CANCEL_AFTER_ACCEPT.toLocalizedString(), conn);
- conn = null;
+ closeCon(LocalizedStrings.ConnectionTable_CANCEL_AFTER_ACCEPT.toLocalizedString(),
+ connection);
+ connection = null;
}
}
- if (conn != null) {
+ if (connection != null) {
synchronized (this.receivers) {
- this.owner.stats.incReceivers();
+ this.owner.getStats().incReceivers();
if (this.closed) {
closeCon(LocalizedStrings.ConnectionTable_CONNECTION_TABLE_NO_LONGER_IN_USE
- .toLocalizedString(), conn);
+ .toLocalizedString(), connection);
return;
}
- this.receivers.add(conn);
+ if (!connection.isSocketClosed()) {
--- End diff --
Why doesn't the connection get closed by another mechanism? If the connection closes after the call to `connection.isSocketClosed()` does it show the same symptoms? (thereby changing a leak into a smaller leak)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by galen-pivotal <gi...@git.apache.org>.
Github user galen-pivotal commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129640243
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/Connection.java ---
@@ -568,6 +568,12 @@ protected Connection(ConnectionTable t, Socket socket) throws IOException, Conne
}
}
+ protected void initRecevier() {
--- End diff --
typo: recevier -> receiver.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by galen-pivotal <gi...@git.apache.org>.
Github user galen-pivotal commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129640056
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/ConnectionTable.java ---
@@ -279,26 +280,29 @@ protected void acceptConnection(Socket sock) throws IOException, ConnectionExcep
// in our caller.
// no need to log error here since caller will log warning
- if (conn != null && !finishedConnecting) {
+ if (connection != null && !finishedConnecting) {
// we must be throwing from checkCancelInProgress so close the connection
- closeCon(LocalizedStrings.ConnectionTable_CANCEL_AFTER_ACCEPT.toLocalizedString(), conn);
- conn = null;
+ closeCon(LocalizedStrings.ConnectionTable_CANCEL_AFTER_ACCEPT.toLocalizedString(),
+ connection);
+ connection = null;
}
}
- if (conn != null) {
+ if (connection != null) {
synchronized (this.receivers) {
- this.owner.stats.incReceivers();
+ this.owner.getStats().incReceivers();
if (this.closed) {
closeCon(LocalizedStrings.ConnectionTable_CONNECTION_TABLE_NO_LONGER_IN_USE
- .toLocalizedString(), conn);
+ .toLocalizedString(), connection);
return;
}
- this.receivers.add(conn);
+ if (!connection.isSocketClosed()) {
--- End diff --
It looks like this is the critical line. What is it that makes the connections get removed from the table when they are unexpectedly closed? What happens if the connection is closed after being added to the table?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by pivotal-amurmann <gi...@git.apache.org>.
Github user pivotal-amurmann commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129689404
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/ConnectionTable.java ---
@@ -279,26 +280,29 @@ protected void acceptConnection(Socket sock) throws IOException, ConnectionExcep
// in our caller.
// no need to log error here since caller will log warning
- if (conn != null && !finishedConnecting) {
+ if (connection != null && !finishedConnecting) {
// we must be throwing from checkCancelInProgress so close the connection
- closeCon(LocalizedStrings.ConnectionTable_CANCEL_AFTER_ACCEPT.toLocalizedString(), conn);
- conn = null;
+ closeCon(LocalizedStrings.ConnectionTable_CANCEL_AFTER_ACCEPT.toLocalizedString(),
+ connection);
+ connection = null;
}
}
- if (conn != null) {
+ if (connection != null) {
synchronized (this.receivers) {
- this.owner.stats.incReceivers();
+ this.owner.getStats().incReceivers();
if (this.closed) {
closeCon(LocalizedStrings.ConnectionTable_CONNECTION_TABLE_NO_LONGER_IN_USE
- .toLocalizedString(), conn);
+ .toLocalizedString(), connection);
return;
}
- this.receivers.add(conn);
+ if (!connection.isSocketClosed()) {
--- End diff --
Maybe once this gets squashed into a single commit the explanation can be added to the commit message to be preserved with the code? It's always awesome to do a `git annotate` and get a great explanation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by kohlmu-pivotal <gi...@git.apache.org>.
Github user kohlmu-pivotal commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129879726
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/Connection.java ---
@@ -1322,6 +1328,14 @@ private void createBatchSendBuffer() {
this.batchFlusher.start();
}
+ public void onIdleCancel() {
--- End diff --
Has this work been done yet?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] geode pull request #657: GEODE-3286: Failing to cleanup connections from Con...
Posted by galen-pivotal <gi...@git.apache.org>.
Github user galen-pivotal commented on a diff in the pull request:
https://github.com/apache/geode/pull/657#discussion_r129639921
--- Diff: geode-core/src/main/java/org/apache/geode/internal/tcp/Connection.java ---
@@ -1322,6 +1328,14 @@ private void createBatchSendBuffer() {
this.batchFlusher.start();
}
+ public void onIdleCancel() {
--- End diff --
The name of this function is not intuitive to me. Could this be named `removeIdleReceiver` or something? Or just `removeReceiver`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---