You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jim Ancona (JIRA)" <ji...@apache.org> on 2011/09/06 04:11:09 UTC
[jira] [Created] (CASSANDRA-3142) CustomTThreadPoolServer should
log TTransportException at DEBUG level
CustomTThreadPoolServer should log TTransportException at DEBUG level
---------------------------------------------------------------------
Key: CASSANDRA-3142
URL: https://issues.apache.org/jira/browse/CASSANDRA-3142
Project: Cassandra
Issue Type: Bug
Reporter: Jim Ancona
Currently CustomTThreadPoolServer, like the Thrift TThreadPoolServer, silently ignores TTransportException in its run() method. This is appropriate in most cases because TTransportException occurs fairly often when client connections die. However TTransportException is also thrown when TFramedTransport encounters a frame that is larger than thrift_framed_transport_size_in_mb. In that case, silently exiting the run loop leads to a SocketException on the client side which can be both difficult to diagnose, in part because nothing is logged by Cassandra, and high-impact, because the client may respond by marking the server node down and retrying the too-large request on another node, where it also fails. This process repeated leads to the entire cluster being marked down (see https://github.com/rantav/hector/issues/212). I've filed two Thrift issues (https://issues.apache.org/jira/browse/THRIFT-1323 and https://issues.apache.org/jira/browse/THRIFT-1324), but in the meantime, I suggest that CustomTThreadPoolServer log the exception at DEBUG level in order to support easier troubleshooting.
I can submit a patch with the added log message.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3142) CustomTThreadPoolServer should
log TTransportException at DEBUG level
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich resolved CASSANDRA-3142.
----------------------------------------
Resolution: Fixed
Committed.
> CustomTThreadPoolServer should log TTransportException at DEBUG level
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-3142
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3142
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jim Ancona
> Assignee: Jim Ancona
> Priority: Minor
> Fix For: 0.8.6
>
> Attachments: v1-0001-CASSANDRA-3142-Add-debug-level-logging-of-TTransportEx.txt
>
>
> Currently CustomTThreadPoolServer, like the Thrift TThreadPoolServer, silently ignores TTransportException in its run() method. This is appropriate in most cases because TTransportException occurs fairly often when client connections die. However TTransportException is also thrown when TFramedTransport encounters a frame that is larger than thrift_framed_transport_size_in_mb. In that case, silently exiting the run loop leads to a SocketException on the client side which can be both difficult to diagnose, in part because nothing is logged by Cassandra, and high-impact, because the client may respond by marking the server node down and retrying the too-large request on another node, where it also fails. This process repeated leads to the entire cluster being marked down (see https://github.com/rantav/hector/issues/212). I've filed two Thrift issues (https://issues.apache.org/jira/browse/THRIFT-1323 and https://issues.apache.org/jira/browse/THRIFT-1324), but in the meantime, I suggest that CustomTThreadPoolServer log the exception at DEBUG level in order to support easier troubleshooting.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3142) CustomTThreadPoolServer should
log TTransportException at DEBUG level
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3142:
--------------------------------------
Reviewer: xedin
Priority: Minor (was: Major)
Fix Version/s: 0.8.6
Assignee: Jim Ancona
> CustomTThreadPoolServer should log TTransportException at DEBUG level
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-3142
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3142
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jim Ancona
> Assignee: Jim Ancona
> Priority: Minor
> Fix For: 0.8.6
>
> Attachments: v1-0001-CASSANDRA-3142-Add-debug-level-logging-of-TTransportEx.txt
>
>
> Currently CustomTThreadPoolServer, like the Thrift TThreadPoolServer, silently ignores TTransportException in its run() method. This is appropriate in most cases because TTransportException occurs fairly often when client connections die. However TTransportException is also thrown when TFramedTransport encounters a frame that is larger than thrift_framed_transport_size_in_mb. In that case, silently exiting the run loop leads to a SocketException on the client side which can be both difficult to diagnose, in part because nothing is logged by Cassandra, and high-impact, because the client may respond by marking the server node down and retrying the too-large request on another node, where it also fails. This process repeated leads to the entire cluster being marked down (see https://github.com/rantav/hector/issues/212). I've filed two Thrift issues (https://issues.apache.org/jira/browse/THRIFT-1323 and https://issues.apache.org/jira/browse/THRIFT-1324), but in the meantime, I suggest that CustomTThreadPoolServer log the exception at DEBUG level in order to support easier troubleshooting.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3142) CustomTThreadPoolServer should
log TTransportException at DEBUG level
Posted by "Jim Ancona (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Ancona updated CASSANDRA-3142:
----------------------------------
Description:
Currently CustomTThreadPoolServer, like the Thrift TThreadPoolServer, silently ignores TTransportException in its run() method. This is appropriate in most cases because TTransportException occurs fairly often when client connections die. However TTransportException is also thrown when TFramedTransport encounters a frame that is larger than thrift_framed_transport_size_in_mb. In that case, silently exiting the run loop leads to a SocketException on the client side which can be both difficult to diagnose, in part because nothing is logged by Cassandra, and high-impact, because the client may respond by marking the server node down and retrying the too-large request on another node, where it also fails. This process repeated leads to the entire cluster being marked down (see https://github.com/rantav/hector/issues/212). I've filed two Thrift issues (https://issues.apache.org/jira/browse/THRIFT-1323 and https://issues.apache.org/jira/browse/THRIFT-1324), but in the meantime, I suggest that CustomTThreadPoolServer log the exception at DEBUG level in order to support easier troubleshooting.
was:
Currently CustomTThreadPoolServer, like the Thrift TThreadPoolServer, silently ignores TTransportException in its run() method. This is appropriate in most cases because TTransportException occurs fairly often when client connections die. However TTransportException is also thrown when TFramedTransport encounters a frame that is larger than thrift_framed_transport_size_in_mb. In that case, silently exiting the run loop leads to a SocketException on the client side which can be both difficult to diagnose, in part because nothing is logged by Cassandra, and high-impact, because the client may respond by marking the server node down and retrying the too-large request on another node, where it also fails. This process repeated leads to the entire cluster being marked down (see https://github.com/rantav/hector/issues/212). I've filed two Thrift issues (https://issues.apache.org/jira/browse/THRIFT-1323 and https://issues.apache.org/jira/browse/THRIFT-1324), but in the meantime, I suggest that CustomTThreadPoolServer log the exception at DEBUG level in order to support easier troubleshooting.
I can submit a patch with the added log message.
> CustomTThreadPoolServer should log TTransportException at DEBUG level
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-3142
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3142
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jim Ancona
> Attachments: v1-0001-CASSANDRA-3142-Add-debug-level-logging-of-TTransportEx.txt
>
>
> Currently CustomTThreadPoolServer, like the Thrift TThreadPoolServer, silently ignores TTransportException in its run() method. This is appropriate in most cases because TTransportException occurs fairly often when client connections die. However TTransportException is also thrown when TFramedTransport encounters a frame that is larger than thrift_framed_transport_size_in_mb. In that case, silently exiting the run loop leads to a SocketException on the client side which can be both difficult to diagnose, in part because nothing is logged by Cassandra, and high-impact, because the client may respond by marking the server node down and retrying the too-large request on another node, where it also fails. This process repeated leads to the entire cluster being marked down (see https://github.com/rantav/hector/issues/212). I've filed two Thrift issues (https://issues.apache.org/jira/browse/THRIFT-1323 and https://issues.apache.org/jira/browse/THRIFT-1324), but in the meantime, I suggest that CustomTThreadPoolServer log the exception at DEBUG level in order to support easier troubleshooting.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3142) CustomTThreadPoolServer should
log TTransportException at DEBUG level
Posted by "Jim Ancona (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Ancona updated CASSANDRA-3142:
----------------------------------
Attachment: v1-0001-CASSANDRA-3142-Add-debug-level-logging-of-TTransportEx.txt
> CustomTThreadPoolServer should log TTransportException at DEBUG level
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-3142
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3142
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jim Ancona
> Attachments: v1-0001-CASSANDRA-3142-Add-debug-level-logging-of-TTransportEx.txt
>
>
> Currently CustomTThreadPoolServer, like the Thrift TThreadPoolServer, silently ignores TTransportException in its run() method. This is appropriate in most cases because TTransportException occurs fairly often when client connections die. However TTransportException is also thrown when TFramedTransport encounters a frame that is larger than thrift_framed_transport_size_in_mb. In that case, silently exiting the run loop leads to a SocketException on the client side which can be both difficult to diagnose, in part because nothing is logged by Cassandra, and high-impact, because the client may respond by marking the server node down and retrying the too-large request on another node, where it also fails. This process repeated leads to the entire cluster being marked down (see https://github.com/rantav/hector/issues/212). I've filed two Thrift issues (https://issues.apache.org/jira/browse/THRIFT-1323 and https://issues.apache.org/jira/browse/THRIFT-1324), but in the meantime, I suggest that CustomTThreadPoolServer log the exception at DEBUG level in order to support easier troubleshooting.
> I can submit a patch with the added log message.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3142) CustomTThreadPoolServer should
log TTransportException at DEBUG level
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098983#comment-13098983 ]
Hudson commented on CASSANDRA-3142:
-----------------------------------
Integrated in Cassandra-0.8 #318 (See [https://builds.apache.org/job/Cassandra-0.8/318/])
CustomTThreadPoolServer to log TTransportException at DEBUG level
patch by Jim Ancona; reviewed by Pavel Yaskevich for CASSANDRA-3142
xedin : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166173
Files :
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/thrift/CustomTThreadPoolServer.java
> CustomTThreadPoolServer should log TTransportException at DEBUG level
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-3142
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3142
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jim Ancona
> Assignee: Jim Ancona
> Priority: Minor
> Fix For: 0.8.6
>
> Attachments: v1-0001-CASSANDRA-3142-Add-debug-level-logging-of-TTransportEx.txt
>
>
> Currently CustomTThreadPoolServer, like the Thrift TThreadPoolServer, silently ignores TTransportException in its run() method. This is appropriate in most cases because TTransportException occurs fairly often when client connections die. However TTransportException is also thrown when TFramedTransport encounters a frame that is larger than thrift_framed_transport_size_in_mb. In that case, silently exiting the run loop leads to a SocketException on the client side which can be both difficult to diagnose, in part because nothing is logged by Cassandra, and high-impact, because the client may respond by marking the server node down and retrying the too-large request on another node, where it also fails. This process repeated leads to the entire cluster being marked down (see https://github.com/rantav/hector/issues/212). I've filed two Thrift issues (https://issues.apache.org/jira/browse/THRIFT-1323 and https://issues.apache.org/jira/browse/THRIFT-1324), but in the meantime, I suggest that CustomTThreadPoolServer log the exception at DEBUG level in order to support easier troubleshooting.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira