[jira] [Commented] (CASSANDRA-7886) Coordinator should not wait for read timeouts when replicas hit Exceptions

["Tyler Hobbs (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-7886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232299#comment-14232299 ] 

Tyler Hobbs commented on CASSANDRA-7886:
----------------------------------------

Thanks for your work so far, [~christianmovi]!  Here's my review of the current patch:

* Instead of using Unavailable when the protocol version is less than 4, use ReadTimeout.  Unavailable signals that some of the replicas are considered to be down, which is not the case here.  Plus, ReadTimeout is the error that is currently returned in these circumstances.
* In {{ErrorMessage.encodedSize()}}, there's some commented out code for READ_FAILURE handling.
* Instead of catching and ignoring {{TombstoneOverwhelmingException}} in multiple places, I suggest you move the logged error message into the TOE message and let it propagate (and be logged) like any other exception.
* Can you update {{docs/native_protocol_v4.spec}} with these changes?  You can look at the previous specs to see examples of the "changes from the previous version" section.
* In {{StorageProxy}}, the {{unavailables}} counter should not be incremented for read failures.  I suggest creating a new, separate failure counter.
* Also in {{StorageProxy}}, there's now quite a bit of code duplication around building error messages for {{ReadTimeoutExceptions}} and {{ReadFailureExceptions}}.  Can you condense those somewhat?

I'm also wondering if we should integrate failure notifications into speculative retry.  For example, if we get a failure response from our single data request replica (before the speculative retry timer is hit), it seems like we should preemptively send the extra request.  Similarly, if a failure response indicates that we won't meet the consistency level, we should preemptively send the extra request.  This doesn't need to be tackled in this ticket, but does anybody have thoughts on that?

> Coordinator should not wait for read timeouts when replicas hit Exceptions
> --------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7886
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7886
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>         Environment: Tested with Cassandra 2.0.8
>            Reporter: Christian Spriegel
>            Assignee: Christian Spriegel
>            Priority: Minor
>              Labels: protocolv4
>             Fix For: 3.0
>
>         Attachments: 7886_v1.txt, 7886_v2_trunk.txt, 7886_v3_trunk.txt
>
>
> *Issue*
> When you have TombstoneOverwhelmingExceptions occuring in queries, this will cause the query to be simply dropped on every data-node, but no response is sent back to the coordinator. Instead the coordinator waits for the specified read_request_timeout_in_ms.
> On the application side this can cause memory issues, since the application is waiting for the timeout interval for every request.Therefore, if our application runs into TombstoneOverwhelmingExceptions, then (sooner or later) our entire application cluster goes down :-(
> *Proposed solution*
> I think the data nodes should send a error message to the coordinator when they run into a TombstoneOverwhelmingException. Then the coordinator does not have to wait for the timeout-interval.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)