You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2011/07/07 21:22:17 UTC
[jira] [Created] (CASSANDRA-2870) dynamic snitch + read repair off
can cause LOCAL_QUORUM reads to return spurious UnavailableException
dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return spurious UnavailableException
-----------------------------------------------------------------------------------------------------
Key: CASSANDRA-2870
URL: https://issues.apache.org/jira/browse/CASSANDRA-2870
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
Fix For: 0.7.8
When Read Repair is off, we want to avoid doing requests to more nodes than necessary to satisfy the ConsistencyLevel. ReadCallback does this here:
{code}
this.endpoints = repair || resolver instanceof RowRepairResolver
? endpoints
: endpoints.subList(0, Math.min(endpoints.size(), blockfor)); // min so as to not throw exception until assureSufficient is called
{code}
You can see that it is assuming that the "endpoints" list is sorted in order of preferred-ness for the read.
Then the LOCAL_QUORUM code in DatacenterReadCallback checks to see if we have enough nodes to do the read:
{code}
int localEndpoints = 0;
for (InetAddress endpoint : endpoints)
{
if (localdc.equals(snitch.getDatacenter(endpoint)))
localEndpoints++;
}
if (localEndpoints < blockfor)
throw new UnavailableException();
{code}
So if repair is off (so we truncate our endpoints list) AND dynamic snitch has decided that nodes in another DC are to be preferred over local ones, we'll throw UE even if all the replicas are healthy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2870) dynamic snitch + read repair
off can cause LOCAL_QUORUM reads to return spurious UnavailableException
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062308#comment-13062308 ]
Jonathan Ellis commented on CASSANDRA-2870:
-------------------------------------------
This has been present since LOCAL_QUORUM was introduced, so it's not a new regression. And a reasonable workaround exists (disable dynamic snitch). So no, I don't think we should hold up 0.7.7 for this.
> dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return spurious UnavailableException
> -----------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2870
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2870.txt
>
>
> When Read Repair is off, we want to avoid doing requests to more nodes than necessary to satisfy the ConsistencyLevel. ReadCallback does this here:
> {code}
> this.endpoints = repair || resolver instanceof RowRepairResolver
> ? endpoints
> : endpoints.subList(0, Math.min(endpoints.size(), blockfor)); // min so as to not throw exception until assureSufficient is called
> {code}
> You can see that it is assuming that the "endpoints" list is sorted in order of preferred-ness for the read.
> Then the LOCAL_QUORUM code in DatacenterReadCallback checks to see if we have enough nodes to do the read:
> {code}
> int localEndpoints = 0;
> for (InetAddress endpoint : endpoints)
> {
> if (localdc.equals(snitch.getDatacenter(endpoint)))
> localEndpoints++;
> }
> if (localEndpoints < blockfor)
> throw new UnavailableException();
> {code}
> So if repair is off (so we truncate our endpoints list) AND dynamic snitch has decided that nodes in another DC are to be preferred over local ones, we'll throw UE even if all the replicas are healthy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2870) dynamic snitch + read repair
off can cause LOCAL_QUORUM reads to return spurious UnavailableException
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062324#comment-13062324 ]
Jeremy Hanna commented on CASSANDRA-2870:
-----------------------------------------
Okay - it just seemed like a higher priority issue with the scope expanded. We'll probably just disable dynamic snitch until the fix is in a release then.
> dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return spurious UnavailableException
> -----------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2870
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2870.txt
>
>
> When Read Repair is off, we want to avoid doing requests to more nodes than necessary to satisfy the ConsistencyLevel. ReadCallback does this here:
> {code}
> this.endpoints = repair || resolver instanceof RowRepairResolver
> ? endpoints
> : endpoints.subList(0, Math.min(endpoints.size(), blockfor)); // min so as to not throw exception until assureSufficient is called
> {code}
> You can see that it is assuming that the "endpoints" list is sorted in order of preferred-ness for the read.
> Then the LOCAL_QUORUM code in DatacenterReadCallback checks to see if we have enough nodes to do the read:
> {code}
> int localEndpoints = 0;
> for (InetAddress endpoint : endpoints)
> {
> if (localdc.equals(snitch.getDatacenter(endpoint)))
> localEndpoints++;
> }
> if (localEndpoints < blockfor)
> throw new UnavailableException();
> {code}
> So if repair is off (so we truncate our endpoints list) AND dynamic snitch has decided that nodes in another DC are to be preferred over local ones, we'll throw UE even if all the replicas are healthy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2870) dynamic snitch + read repair
off can cause LOCAL_QUORUM reads to return spurious UnavailableException
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061846#comment-13061846 ]
Sylvain Lebresne commented on CASSANDRA-2870:
---------------------------------------------
+1
> dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return spurious UnavailableException
> -----------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2870
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2870.txt
>
>
> When Read Repair is off, we want to avoid doing requests to more nodes than necessary to satisfy the ConsistencyLevel. ReadCallback does this here:
> {code}
> this.endpoints = repair || resolver instanceof RowRepairResolver
> ? endpoints
> : endpoints.subList(0, Math.min(endpoints.size(), blockfor)); // min so as to not throw exception until assureSufficient is called
> {code}
> You can see that it is assuming that the "endpoints" list is sorted in order of preferred-ness for the read.
> Then the LOCAL_QUORUM code in DatacenterReadCallback checks to see if we have enough nodes to do the read:
> {code}
> int localEndpoints = 0;
> for (InetAddress endpoint : endpoints)
> {
> if (localdc.equals(snitch.getDatacenter(endpoint)))
> localEndpoints++;
> }
> if (localEndpoints < blockfor)
> throw new UnavailableException();
> {code}
> So if repair is off (so we truncate our endpoints list) AND dynamic snitch has decided that nodes in another DC are to be preferred over local ones, we'll throw UE even if all the replicas are healthy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2870) dynamic snitch + read repair off
can cause LOCAL_QUORUM reads to return spurious UnavailableException
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2870:
--------------------------------------
Attachment: 2870.txt
patch extracts the "give me the minimum set of endpoints necessary" code into preferredEndpoints(), and makes it DC-aware for LOCAL_QUORUM reads.
> dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return spurious UnavailableException
> -----------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2870
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2870.txt
>
>
> When Read Repair is off, we want to avoid doing requests to more nodes than necessary to satisfy the ConsistencyLevel. ReadCallback does this here:
> {code}
> this.endpoints = repair || resolver instanceof RowRepairResolver
> ? endpoints
> : endpoints.subList(0, Math.min(endpoints.size(), blockfor)); // min so as to not throw exception until assureSufficient is called
> {code}
> You can see that it is assuming that the "endpoints" list is sorted in order of preferred-ness for the read.
> Then the LOCAL_QUORUM code in DatacenterReadCallback checks to see if we have enough nodes to do the read:
> {code}
> int localEndpoints = 0;
> for (InetAddress endpoint : endpoints)
> {
> if (localdc.equals(snitch.getDatacenter(endpoint)))
> localEndpoints++;
> }
> if (localEndpoints < blockfor)
> throw new UnavailableException();
> {code}
> So if repair is off (so we truncate our endpoints list) AND dynamic snitch has decided that nodes in another DC are to be preferred over local ones, we'll throw UE even if all the replicas are healthy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2870) dynamic snitch + read repair
off can cause LOCAL_QUORUM reads to return spurious UnavailableException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062053#comment-13062053 ]
Hudson commented on CASSANDRA-2870:
-----------------------------------
Integrated in Cassandra-0.7 #526 (See [https://builds.apache.org/job/Cassandra-0.7/526/])
fix possibility of spuriousUnavailableException for LOCAL_QUORUM reads with dynamic snitch and read repair disabled
patch by jbellis; reviewed by slebresne for CASSANDRA-2870
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144380
Files :
* /cassandra/branches/cassandra-0.7/CHANGES.txt
* /cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/service/ReadCallback.java
* /cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/service/DatacenterReadCallback.java
> dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return spurious UnavailableException
> -----------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2870
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2870.txt
>
>
> When Read Repair is off, we want to avoid doing requests to more nodes than necessary to satisfy the ConsistencyLevel. ReadCallback does this here:
> {code}
> this.endpoints = repair || resolver instanceof RowRepairResolver
> ? endpoints
> : endpoints.subList(0, Math.min(endpoints.size(), blockfor)); // min so as to not throw exception until assureSufficient is called
> {code}
> You can see that it is assuming that the "endpoints" list is sorted in order of preferred-ness for the read.
> Then the LOCAL_QUORUM code in DatacenterReadCallback checks to see if we have enough nodes to do the read:
> {code}
> int localEndpoints = 0;
> for (InetAddress endpoint : endpoints)
> {
> if (localdc.equals(snitch.getDatacenter(endpoint)))
> localEndpoints++;
> }
> if (localEndpoints < blockfor)
> throw new UnavailableException();
> {code}
> So if repair is off (so we truncate our endpoints list) AND dynamic snitch has decided that nodes in another DC are to be preferred over local ones, we'll throw UE even if all the replicas are healthy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2870) dynamic snitch + read repair
off can cause LOCAL_QUORUM reads to return spurious UnavailableException
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062256#comment-13062256 ]
Jeremy Hanna commented on CASSANDRA-2870:
-----------------------------------------
This also appears to affect 0.7.6 and when read repair is not off. I didn't set read repair on my CFs (defaults to 100%) and tried a simple rowcount pig script using read consistency LOCAL_QUORUM and it fails with UE. I would think if that's the case, the priority should be higher and it should go in 0.7.7. Any thoughts?
> dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return spurious UnavailableException
> -----------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2870
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2870.txt
>
>
> When Read Repair is off, we want to avoid doing requests to more nodes than necessary to satisfy the ConsistencyLevel. ReadCallback does this here:
> {code}
> this.endpoints = repair || resolver instanceof RowRepairResolver
> ? endpoints
> : endpoints.subList(0, Math.min(endpoints.size(), blockfor)); // min so as to not throw exception until assureSufficient is called
> {code}
> You can see that it is assuming that the "endpoints" list is sorted in order of preferred-ness for the read.
> Then the LOCAL_QUORUM code in DatacenterReadCallback checks to see if we have enough nodes to do the read:
> {code}
> int localEndpoints = 0;
> for (InetAddress endpoint : endpoints)
> {
> if (localdc.equals(snitch.getDatacenter(endpoint)))
> localEndpoints++;
> }
> if (localEndpoints < blockfor)
> throw new UnavailableException();
> {code}
> So if repair is off (so we truncate our endpoints list) AND dynamic snitch has decided that nodes in another DC are to be preferred over local ones, we'll throw UE even if all the replicas are healthy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2870) dynamic snitch + read repair
off can cause LOCAL_QUORUM reads to return spurious UnavailableException
Posted by "Patrick Mackinlay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064609#comment-13064609 ]
Patrick Mackinlay commented on CASSANDRA-2870:
----------------------------------------------
In the default configuration of 0.7.6-2 (and other versions) LOCAL_QUORUM reads dont work. This is not a minor bug and should be fixed in the next release.
By default configuration I mean the tar ball that is distributed by the cassandra website.
The fact that it is not a regression just shows that this functionality was never properly tested.
> dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return spurious UnavailableException
> -----------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2870
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2870.txt
>
>
> When Read Repair is off, we want to avoid doing requests to more nodes than necessary to satisfy the ConsistencyLevel. ReadCallback does this here:
> {code}
> this.endpoints = repair || resolver instanceof RowRepairResolver
> ? endpoints
> : endpoints.subList(0, Math.min(endpoints.size(), blockfor)); // min so as to not throw exception until assureSufficient is called
> {code}
> You can see that it is assuming that the "endpoints" list is sorted in order of preferred-ness for the read.
> Then the LOCAL_QUORUM code in DatacenterReadCallback checks to see if we have enough nodes to do the read:
> {code}
> int localEndpoints = 0;
> for (InetAddress endpoint : endpoints)
> {
> if (localdc.equals(snitch.getDatacenter(endpoint)))
> localEndpoints++;
> }
> if (localEndpoints < blockfor)
> throw new UnavailableException();
> {code}
> So if repair is off (so we truncate our endpoints list) AND dynamic snitch has decided that nodes in another DC are to be preferred over local ones, we'll throw UE even if all the replicas are healthy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2870) dynamic snitch + read repair
off can cause LOCAL_QUORUM reads to return spurious UnavailableException
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064621#comment-13064621 ]
Jonathan Ellis commented on CASSANDRA-2870:
-------------------------------------------
It will be fixed in 0.7.8; 0.7.7 entered the release process before this was reported.
> dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return spurious UnavailableException
> -----------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2870
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2870.txt
>
>
> When Read Repair is off, we want to avoid doing requests to more nodes than necessary to satisfy the ConsistencyLevel. ReadCallback does this here:
> {code}
> this.endpoints = repair || resolver instanceof RowRepairResolver
> ? endpoints
> : endpoints.subList(0, Math.min(endpoints.size(), blockfor)); // min so as to not throw exception until assureSufficient is called
> {code}
> You can see that it is assuming that the "endpoints" list is sorted in order of preferred-ness for the read.
> Then the LOCAL_QUORUM code in DatacenterReadCallback checks to see if we have enough nodes to do the read:
> {code}
> int localEndpoints = 0;
> for (InetAddress endpoint : endpoints)
> {
> if (localdc.equals(snitch.getDatacenter(endpoint)))
> localEndpoints++;
> }
> if (localEndpoints < blockfor)
> throw new UnavailableException();
> {code}
> So if repair is off (so we truncate our endpoints list) AND dynamic snitch has decided that nodes in another DC are to be preferred over local ones, we'll throw UE even if all the replicas are healthy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira