[jira] [Created] (CASSANDRA-20059) TCM's Retry.Deadline#retryIndefinitely is dangerous if used with RemoteProcessor as the deadline does not impact message retries

["David Capwell (Jira)" <ji...@apache.org>]

David Capwell created CASSANDRA-20059:
-----------------------------------------

             Summary: TCM's Retry.Deadline#retryIndefinitely is dangerous if used with RemoteProcessor as the deadline does not impact message retries
                 Key: CASSANDRA-20059
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-20059
             Project: Cassandra
          Issue Type: Bug
          Components: Transactional Cluster Metadata
            Reporter: David Capwell


{code}
public static Deadline retryIndefinitely(long timeoutNanos, Meter retryMeter)
{
    return new Deadline(Clock.Global.nanoTime() + timeoutNanos,
                        new Retry.Jitter(Integer.MAX_VALUE, DEFAULT_BACKOFF_MS, new Random(), retryMeter))
    {
        @Override
        public boolean reachedMax()
        {
            return false;
        }

        @Override
        public long remainingNanos()
        {
            return timeoutNanos;
        }

        public String toString()
        {
            return String.format("RetryIndefinitely{tries=%d}", currentTries());
        }
    };
}
{code}

Sample usage pattern (example is in Accord, but same pattern exists in RemoteProcessor.commit)

{code}
Promise<LogState> request = new AsyncPromise<>();
List<InetAddressAndPort> candidates = new ArrayList<>(log.metadata().fullCMSMembers());
sendWithCallbackAsync(request,
                      Verb.TCM_RECONSTRUCT_EPOCH_REQ,
                      new ReconstructLogState(lowEpoch, highEpoch, includeSnapshot),
                      new CandidateIterator(candidates),
                      retryPolicy);
return request.get(retryPolicy.remainingNanos(), TimeUnit.NANOSECONDS);
{code}

The issue here is that the networking retry has no clue that we gave up waiting on the request, so we will keep retrying until success!  The reason for this is “reachedMax” is used to see if its safe to run again, but it isn’t as the deadline has passed!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org