[jira] [Created] (CASSANDRA-12171) counter mismatch during rolling upgrade from 2.2 to 3.0

["Russ Hatch (JIRA)" <ji...@apache.org>]

Russ Hatch created CASSANDRA-12171:
--------------------------------------

             Summary: counter mismatch during rolling upgrade from 2.2 to 3.0
                 Key: CASSANDRA-12171
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12171
             Project: Cassandra
          Issue Type: Bug
            Reporter: Russ Hatch
            Assignee: Aleksey Yeschenko


This may occur on other versions, but 3.0 is where I observed it recently.

N=RF=3, counter writes at quorum, reads at quorum.

This is being seen on some upgrade tests I'm currently repairing here: https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix (this branch is to resolve an issue where counters were not being properly tested during rolling upgrade tests).

The test runs a continuous counter incrementing process, as well as a continuous counter checking process. Once a counter value has been verified, the test code makes it eligible to be incremented again.

The test is encountering the problem when trying to check an expected counter value and not matching expectations, for example:
{noformat}
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rhatch/git/cstar/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", line 210, in counter_checker
    tester.assertEqual(expected_count, actual_count)
  File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 1 != 2
ERROR
{noformat}

To check if something else could be going on, I did an experiment where I changed the test to not upgrade nodes (just drain, stop, start) and the mismatch didn't occur in several attempts. So it appears something about upgrading is possibly the culprit.

To run the test and repro locally:
{noformat}
grab my dtest branch at https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix

export UPGRADE_TEST_RUN=true

nosetests -v upgrade_tests/upgrade_through_versions_test.py:TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_test
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)