[jira] [Commented] (CASSANDRA-9893) Fix upgrade tests from #9704 that are still failing

["Tyler Hobbs (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-9893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662626#comment-14662626 ] 

Tyler Hobbs commented on CASSANDRA-9893:
----------------------------------------

The three tests that appear to still be failing for me are: {{bug_5240_test}}, {{bug_5732_test}}, and {{edge_2i_on_complex_pk_test}}.  There may be other occasional failures that don't show up on every run due to different token assignments (only the upgraded or non-upgraded node may own a particularly partition).

The {{bug_5732_test}} failure is due to CASSANDRA-10006.

The new tests have a few useful environment variables that make it easier to debug problems.  These are documented at the top of {{upgrade_tests/upgrade_base.py}}.  While working on these tests, I normally used two cassandra directories: one for 3.0, and one for 2.1 with debug logging.  For example, I might run a test like this:

{noformat}
UPGRADE_MODE=normal DEBUG=true CASSANDRA_DIR=/home/thobbs/cassandra OLD_CASSANDRA_DIR=/home/thobbs/cassandra2 nosetests  upgrade_tests/cql_tests.py:TestCQL.select_distinct_with_deletions_test
{noformat}

For 2.1 debug logging, I started this branch with an excessive amount of logging around command and response serialization: https://github.com/thobbs/cassandra/tree/8099-backward-compat-2.1-debug.

The {{UPGRADE_MODE}} variable is useful for seeing exactly what commands or responses 2.1 or 3.0 expect.  For example, to see how 2.1 handles a particular query, you can do:

{noformat}
UPGRADE_MODE=none KEEP_LOGS=true DEBUG=true CASSANDRA_DIR=/home/thobbs/cassandra OLD_CASSANDRA_DIR=/home/thobbs/cassandra2 nosetests  upgrade_tests/cql_tests.py:TestCQL.select_distinct_with_deletions_test
{noformat}

Note the extra {{KEEP_LOGS=true}}, as the test should  pass without the upgrade step.

In some cases I've had problems with the new auth schema in 3.0.  This can be worked around by doing something like this in {{upgrade_base.py}}:

{noformat}
node1.start(wait_for_binary_proto=True, jvm_args=['-Dcassandra.superuser_setup_delay_ms=1000000'])
{noformat}

However, I have not committed this, because I think this might cause legitimate problems for users.  We probably either need to do CASSANDRA-9289 or investigate why reconnections are still causing queries to be dropped.


> Fix upgrade tests from #9704 that are still failing
> ---------------------------------------------------
>
>                 Key: CASSANDRA-9893
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9893
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Sylvain Lebresne
>             Fix For: 3.0.0 rc1
>
>
> The first things to do on this ticket would be to commit Tyler's branch (https://github.com/thobbs/cassandra-dtest/tree/8099-backwards-compat) to the dtests so cassci run them. I've had to do a few minor modifications to have them run locally so someone which access to cassci should do it and make sure it runs properly.
> Once we have that, we should fix any test that isn't passing. I've ran the tests locally and I had 8 failures. for 2 of them, it sounds plausible that they'll get fixed by the patch of CASSANDRA-9775, though that's just a guess.  The rest where test that timeouted without a particular error in the log, and running some of them individually, they passed.  So we'll have to see if it's just my machine being overly slow when running them all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)