[jira] [Commented] (CASSANDRA-10686) cqlsh schema refresh on timeout dtest is flaky

["Paulo Motta (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065886#comment-15065886 ] 

Paulo Motta commented on CASSANDRA-10686:
-----------------------------------------

We assume a {{TimedOutError}} is thrown when there is a schema mismatch after a DDL statement, so we reload the schema best-effort and print a warning message so the user knows there is a schema mismatch happening. However, when {{max_schema_agreement_wait < client_timeout}}, a {{TimedOutError}} is not thrown because the driver gives up reloading the schema and the original request complete successfully. 

A funny thing is that when {{max_schema_agreement_wait == client_timeout}}, the behavior is undefined in that most of the times the DDL request fails with {{TimedOutError}} but sometimes it may not throw that. This happens because most of the times the {{client_timeout}} is triggered before the {{max_schema_agreement_wait}}. However, in some edge situations the {{max_schema_agreement_wait}} may give up before the {{client_timeout}} so the update schema does not throw a {{TimedOutError}} as expected. 

In our case {{max_schema_agreement_wait == client_timeout == 10}}, so that's why the test is flaky (works most of the times, but fails sometimes). The issue is reproducible by making {{max_schema_agreement_wait}} < {{client_timeout}} to force the {{TimedOutError}} not being thrown, so the test fails.

In order to force {{max_schema_agrement_wait > client_timeout}} we set {{max_schema_agreement_wait = max(default_max_schema_agreement_wait, client_timeout+1)}}. I also opened a python driver issue ([PYTHON-458|https://datastax-oss.atlassian.net/browse/PYTHON-458]) to unify the behavior and always throw an exception on schema mismatch after a DDL regardless of the value of {{max_schema_agreement_wait}}.

Setting [~blerer] as reviewer, since he was the reviewer of the original issue (CASSANDRA-9689).

||2.1||2.2||3.0||trunk||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.1...pauloricardomg:2.1-10686]|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...pauloricardomg:2.2-10686]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...pauloricardomg:3.0-10686]|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-10686]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-10686-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-10686-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-10686-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-10686-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-10686-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-10686-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-10686-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-10686-dtest/lastCompletedBuild/testReport/]|

> cqlsh schema refresh on timeout dtest is flaky 
> -----------------------------------------------
>
>                 Key: CASSANDRA-10686
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10686
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Testing, Tools
>            Reporter: Joel Knighton
>            Assignee: Paulo Motta
>            Priority: Minor
>
> [flaky 3.0 runs|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/lastCompletedBuild/testReport/cqlsh_tests.cqlsh_tests/TestCqlsh/test_refresh_schema_on_timeout_error/history/]
> [flaky 2.2 runs|http://cassci.datastax.com/job/cassandra-2.2_dtest/381/testReport/cqlsh_tests.cqlsh_tests/TestCqlsh/test_refresh_schema_on_timeout_error/history/]
> [flaky 2.1 runs|http://cassci.datastax.com/job/cassandra-2.1_dtest/324/testReport/cqlsh_tests.cqlsh_tests/TestCqlsh/test_refresh_schema_on_timeout_error/history/]
> As far as I can tell, the issue could be with the test or the original issue. Pinging [~pauloricardomg] since he knows this best.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)