[jira] [Comment Edited] (THRIFT-2946) Enhance usability of cross test framework

["Nobuaki Sukegawa (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/THRIFT-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491032#comment-14491032 ] 

Nobuaki Sukegawa edited comment on THRIFT-2946 at 4/11/15 5:31 PM:
-------------------------------------------------------------------

Sorry, didn't notice until now.

Although it seems that it's already merged, a possible concern is that single test run with {{--update-expected-failures=overwrite}} may exclude flaky tests that occasionally fail.
Tests that involve C++ http server or nodejs buffered transport are major examples of such flakiness.

To avoid this, running tests several times with {{--update-expected-failures=merge}} after the first {{--update-expected-failures=overwrite}} run might be a reasonable solution.
I think I can submit a patch myself shortly after.

Alternatively, manually tracking list of flaky tests may be better in that we don't need to run the whole test multiple times and then hope we are lucky enough to catch every occasional failures.
But I'm not sure it's worth the complexity and the work (i.e. manual maintenance of the list) it introduces.

edit:
It turned out c_glib were only failing for clang (THRIFT-3091).
I updated the pullrequest to include the patch because it's needed for the correct list anyway.


was (Author: nsuke):
Sorry, didn't notice until now.

Although it seems that it's already merged, a possible concern is that single test run with {{--update-expected-failures=overwrite}} may exclude flaky tests that occasionally fail.
Tests that involve C++ http server or nodejs buffered transport are major examples of such flakiness.

To avoid this, running tests several times with {{--update-expected-failures=merge}} after the first {{--update-expected-failures=overwrite}} run might be a reasonable solution.
I think I can submit a patch myself shortly after.

Alternatively, manually tracking list of flaky tests may be better in that we don't need to run the whole test multiple times and then hope we are lucky enough to catch every occasional failures.
But I'm not sure it's worth the complexity and the work (i.e. manual maintenance of the list) it introduces.

By the way, seeing the failure list update, it seems that glib_c server is properly working on your environment.
I'm surprised by that because it never worked (for the cross tests) on my ArchLinux nor on Ubuntu VM.
Maybe the server code itself is fine and some tweak is needed in cross test side.

> Enhance usability of cross test framework
> -----------------------------------------
>
>                 Key: THRIFT-2946
>                 URL: https://issues.apache.org/jira/browse/THRIFT-2946
>             Project: Thrift
>          Issue Type: Sub-task
>          Components: Test Suite
>            Reporter: Nobuaki Sukegawa
>            Assignee: Nobuaki Sukegawa
>         Attachments: THRIFT-2946_add_make_fail.patch
>
>
> This patch makes cross test script more accessible with:
> * Reduced burden of preparation
> * Reduced execution time
> * Error reporting for newly introduced failrues
> Major enhancement:
> * Automake target that builds cross test dependencies, without invoking all the "make check" tests
> * Parallel execution
> * Tests known to be failing are listed in "known_failures_<platform>.json".
>    If anything not listed fails, developers/buildbots are notified with clear message/return code.
> For details and other minor changes, please refer to commit log of the pull request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)