You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Nobuaki Sukegawa (JIRA)" <ji...@apache.org> on 2015/07/01 15:41:05 UTC
[jira] [Commented] (THRIFT-3213) make cross should indicate when it
skips a known failing test
[ https://issues.apache.org/jira/browse/THRIFT-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14610256#comment-14610256 ]
Nobuaki Sukegawa commented on THRIFT-3213:
------------------------------------------
{quote}
Ideally I would like to see make cross actually try the test anyway,
{quote}
This can be achieved by removing {{-s}} flag from the command in Makefile.am.
Downside of this is the time it takes. Many of them result in timeout with client hang, taking more time than others.
I believe it was far less than 50min limit of Travis so it should be definitely doable, though.
By the way, I recommend running {{test/test.py}} directly rather than {{make cross}} if you're looking into this.
Minimal documentation can be found in {{test/README.md}}.
> make cross should indicate when it skips a known failing test
> -------------------------------------------------------------
>
> Key: THRIFT-3213
> URL: https://issues.apache.org/jira/browse/THRIFT-3213
> Project: Thrift
> Issue Type: Improvement
> Components: Test Suite
> Affects Versions: 0.9.2
> Environment: Ubuntu 12.04 LTS with cpp, java, python, perl, php, and go enabled.
> Reporter: James E. King, III
> Assignee: Roger Meier
> Priority: Minor
>
> I do not know if this affects earlier releases. I am trying to get {{cpp}} and {{go}} to run in a {{make cross}}. Currently I can make cpp on both sides, and go on both sides, but when I try to mix them up then no tests run:
> {noformat}
> jking@dvm-4:~/thrift$ /usr/bin/python test/test.py -s --server cpp --client cpp
> Apache Thrift - Integration Test Suite
> Tue Jun 30 16:26:24 2015
> ======================================================================
> server-client: protocol: transport: result:
> cpp-cpp compact buffered-domain success(0)
> cpp-cpp compact framed-ip success(0)
> cpp-cpp compact buffered-ip success(0)
> cpp-cpp compact http-ip-ssl success(0)
> cpp-cpp compact framed-ip-ssl success(0)
> cpp-cpp compact buffered-ip-ssl success(0)
> cpp-cpp binary buffered-ip-ssl success(0)
> cpp-cpp compact framed-domain success(0)
> cpp-cpp binary http-ip-ssl success(0)
> cpp-cpp binary buffered-ip success(0)
> cpp-cpp binary framed-ip-ssl success(0)
> cpp-cpp binary buffered-domain success(0)
> cpp-cpp binary framed-domain success(0)
> cpp-cpp binary framed-ip success(0)
> cpp-cpp json http-ip-ssl success(0)
> cpp-cpp json buffered-ip success(0)
> cpp-cpp json buffered-ip-ssl success(0)
> cpp-cpp json buffered-domain success(0)
> cpp-cpp json framed-ip-ssl success(0)
> cpp-cpp json framed-ip success(0)
> cpp-cpp json framed-domain success(0)
> ======================================================================
> No unexpected failures.
> You can browse results at:
> file:///home/jking/thrift/test/result.html
> # If you use Chrome, run:
> # cd /home/jking/thrift/test
> # python -m SimpleHTTPServer 8001
> # then browse:
> # http://localhost:8001/result.html
> Full log for each test is here:
> test/log/client_server_protocol_transport_client.log
> test/log/client_server_protocol_transport_server.log
> 0 failed of 21 tests in total.
> Test execution took 12.6 seconds.
> Tue Jun 30 16:26:37 2015
> jking@dvm-4:~/thrift$ ps ax | grep est
> 20778 pts/4 S+ 0:00 grep --color=auto est
> jking@dvm-4:~/thrift$ /usr/bin/python test/test.py -s --server cpp --client go
> Apache Thrift - Integration Test Suite
> Tue Jun 30 16:26:45 2015
> ======================================================================
> server-client: protocol: transport: result:
> ======================================================================
> No unexpected failures.
> You can browse results at:
> file:///home/jking/thrift/test/result.html
> # If you use Chrome, run:
> # cd /home/jking/thrift/test
> # python -m SimpleHTTPServer 8001
> # then browse:
> # http://localhost:8001/result.html
> Full log for each test is here:
> test/log/client_server_protocol_transport_client.log
> test/log/client_server_protocol_transport_server.log
> 0 failed of 0 tests in total.
> Test execution took 0.1 seconds.
> Tue Jun 30 16:26:45 2015
> jking@dvm-4:~/thrift$ /usr/bin/python test/test.py -s --server go --client go
> Apache Thrift - Integration Test Suite
> Tue Jun 30 16:26:49 2015
> ======================================================================
> server-client: protocol: transport: result:
> go-go compact framed-ip success(0)
> go-go binary buffered-ip success(0)
> go-go compact buffered-ip success(0)
> go-go compact framed-ip-ssl success(0)
> go-go compact buffered-ip-ssl success(0)
> go-go binary buffered-ip-ssl success(0)
> go-go json buffered-ip success(0)
> go-go binary framed-ip success(0)
> go-go json framed-ip success(0)
> go-go json buffered-ip-ssl success(0)
> go-go json framed-ip-ssl success(0)
> go-go binary framed-ip-ssl success(0)
> ======================================================================
> No unexpected failures.
> You can browse results at:
> file:///home/jking/thrift/test/result.html
> # If you use Chrome, run:
> # cd /home/jking/thrift/test
> # python -m SimpleHTTPServer 8001
> # then browse:
> # http://localhost:8001/result.html
> Full log for each test is here:
> test/log/client_server_protocol_transport_client.log
> test/log/client_server_protocol_transport_server.log
> 0 failed of 12 tests in total.
> Test execution took 6.3 seconds.
> Tue Jun 30 16:26:55 2015
> jking@dvm-4:~/thrift$ /usr/bin/python test/test.py -s --server go --client cpp
> Apache Thrift - Integration Test Suite
> Tue Jun 30 16:26:59 2015
> ======================================================================
> server-client: protocol: transport: result:
> ======================================================================
> No unexpected failures.
> You can browse results at:
> file:///home/jking/thrift/test/result.html
> # If you use Chrome, run:
> # cd /home/jking/thrift/test
> # python -m SimpleHTTPServer 8001
> # then browse:
> # http://localhost:8001/result.html
> Full log for each test is here:
> test/log/client_server_protocol_transport_client.log
> test/log/client_server_protocol_transport_server.log
> 0 failed of 0 tests in total.
> Test execution took 0.1 seconds.
> Tue Jun 30 16:26:59 2015
> {noformat}
> By all accounts, the tests.json file indicates these tests have commonality and should be tested against each-other. Curiously when I use java as the server and go as the client, it works:
> {noformat}
> jking@dvm-4:~/thrift$ /usr/bin/python test/test.py -s --server java --client go
> Apache Thrift - Integration Test Suite
> Tue Jun 30 16:37:52 2015
> ======================================================================
> server-client: protocol: transport: result:
> java-go compact framed-ip success(0)
> java-go compact fastframed-framed-ip success(0)
> java-go compact buffered-ip success(0)
> java-go compact framed-ip-ssl success(0)
> java-go compact buffered-ip-ssl success(0)
> java-go compact fastframed-framed-ip-ssl success(0)
> java-go binary framed-ip success(0)
> java-go binary buffered-ip success(0)
> java-go binary fastframed-framed-ip success(0)
> java-go binary framed-ip-ssl success(0)
> java-go binary fastframed-framed-ip-ssl success(0)
> java-go binary buffered-ip-ssl success(0)
> java-go json framed-ip success(0)
> java-go json framed-ip-ssl success(0)
> java-go json buffered-ip success(0)
> java-go json fastframed-framed-ip success(0)
> java-go json buffered-ip-ssl success(0)
> java-go json fastframed-framed-ip-ssl success(0)
> ======================================================================
> {noformat}
> Assigning this to [~roger.meier] for initial triage.
> It turns out that this is a function of the {{test/known_failures_Linux.json}} file. If a test is omitted because it is specifically skipped, there should be an entry on the output that indicates it was skipped so that it doesn't look like something went wrong. Ideally I would like to see {{make cross}} actually try the test anyway, and then if it fails and the entry is in the known failures list, then mark it as skipped. That way tests that were marked as failed in the past will be tried and might even succeed, giving better test coverage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)