[jira] [Commented] (CASSANDRA-9418) Fix dtests on WIndows

["Joshua McKenzie (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-9418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561704#comment-14561704 ] 

Joshua McKenzie commented on CASSANDRA-9418:
--------------------------------------------

A simple ccm PR ([link|https://github.com/pcmanus/ccm/pull/289]) and a dtest PR ([link|https://github.com/riptano/cassandra-dtest/pull/299]) have knocked off 226 more failures and gotten us down to 338 failures from 564 ([test run here|http://cassci.datastax.com/view/trunk/job/trunk_dtest_win32/271/]).

In the logs there's a very high count of "Found running cassandra process with pid: 15776. Killing." from the dtest change to kill running cassandra processes during Tester.setUp(). The matching ccm PR was intended to give us something to correlate w/those failures to find out if there were specific tests that were hanging and address them, however this first test run gave us over 100 instances of hung tests that had to be killed. Hopefully there's a systemic infrastructural issue that we can address that will help with those errors.

There's still a high number of errors indicating missing system.log, conf\*, or bin\* files; there's some environmental silliness occurring I haven't gotten any clarity on yet as it's CI specific. The PR for ccm didn't actually print to stderr so once I figure out why CI / nosetests is absorbing that output, I'll probably also put in some more debug information from ccm regarding starting and stopping clusters and point CI to that debug branch to get more information out of it.

> Fix dtests on WIndows
> ---------------------
>
>                 Key: CASSANDRA-9418
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9418
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Joshua McKenzie
>            Assignee: Joshua McKenzie
>              Labels: Windows
>             Fix For: 2.2.x
>
>
> There's a variety of infrastructural failures within dtest w/regards to windows that are causing tests to fail and those failures to cascade.
> Error: failure to delete commit log after a test / ccm cluster is stopped:
> {noformat}
> Traceback (most recent call last):
>   File "C:\src\cassandra-dtest\dtest.py", line 452, in tearDown
>     self._cleanup_cluster()
>   File "C:\src\cassandra-dtest\dtest.py", line 172, in _cleanup_cluster
>     self.cluster.remove()
>   File "build\bdist.win-amd64\egg\ccmlib\cluster.py", line 212, in remove
>     shutil.rmtree(self.get_path())
>   File "C:\Python27\lib\shutil.py", line 247, in rmtree
>     rmtree(fullname, ignore_errors, onerror)
>   File "C:\Python27\lib\shutil.py", line 247, in rmtree
>     rmtree(fullname, ignore_errors, onerror)
>   File "C:\Python27\lib\shutil.py", line 252, in rmtree
>     onerror(os.remove, fullname, sys.exc_info())
>   File "C:\Python27\lib\shutil.py", line 250, in rmtree
>     os.remove(fullname)
> WindowsError: [Error 5] Access is denied: 'c:\\temp\\dtest-4rxq2i\\test\\node1\\commitlogs\\CommitLog-5-1431969131917.log'
> {noformat}
> Cascading error: implication is that tests aren't shutting down correctly and subsequent tests cannot start:
> {noformat}
> 06:00:20 ERROR: test_incr_decr_super_remove (thrift_tests.TestMutations)
> 06:00:20 ----------------------------------------------------------------------
> 06:00:20 Traceback (most recent call last):
> 06:00:20   File "D:\jenkins\workspace\trunk_dtest_win32\cassandra-dtest\thrift_tests.py", line 55, in setUp
> 06:00:20     cluster.start()
> 06:00:20   File "build\bdist.win-amd64\egg\ccmlib\cluster.py", line 249, in start
> 06:00:20     p = node.start(update_pid=False, jvm_args=jvm_args, profile_options=profile_options)
> 06:00:20   File "build\bdist.win-amd64\egg\ccmlib\node.py", line 457, in start
> 06:00:20     common.check_socket_available(itf)
> 06:00:20   File "build\bdist.win-amd64\egg\ccmlib\common.py", line 341, in check_socket_available
> 06:00:20     raise UnavailableSocketError("Inet address %s:%s is not available: %s" % (addr, port, msg))
> 06:00:20 UnavailableSocketError: Inet address 127.0.0.1:9042 is not available: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions
> 06:00:20 -------------------- >> begin captured logging << --------------------
> 06:00:20 dtest: DEBUG: removing ccm cluster test at: d:\temp\dtest-a5iny5
> 06:00:20 dtest: DEBUG: cluster ccm directory: d:\temp\dtest-dalzcy
> 06:00:20 --------------------- >> end captured logging << ---------------------
> {noformat}
> I've also seen (and am debugging) an error where a node just fails to start via ccm.
> I'll update this ticket with PR's to dtest or other observations of interest.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)