You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Alan Myrvold (JIRA)" <ji...@apache.org> on 2018/10/01 17:30:00 UTC
[jira] [Commented] (BEAM-5492) Python Dataflow integration tests
should export the pipeline console output to Jenkins Test Result section
[ https://issues.apache.org/jira/browse/BEAM-5492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634362#comment-16634362 ]
Alan Myrvold commented on BEAM-5492:
------------------------------------
The Java unit tests have:
// Publish all test results to Jenkins
publishers {
archiveJunit('**/build/test-results/**/*.xml')
}
Depending on where the logs are, we can update the job_PostCommit_Python_ValidatesRunner_Dataflow job to included the needed logs.
> Python Dataflow integration tests should export the pipeline console output to Jenkins Test Result section
> ----------------------------------------------------------------------------------------------------------
>
> Key: BEAM-5492
> URL: https://issues.apache.org/jira/browse/BEAM-5492
> Project: Beam
> Issue Type: New Feature
> Components: testing
> Reporter: Valentyn Tymofieiev
> Assignee: Jason Kuster
> Priority: Major
>
> I'm looking at a flake in a postcommit suite that happened last week in Python Dataflow integration test (https://issues.apache.org/jira/browse/BEAM-5415), and the existing logs are not sufficient to debug the problem. The test suite failed, however:
> 1. Test Result section in Jenkins postcommit only includes unit tests, and does not include any integration tests.
> 2. The only available information about the failure is buried in the Full Console Log and is not very informative (copy-pasted below).
> 3. There is a line in the console log pointing to: "XML: /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml", however as a non-committer I don't have access to Jenkins machines.
> This makes rootcausing integration tests failures quite difficult, since we cannot even reliably identify an ID of a failed Dataflow job from the test output.
> It would be helpful to capture and persist in Jenkins the invocation of integration test, and all the console output that SDK produces during test execution.
> Sample available logs from console output.
> 13:41:54 test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... ERROR
> 13:41:54
> 13:41:54 ======================================================================
> 13:41:54 ERROR: test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)
> 13:41:54 ----------------------------------------------------------------------
> 13:41:54 Traceback (most recent call last):
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 812, in run
> 13:41:54 test(orig)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 45, in __call__
> 13:41:54 return self.run(*arg, **kwarg)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 133, in run
> 13:41:54 self.runTest(result)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest
> 13:41:54 test(result)
> 13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
> 13:41:54 return self.run(*args, **kwds)
> 13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> 13:41:54 testMethod()
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py", line 161, in test_leader_board_it
> 13:41:54 self.test_pipeline.get_full_options_as_args(**extra_opts))
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py", line 345, in run
> 13:41:54 'total_score': 'INTEGER',
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 414, in __exit__
> 13:41:54 self.run().wait_until_finish()
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 394, in run
> 13:41:54 self.to_runner_api(), self.runner, self._options).run(False)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 407, in run
> 13:41:54 return self.runner.run_pipeline(self)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", line 68, in run_pipeline
> 13:41:54 self.result.cancel()
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 1167, in cancel
> 13:41:54 self._update_job()
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 1078, in _update_job
> 13:41:54 self._job = self._runner.dataflow_client.get_job(self.job_id())
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper
> 13:41:54 return fun(*args, **kwargs)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py", line 629, in get_job
> 13:41:54 response = self._client.projects_locations_jobs.Get(request)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py", line 659, in Get
> 13:41:54 config, request, global_params=global_params)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py", line 720, in _RunMethod
> 13:41:54 http, http_request, **opts)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py", line 346, in MakeRequest
> 13:41:54 check_response_func=check_response_func)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py", line 396, in _MakeRequestNoRetry
> 13:41:54 redirections=redirections, connection_type=connection_type)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py", line 193, in new_request
> 13:41:54 redirections, connection_type)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py", line 282, in request
> 13:41:54 connection_type=connection_type)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1694, in request
> 13:41:54 (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1434, in _request
> 13:41:54 (response, content) = self._conn_request(conn, request_uri, method, body, headers)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1390, in _conn_request
> 13:41:54 response = conn.getresponse()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
> 13:41:54 response.begin()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 453, in begin
> 13:41:54 version, status, reason = self._read_status()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
> 13:41:54 line = self.fp.readline(_MAXLINE + 1)
> 13:41:54 File "/usr/lib/python2.7/socket.py", line 480, in readline
> 13:41:54 data = self._sock.recv(self._rbufsize)
> 13:41:54 File "/usr/lib/python2.7/ssl.py", line 756, in recv
> 13:41:54 return self.read(buflen)
> 13:41:54 File "/usr/lib/python2.7/ssl.py", line 643, in read
> 13:41:54 v = self._sslobj.read(len)
> 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 276, in signalhandler
> 13:41:54 raise TimedOutException()
> 13:41:54 TimedOutException: 'test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)'
> 13:41:54
> 13:41:54 ----------------------------------------------------------------------
> 13:41:54 XML: /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml
> 13:41:54 ----------------------------------------------------------------------
> 13:41:54 Ran 14 tests in 3001.679s
> 13:41:54
> 13:41:54 FAILED (errors=1)
> 13:41:54
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)