You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/03 23:03:08 UTC
[GitHub] [beam] kennknowles opened a new issue, #19241: Python Dataflow integration tests should export the pipeline console output to Jenkins Test Result section
kennknowles opened a new issue, #19241:
URL: https://github.com/apache/beam/issues/19241
I'm looking at a flake in a postcommit suite that happened last week in Python Dataflow integration test (https://issues.apache.org/jira/browse/BEAM-5415), and the existing logs are not sufficient to debug the problem. The test suite failed, however:
1. Test Result section in Jenkins postcommit only includes unit tests, and does not include any integration tests.
2. The only available information about the failure is buried in the Full Console Log and is not very informative (copy-pasted below).
3. There is a line in the console log pointing to: "XML: /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml", however as a non-committer I don't have access to Jenkins machines.
This makes rootcausing integration tests failures quite difficult, since we cannot even reliably identify an ID of a failed Dataflow job from the test output.
It would be helpful to capture and persist in Jenkins the invocation of integration test, and all the console output that SDK produces during test execution.
Sample available logs from console output.
13:41:54 test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... ERROR
13:41:54
13:41:54 \==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==
13:41:54 ERROR: test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)
13:41:54 \--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--
13:41:54 Traceback (most recent call last):
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 812, in run
13:41:54 test(orig)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 45, in __call__
13:41:54 return self.run(*arg, **kwarg)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 133, in run
13:41:54 self.runTest(result)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest
13:41:54 test(result)
13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
13:41:54 return self.run(*args, **kwds)
13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 329, in run
13:41:54 testMethod()
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py", line 161, in test_leader_board_it
13:41:54 self.test_pipeline.get_full_options_as_args(**extra_opts))
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py", line 345, in run
13:41:54 'total_score': 'INTEGER',
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 414, in __exit__
13:41:54 self.run().wait_until_finish()
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 394, in run
13:41:54 self.to_runner_api(), self.runner, self._options).run(False)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 407, in run
13:41:54 return self.runner.run_pipeline(self)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", line 68, in run_pipeline
13:41:54 self.result.cancel()
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 1167, in cancel
13:41:54 self._update_job()
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 1078, in _update_job
13:41:54 self._job = self._runner.dataflow_client.get_job(self.job_id())
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper
13:41:54 return fun(*args, **kwargs)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py", line 629, in get_job
13:41:54 response = self._client.projects_locations_jobs.Get(request)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py", line 659, in Get
13:41:54 config, request, global_params=global_params)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py", line 720, in _RunMethod
13:41:54 http, http_request, **opts)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py", line 346, in MakeRequest
13:41:54 check_response_func=check_response_func)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py", line 396, in _MakeRequestNoRetry
13:41:54 redirections=redirections, connection_type=connection_type)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py", line 193, in new_request
13:41:54 redirections, connection_type)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py", line 282, in request
13:41:54 connection_type=connection_type)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1694, in request
13:41:54 (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1434, in _request
13:41:54 (response, content) = self._conn_request(conn, request_uri, method, body, headers)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1390, in _conn_request
13:41:54 response = conn.getresponse()
13:41:54 File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
13:41:54 response.begin()
13:41:54 File "/usr/lib/python2.7/httplib.py", line 453, in begin
13:41:54 version, status, reason = self._read_status()
13:41:54 File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
13:41:54 line = self.fp.readline(_MAXLINE **** 1)
13:41:54 File "/usr/lib/python2.7/socket.py", line 480, in readline
13:41:54 data = self._sock.recv(self._rbufsize)
13:41:54 File "/usr/lib/python2.7/ssl.py", line 756, in recv
13:41:54 return self.read(buflen)
13:41:54 File "/usr/lib/python2.7/ssl.py", line 643, in read
13:41:54 v = self._sslobj.read(len)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 276, in signalhandler
13:41:54 raise TimedOutException()
13:41:54 TimedOutException: 'test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)'
13:41:54
13:41:54 \--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--
13:41:54 XML: /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml
13:41:54 \--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--
13:41:54 Ran 14 tests in 3001.679s
13:41:54
13:41:54 FAILED (errors=1)
13:41:54
Imported from Jira [BEAM-5492](https://issues.apache.org/jira/browse/BEAM-5492). Original Jira may contain additional context.
Reported by: tvalentyn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #19241: Python Dataflow integration tests should export the pipeline Job ID and console output to Jenkins Test Result section
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #19241:
URL: https://github.com/apache/beam/issues/19241#issuecomment-1287330331
@andoni-guzman @benWize hi folks, just checking if this is something you'll be able to look at. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #19241: Python Dataflow integration tests should export the pipeline console output to Jenkins Test Result section
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #19241:
URL: https://github.com/apache/beam/issues/19241#issuecomment-1146438686
Unable to assign user @andoni-guzman. If able, self-assign, otherwise tag @damccorm so that he can assign you. Because of GitHub's spam prevention system, your activity is required to enable assignment in this repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #19241: Python Dataflow integration tests should export the pipeline Job ID and console output to Jenkins Test Result section
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #19241:
URL: https://github.com/apache/beam/issues/19241#issuecomment-1453680171
Based on how long we have gone without this, I think P2 is the right priority.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #19241: Python Dataflow integration tests should export the pipeline console output to Jenkins Test Result section
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #19241:
URL: https://github.com/apache/beam/issues/19241#issuecomment-1249660293
I think some work was done on this front, but I believe the problem is still present, at least in some suites: https://github.com/apache/beam/issues/23277
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org