You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Tyson Hamilton (Jira)" <ji...@apache.org> on 2021/07/12 22:32:00 UTC
[jira] [Created] (BEAM-12603) Flaky:
apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers
Tyson Hamilton created BEAM-12603:
-------------------------------------
Summary: Flaky: apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers
Key: BEAM-12603
URL: https://issues.apache.org/jira/browse/BEAM-12603
Project: Beam
Issue Type: Improvement
Components: sdk-py-core, test-failures
Reporter: Tyson Hamilton
The `apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers` tests are flaky and causing precommit failures that seem similar.
`test_pardo_windowed_side_inputs` : https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/4417/console
{code:java}
23:24:27 Traceback (most recent call last):
23:24:27 File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
23:24:27 self.run()
23:24:27 File "/usr/lib/python3.6/threading.py", line 864, in run
23:24:27 self._target(*self._args, **self._kwargs)
23:24:27 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/runners/portability/local_job_service.py", line 274, in _run_job
23:24:27 self._pipeline_proto)
23:24:27 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 202, in run_via_runner_api
23:24:27 self._check_requirements(pipeline_proto)
23:24:27 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 301, in _check_requirements
23:24:27 'Unable to run pipeline with requirement: %s' % requirement)
23:24:27 ValueError: Unable to run pipeline with requirement: unsupported_requirement
23:24:27
23:24:27 .... [ 42%]
23:24:29 ...sssssssss...........ssssssss.sssssssssssssssssss................s.... [ 44%]
23:24:32 ...s.........s.
23:24:32 > Task :sdks:python:test-suites:dataflow:py37:preCommitIT_batch_V2
23:24:32 INFO:apache_beam.runners.dataflow.dataflow_runner:2021-07-12T06:24:29.370Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
23:24:52
23:24:52 > Task :sdks:python:test-suites:tox:py37:testPy37Cython
23:24:52 .....s.............................Exception in thread Thread-943:
23:24:52 Traceback (most recent call last):
23:24:52 File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
23:24:52 self.run()
23:24:52 File "/usr/lib/python3.7/threading.py", line 865, in run
23:24:52 self._target(*self._args, **self._kwargs)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/local_job_service.py", line 274, in _run_job
23:24:52 self._pipeline_proto)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 202, in run_via_runner_api
23:24:52 self._check_requirements(pipeline_proto)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 301, in _check_requirements
23:24:52 'Unable to run pipeline with requirement: %s' % requirement)
23:24:52 ValueError: Unable to run pipeline with requirement: unsupported_requirement
23:24:52
23:24:52 ........ [ 41%]
23:24:52 .........sssssssssssssssssssssssssssssssssssssssssssssssssssssssssss...s [ 42%]
23:24:52 sssss.ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 44%]
23:24:52 ssssssssssss..............Exception in thread Thread-947:
23:24:52 Traceback (most recent call last):
23:24:52 File "apache_beam/runners/common.py", line 1223, in apache_beam.runners.common.DoFnRunner.process
23:24:52 return self.do_fn_invoker.invoke_process(windowed_value)
23:24:52 File "apache_beam/runners/common.py", line 752, in apache_beam.runners.common.PerWindowInvoker.invoke_process
23:24:52 self._invoke_process_per_window(
23:24:52 File "apache_beam/runners/common.py", line 877, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
23:24:52 self.process_method(*args_for_process),
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/transforms/core.py", line 1568, in <lambda>
23:24:52 wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)]
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/testing/util.py", line 190, in _equal
23:24:52 raise BeamAssertException(msg)
23:24:52 apache_beam.testing.util.BeamAssertException: Failed assert: ['a'] == ['a', 'b'], unexpected elements ['b']
23:24:52
23:24:52 During handling of the above exception, another exception occurred:
23:24:52
23:24:52 Traceback (most recent call last):
23:24:52 File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
23:24:52 self.run()
23:24:52 File "/usr/lib/python3.7/threading.py", line 865, in run
23:24:52 self._target(*self._args, **self._kwargs)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/local_job_service.py", line 274, in _run_job
23:24:52 self._pipeline_proto)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 206, in run_via_runner_api
23:24:52 return self.run_stages(stage_context, stages)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 385, in run_stages
23:24:52 runner_execution_context, bundle_context_manager)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 653, in _run_stage
23:24:52 bundle_manager))
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 770, in _run_bundle
23:24:52 data_input, data_output, input_timers, expected_timer_output)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1080, in process_bundle
23:24:52 result_future = self._worker_handler.control_conn.push(process_bundle_req)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py", line 378, in push
23:24:52 response = self.worker.do_instruction(request)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 602, in do_instruction
23:24:52 getattr(request, request_type), request.instruction_id)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 639, in process_bundle
23:24:52 bundle_processor.process_bundle(instruction_id))
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/bundle_processor.py", line 997, in process_bundle
23:24:52 element.data)
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/worker/bundle_processor.py", line 222, in process_encoded
23:24:52 self.output(decoded_value)
23:24:52 File "apache_beam/runners/worker/operations.py", line 351, in apache_beam.runners.worker.operations.Operation.output
23:24:52 def output(self, windowed_value, output_index=0):
23:24:52 File "apache_beam/runners/worker/operations.py", line 353, in apache_beam.runners.worker.operations.Operation.output
23:24:52 cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
23:24:52 File "apache_beam/runners/worker/operations.py", line 215, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
23:24:52 self.consumer.process(windowed_value)
23:24:52 File "apache_beam/runners/worker/operations.py", line 712, in apache_beam.runners.worker.operations.DoOperation.process
23:24:52 with self.scoped_process_state:
23:24:52 File "apache_beam/runners/worker/operations.py", line 713, in apache_beam.runners.worker.operations.DoOperation.process
23:24:52 delayed_application = self.dofn_runner.process(o)
23:24:52 File "apache_beam/runners/common.py", line 1225, in apache_beam.runners.common.DoFnRunner.process
23:24:52 self._reraise_augmented(exn)
23:24:52 File "apache_beam/runners/common.py", line 1290, in apache_beam.runners.common.DoFnRunner._reraise_augmented
23:24:52 raise
23:24:52 File "apache_beam/runners/common.py", line 1223, in apache_beam.runners.common.DoFnRunner.process
23:24:52 return self.do_fn_invoker.invoke_process(windowed_value)
23:24:52 File "apache_beam/runners/common.py", line 571, in apache_beam.runners.common.SimpleInvoker.invoke_process
23:24:52 self.output_processor.process_outputs(
23:24:52 File "apache_beam/runners/common.py", line 1386, in apache_beam.runners.common._OutputProcessor.process_outputs
23:24:52 self.main_receivers.receive(windowed_value)
23:24:52 File "apache_beam/runners/worker/operations.py", line 215, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
23:24:52 self.consumer.process(windowed_value)
23:24:52 File "apache_beam/runners/worker/operations.py", line 712, in apache_beam.runners.worker.operations.DoOperation.process
23:24:52 with self.scoped_process_state:
23:24:52 File "apache_beam/runners/worker/operations.py", line 713, in apache_beam.runners.worker.operations.DoOperation.process
23:24:52 delayed_application = self.dofn_runner.process(o)
23:24:52 File "apache_beam/runners/common.py", line 1225, in apache_beam.runners.common.DoFnRunner.process
23:24:52 self._reraise_augmented(exn)
23:24:52 File "apache_beam/runners/common.py", line 1290, in apache_beam.runners.common.DoFnRunner._reraise_augmented
23:24:52 raise
23:24:52 File "apache_beam/runners/common.py", line 1223, in apache_beam.runners.common.DoFnRunner.process
23:24:52 return self.do_fn_invoker.invoke_process(windowed_value)
23:24:52 File "apache_beam/runners/common.py", line 571, in apache_beam.runners.common.SimpleInvoker.invoke_process
23:24:52 self.output_processor.process_outputs(
23:24:52 File "apache_beam/runners/common.py", line 1386, in apache_beam.runners.common._OutputProcessor.process_outputs
23:24:52 self.main_receivers.receive(windowed_value)
23:24:52 File "apache_beam/runners/worker/operations.py", line 215, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
23:24:52 self.consumer.process(windowed_value)
23:24:52 File "apache_beam/runners/worker/operations.py", line 712, in apache_beam.runners.worker.operations.DoOperation.process
23:24:52 with self.scoped_process_state:
23:24:52 File "apache_beam/runners/worker/operations.py", line 713, in apache_beam.runners.worker.operations.DoOperation.process
23:24:52 delayed_application = self.dofn_runner.process(o)
23:24:52 File "apache_beam/runners/common.py", line 1225, in apache_beam.runners.common.DoFnRunner.process
23:24:52 self._reraise_augmented(exn)
23:24:52 File "apache_beam/runners/common.py", line 1290, in apache_beam.runners.common.DoFnRunner._reraise_augmented
23:24:52 raise
23:24:52 File "apache_beam/runners/common.py", line 1223, in apache_beam.runners.common.DoFnRunner.process
23:24:52 return self.do_fn_invoker.invoke_process(windowed_value)
23:24:52 File "apache_beam/runners/common.py", line 571, in apache_beam.runners.common.SimpleInvoker.invoke_process
23:24:52 self.output_processor.process_outputs(
23:24:52 File "apache_beam/runners/common.py", line 1386, in apache_beam.runners.common._OutputProcessor.process_outputs
23:24:52 self.main_receivers.receive(windowed_value)
23:24:52 File "apache_beam/runners/worker/operations.py", line 215, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
23:24:52 self.consumer.process(windowed_value)
23:24:52 File "apache_beam/runners/worker/operations.py", line 712, in apache_beam.runners.worker.operations.DoOperation.process
23:24:52 with self.scoped_process_state:
23:24:52 File "apache_beam/runners/worker/operations.py", line 713, in apache_beam.runners.worker.operations.DoOperation.process
23:24:52 delayed_application = self.dofn_runner.process(o)
23:24:52 File "apache_beam/runners/common.py", line 1225, in apache_beam.runners.common.DoFnRunner.process
23:24:52 self._reraise_augmented(exn)
23:24:52 File "apache_beam/runners/common.py", line 1306, in apache_beam.runners.common.DoFnRunner._reraise_augmented
23:24:52 raise new_exn.with_traceback(tb)
23:24:52 File "apache_beam/runners/common.py", line 1223, in apache_beam.runners.common.DoFnRunner.process
23:24:52 return self.do_fn_invoker.invoke_process(windowed_value)
23:24:52 File "apache_beam/runners/common.py", line 752, in apache_beam.runners.common.PerWindowInvoker.invoke_process
23:24:52 self._invoke_process_per_window(
23:24:52 File "apache_beam/runners/common.py", line 877, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
23:24:52 self.process_method(*args_for_process),
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/transforms/core.py", line 1568, in <lambda>
23:24:52 wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)]
23:24:52 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/testing/util.py", line 190, in _equal
23:24:52 raise BeamAssertException(msg)
23:24:52 apache_beam.testing.util.BeamAssertException: Failed assert: ['a'] == ['a', 'b'], unexpected elements ['b'] [while running 'assert_that/Match']
23:24:52 .
{code}
`test_pack_combiners` : https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/4415/consoleFull
{code:java}
11:20:01 Traceback (most recent call last):
11:20:01 File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
11:20:01 self.run()
11:20:01 File "/usr/lib/python3.6/threading.py", line 864, in run
11:20:01 self._target(*self._args, **self._kwargs)
11:20:01 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/runners/portability/local_job_service.py", line 274, in _run_job
11:20:01 self._pipeline_proto)
11:20:01 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 202, in run_via_runner_api
11:20:01 self._check_requirements(pipeline_proto)
11:20:01 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 301, in _check_requirements
11:20:01 'Unable to run pipeline with requirement: %s' % requirement)
11:20:01 ValueError: Unable to run pipeline with requirement: unsupported_requirement
11:20:01 .
11:20:08 ...............................ssss..... [ 48%]
11:20:09 ...sssss.........s....sss.ssssssssssssssssssssssssssssssssssssssssssssss [ 50%]
11:20:13 sssssssssssssssss.ssssssssssssssss...................................... [ 51%]
11:20:18 ..............sss.sssssss..................................s............ [ 53%]
11:20:25 .........................................
11:20:25 > Task :sdks:python:test-suites:tox:py37:testPy37Cloud
11:20:25 ...............ssss.......... [ 48%]
11:20:26 .........................sss.sssssss.................................
11:20:26 > Task :sdks:python:test-suites:tox:py38:testPy38CloudCoverage
11:20:26 ................................................. [ 22%]
11:20:26 .................s..s................................................... [ 24%]
11:20:26 ...sssss................................................................ [ 25%]
11:20:26 ..................................s..................................... [ 27%]
11:20:26 .....................................................sssssss............ [ 28%]
11:20:26 ......................ss......................................ss........ [ 30%]
11:20:26 ......................................................................s. [ 31%]
11:20:27 .....s................................sssssssss...ssssss................ [ 33%]
11:20:37 .............s...sss.....................................sssssss........ [ 34%]
11:20:41 ....................................s...........................s...s... [ 36%]
11:20:51 .................s............sss......................................s [ 37%]
11:20:54 .sss........................s
11:20:54 > Task :sdks:python:test-suites:tox:py36:testPy36Cloud
11:20:54 .............s................. [ 54%]
11:20:56 .........ss...s..............................ss.....s.......s.....
11:20:56 > Task :sdks:python:test-suites:tox:py37:testPy37Cloud
11:20:56 ... [ 50%]
11:20:56 ..s.........................................s........................... [ 51%]
11:20:56 .................ss.ss........Exception in thread Thread-1007:
11:20:56 Traceback (most recent call last):
11:20:56 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/common.py", line 1223, in process
11:20:56 return self.do_fn_invoker.invoke_process(windowed_value)
11:20:56 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/common.py", line 572, in invoke_process
11:20:56 windowed_value, self.process_method(windowed_value.value))
11:20:56 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/transforms/core.py", line 1570, in <lambda>
11:20:56 wrapper = lambda x: [fn(x)]
11:20:56 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner_test.py", line 820, in raise_error
11:20:56 raise RuntimeError('x')
11:20:56 RuntimeError: x
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)