You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2022/11/24 02:19:51 UTC
Build failed in Jenkins: beam_PreCommit_Python_Cron #6402
See <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/6402/display/redirect?page=changes>
Changes:
[noreply] Add test configurations for deterministic outputs on Dataflow (#24325)
[noreply] Updates ExpansionService to support dynamically discovering and
------------------------------------------
[...truncated 2.45 MB...]
================= 42 passed, 5 skipped, 37 warnings in 14.61s ==================
[1mpy38-pytorch-19 run-test-post: commands[0] | bash <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m___________________________________ summary ____________________________________
[32m py38-pytorch-19: commands succeeded
[0m[32m congratulations :)
[0m
> Task :sdks:python:test-suites:tox:py38:preCommitPy38
> Task :sdks:python:test-suites:dataflow:py37:preCommitIT_streaming_V2
[gw0] PASSED apache_beam/examples/streaming_wordcount_it_test.py::StreamingWordCountIT::test_streaming_wordcount_it
=============================== warnings summary ===============================
../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15
../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15
../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/pytest_preCommitIT-df-py37.xml> -
================== 1 passed, 3 warnings in 803.04s (0:13:23) ===================
> Task :sdks:python:test-suites:dataflow:py310:preCommitIT_streaming_V2
[gw0] FAILED apache_beam/examples/streaming_wordcount_it_test.py::StreamingWordCountIT::test_streaming_wordcount_it
=================================== FAILURES ===================================
_______________ StreamingWordCountIT.test_streaming_wordcount_it _______________
[gw0] linux -- Python 3.10.2 <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/build/gradleenv/2050596098/bin/python3.10>
self = <apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT testMethod=test_streaming_wordcount_it>
@pytest.mark.it_postcommit
def test_streaming_wordcount_it(self):
# Build expected dataset.
expected_msg = [('%d: 1' % num).encode('utf-8')
for num in range(DEFAULT_INPUT_NUMBERS)]
# Set extra options to the pipeline for test purpose
state_verifier = PipelineStateMatcher(PipelineState.RUNNING)
pubsub_msg_verifier = PubSubMessageMatcher(
self.project, self.output_sub.name, expected_msg, timeout=400)
extra_opts = {
'input_subscription': self.input_sub.name,
'output_topic': self.output_topic.name,
'wait_until_finish_duration': WAIT_UNTIL_FINISH_DURATION,
'on_success_matcher': all_of(state_verifier, pubsub_msg_verifier)
}
# Generate input data and inject to PubSub.
self._inject_numbers(self.input_topic, DEFAULT_INPUT_NUMBERS)
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
> streaming_wordcount.run(
self.test_pipeline.get_full_options_as_args(**extra_opts),
save_main_session=False)
apache_beam/examples/streaming_wordcount_it_test.py:118:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam/examples/streaming_wordcount.py:61: in run
with beam.Pipeline(options=pipeline_options) as p:
apache_beam/pipeline.py:600: in __exit__
self.result = self.run()
apache_beam/pipeline.py:577: in run
return self.runner.run_pipeline(self, self._options)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <apache_beam.runners.dataflow.test_dataflow_runner.TestDataflowRunner object at 0x7f9a7054f2e0>
pipeline = <apache_beam.pipeline.Pipeline object at 0x7f9a7054e440>
options = <apache_beam.options.pipeline_options.PipelineOptions object at 0x7f9a7054e650>
def run_pipeline(self, pipeline, options):
"""Execute test pipeline and verify test matcher"""
test_options = options.view_as(TestOptions)
on_success_matcher = test_options.on_success_matcher
wait_duration = test_options.wait_until_finish_duration
is_streaming = options.view_as(StandardOptions).streaming
# [BEAM-1889] Do not send this to remote workers also, there is no need to
# send this option to remote executors.
test_options.on_success_matcher = None
self.result = super().run_pipeline(pipeline, options)
if self.result.has_job:
# TODO(markflyhigh)(https://github.com/apache/beam/issues/18254): Use
# print since Nose dosen't show logs in some cases.
print('Worker logs: %s' % self.build_console_url(options))
_LOGGER.info('Console log: ')
_LOGGER.info(self.build_console_url(options))
try:
self.wait_until_in_state(PipelineState.RUNNING)
if is_streaming and not wait_duration:
_LOGGER.warning('Waiting indefinitely for streaming job.')
self.result.wait_until_finish(duration=wait_duration)
if on_success_matcher:
from hamcrest import assert_that as hc_assert_that
> hc_assert_that(self.result, pickler.loads(on_success_matcher))
E AssertionError:
E Expected: (Test pipeline expected terminated in state: RUNNING and Expected 500 messages.)
E but: Expected 500 messages. Got 514 messages. Diffs (item, count):
E Expected but not in actual: dict_items([(b'218: 1', 1), (b'230: 1', 1), (b'260: 1', 1), (b'264: 1', 1), (b'269: 1', 1), (b'281: 1', 1), (b'294: 1', 1), (b'400: 1', 1), (b'441: 1', 1)])
E Unexpected: dict_items([(b'132: 1', 1), (b'476: 1', 1), (b'233: 1', 1), (b'486: 1', 1), (b'434: 1', 1), (b'277: 1', 1), (b'212: 1', 1), (b'472: 1', 1), (b'467: 1', 1), (b'251: 1', 1), (b'475: 1', 1), (b'48: 1', 1), (b'448: 1', 1), (b'271: 1', 1), (b'101: 1', 1), (b'279: 1', 1), (b'485: 1', 1), (b'420: 1', 1), (b'198: 1', 1), (b'216: 1', 1), (b'249: 1', 1), (b'478: 1', 1), (b'422: 1', 1)])
E Unexpected (with all details): [(b'132: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 24, 651000, tzinfo=datetime.timezone.utc), ''), (b'476: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'233: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'486: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'132: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 330000, tzinfo=datetime.timezone.utc), ''), (b'434: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'277: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'212: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'472: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'467: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'472: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'251: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'475: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'48: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'448: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'271: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'212: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'467: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'101: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'279: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'485: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'420: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'198: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'476: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'233: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'434: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'277: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'216: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'249: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'478: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'422: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'486: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'251: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'475: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'48: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'448: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'271: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'198: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'101: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'279: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'485: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'420: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'216: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'249: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'478: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'422: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), '')]
apache_beam/runners/dataflow/test_dataflow_runner.py:70: AssertionError
------------------------------ Captured log call -------------------------------
INFO apache_beam.runners.portability.stager:stager.py:780 Executing command: ['<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/build/gradleenv/2050596098/bin/python3.10',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpejv1lyas/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp310', '--platform', 'manylinux2014_x86_64']
INFO apache_beam.runners.portability.stager:stager.py:330 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
INFO root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.10_sdk:2.44.0.dev
INFO root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221122
INFO root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221122" for Docker environment
INFO apache_beam.internal.gcp.auth:auth.py:130 Setting socket default timeout to 60 seconds.
INFO apache_beam.internal.gcp.auth:auth.py:132 socket default timeout is 60.0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/requirements.txt...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/requirements.txt in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/mock-2.0.0-py2.py3-none-any.whl...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/mock-2.0.0-py2.py3-none-any.whl in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/seaborn-0.12.1-py3-none-any.whl...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/seaborn-0.12.1-py3-none-any.whl in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/PyHamcrest-1.10.1-py3-none-any.whl...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/PyHamcrest-1.10.1-py3-none-any.whl in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/beautifulsoup4-4.11.1-py3-none-any.whl...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/beautifulsoup4-4.11.1-py3-none-any.whl in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/parameterized-0.7.5-py2.py3-none-any.whl...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/parameterized-0.7.5-py2.py3-none-any.whl in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/dataflow_python_sdk.tar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/dataflow_python_sdk.tar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/matplotlib-3.6.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/matplotlib-3.6.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/dataflow-worker.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/dataflow-worker.jar in 5 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/pipeline.pb...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1124020658-132746-vfw193y3.1669255618.133029/pipeline.pb in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:911 Create job: <Job
clientRequestId: '20221124020658134574-2410'
createTime: '2022-11-24T02:07:06.876277Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-11-23_18_07_06-10773682750582623979'
location: 'us-central1'
name: 'beamapp-jenkins-1124020658-132746-vfw193y3'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-11-24T02:07:06.876277Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:913 Created job with id: [2022-11-23_18_07_06-10773682750582623979]
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:914 Submitted job: 2022-11-23_18_07_06-10773682750582623979
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:915 To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-11-23_18_07_06-10773682750582623979?project=apache-beam-testing
INFO apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:58 Console log:
INFO apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:59 https://console.cloud.google.com/dataflow/jobs/us-central1/2022-11-23_18_07_06-10773682750582623979?project=apache-beam-testing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job 2022-11-23_18_07_06-10773682750582623979 is in state JOB_STATE_RUNNING
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:07.604Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:07.715Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-11-23_18_07_06-10773682750582623979. The number of workers will be between 1 and 100.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:07.750Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-11-23_18_07_06-10773682750582623979.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:11.866Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-b.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.344Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.384Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.464Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.513Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step group: GroupByKey not followed by a combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.552Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.583Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.660Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.736Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.795Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.839Z: JOB_MESSAGE_DETAILED: Fusing consumer decode into ReadFromPubSub/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.875Z: JOB_MESSAGE_DETAILED: Fusing consumer split into decode
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.907Z: JOB_MESSAGE_DETAILED: Fusing consumer pair_with_one into split
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.941Z: JOB_MESSAGE_DETAILED: Fusing consumer WindowInto(WindowIntoFn) into pair_with_one
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.966Z: JOB_MESSAGE_DETAILED: Fusing consumer group/WriteStream into WindowInto(WindowIntoFn)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:14.988Z: JOB_MESSAGE_DETAILED: Fusing consumer group/MergeBuckets into group/ReadStream
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:15.012Z: JOB_MESSAGE_DETAILED: Fusing consumer count into group/MergeBuckets
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:15.047Z: JOB_MESSAGE_DETAILED: Fusing consumer format into count
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:15.091Z: JOB_MESSAGE_DETAILED: Fusing consumer encode into format
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:15.124Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPubSub/ToProtobuf into encode
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:15.155Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPubSub/Write into WriteToPubSub/ToProtobuf
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:15.206Z: JOB_MESSAGE_BASIC: Running job using Streaming Engine
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:15.243Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:15.280Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:15.313Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:15.347Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:16.442Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:16.486Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-b...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:16.535Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:07:42.202Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:08:00.075Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:08:37.570Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-11-24T02:08:51.442Z: JOB_MESSAGE_DETAILED: All workers have finished the startup processes and began to receive work requests.
WARNING apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:254 Timing out on waiting for job 2022-11-23_18_07_06-10773682750582623979 after 360 seconds
=============================== warnings summary ===============================
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
apache_beam/examples/streaming_wordcount_it_test.py::StreamingWordCountIT::test_streaming_wordcount_it
apache_beam/examples/streaming_wordcount_it_test.py::StreamingWordCountIT::test_streaming_wordcount_it
<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/pytest_preCommitIT-df-py310.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/streaming_wordcount_it_test.py::StreamingWordCountIT::test_streaming_wordcount_it - AssertionError:
Expected: (Test pipeline expected terminated in state: RUNNING and Expected 500 messages.)
but: Expected 500 messages. Got 514 messages. Diffs (item, count):
Expected but not in actual: dict_items([(b'218: 1', 1), (b'230: 1', 1), (b'260: 1', 1), (b'264: 1', 1), (b'269: 1', 1), (b'281: 1', 1), (b'294: 1', 1), (b'400: 1', 1), (b'441: 1', 1)])
Unexpected: dict_items([(b'132: 1', 1), (b'476: 1', 1), (b'233: 1', 1), (b'486: 1', 1), (b'434: 1', 1), (b'277: 1', 1), (b'212: 1', 1), (b'472: 1', 1), (b'467: 1', 1), (b'251: 1', 1), (b'475: 1', 1), (b'48: 1', 1), (b'448: 1', 1), (b'271: 1', 1), (b'101: 1', 1), (b'279: 1', 1), (b'485: 1', 1), (b'420: 1', 1), (b'198: 1', 1), (b'216: 1', 1), (b'249: 1', 1), (b'478: 1', 1), (b'422: 1', 1)])
Unexpected (with all details): [(b'132: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 24, 651000, tzinfo=datetime.timezone.utc), ''), (b'476: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'233: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'486: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'132: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 330000, tzinfo=datetime.timezone.utc), ''), (b'434: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'277: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'212: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'472: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'467: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'472: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'251: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'475: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'48: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'448: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'271: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'212: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'467: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'101: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'279: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'485: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'420: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'198: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'476: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'233: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'434: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'277: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'216: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'249: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'478: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'422: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'486: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 23, 969000, tzinfo=datetime.timezone.utc), ''), (b'251: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'475: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'48: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'448: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'271: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'198: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'101: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'279: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'485: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'420: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'216: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'249: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'478: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), ''), (b'422: 1', {}, {}, DatetimeWithNanoseconds(2022, 11, 24, 2, 16, 21, 322000, tzinfo=datetime.timezone.utc), '')]
================== 1 failed, 5 warnings in 779.99s (0:12:59) ===================
> Task :sdks:python:test-suites:dataflow:py310:preCommitIT_streaming_V2 FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 81
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:preCommitIT_streaming_V2'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 20m 21s
129 actionable tasks: 119 executed, 8 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/lflgkourfvzbq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PreCommit_Python_Cron #6403
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/6403/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org