You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2021/06/04 16:00:09 UTC
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #366
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/366/display/redirect?page=changes>
Changes:
[zyichi] [BEAM-9165] clean up python sdk harness status http server.
[noreply] [BEAM-12419] Add Timeout and WaitForReady support in java gRPC calls
[noreply] Minor: Add link to DataFrame API reference in "Languages" nav (#14937)
[rafal.ochyra] [BEAM-12225] List shards using ListShards API instead of DescribeStream
------------------------------------------
[...truncated 57.84 KB...]
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<2,>=1.18.0->apache-beam==2.31.0.dev0) (57.0.0)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.23.0
Using cached google_api_core-1.29.0-py2.py3-none-any.whl (93 kB)
Collecting google-resumable-media<2.0dev,>=0.6.0
Using cached google_resumable_media-1.3.0-py2.py3-none-any.whl (75 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.31.0.dev0) (20.9)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.18.1-py3-none-any.whl (42 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-api-python-client!=1.12.0,!=2.0.2
Using cached google_api_python_client-2.7.0-py2.py3-none-any.whl (7.3 MB)
Collecting google-auth-httplib2
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting uritemplate<4dev,>=3.0.0
Using cached uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.31.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.31.0.dev0) (2.1.1)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.8.0-py3-none-any.whl (48 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.31.0.dev0) (0.13.1)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.31.0.dev0) (1.10.0)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.31.0.dev0) (3.4.1)
Collecting execnet>=1.1
Using cached execnet-1.8.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (160 kB)
Collecting docker
Using cached docker-5.0.0-py2.py3-none-any.whl (146 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.0.1-py2.py3-none-any.whl (68 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.31.0.dev0-py3-none-any.whl size=2475437 sha256=62ed1df0e86526541a291107483ece738d70b40aa6046f881d2200024961f896
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, chardet, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, oauthlib, more-itertools, jmespath, httplib2, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, uritemplate, requests-oauthlib, pytest, numpy, isodate, grpcio-gcp, google-crc32c, google-auth-httplib2, google-api-core, docopt, botocore, apipkg, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, oauth2client, nose, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, google-api-python-client, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, nose-xunitmp, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-pubsub, google-cloud-profiler, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Successfully installed apache-beam-2.31.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.14.0 azure-storage-blob-12.8.1 boto3-1.17.87 botocore-1.20.87 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 crcmod-1.7 cryptography-3.4.7 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.0 docopt-0.6.2 execnet-1.8.1 fastavro-1.4.1 fasteners-0.16.1 freezegun-1.1.0 google-api-core-1.29.0 google-api-python-client-2.7.0 google-apitools-0.5.31 google-auth-1.30.1 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.18.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.6.0 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-profiler-3.0.4 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-1.3.0 googleapis-common-protos-1.53.0 greenlet-1.1.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-2.10 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.8.0 msrest-0.6.21 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 pandas-1.2.4 parameterized-0.7.5 pbr-5.6.0 proto-plus-1.18.1 psycopg2-binary-2.8.6 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.11.4 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.4.2 sqlalchemy-1.4.17 tenacity-5.1.5 testcontainers-3.4.1 typing-extensions-3.7.4.3 uritemplate-3.0.1 urllib3-1.26.5 wcwidth-0.2.5 websocket-client-1.0.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.31.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604150504.1622821841.825922/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604150504.1622821841.825922/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604150504.1622821841.825922/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604150504.1622821841.825922/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604150504.1622821841.825922/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604150504.1622821841.825922/dataflow-****.jar in 8 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604150504.1622821841.825922/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604150504.1622821841.825922/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2021-06-04T15:50:52.420807Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-04_08_50_51-16819794033715233416'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0604150504'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-04T15:50:52.420807Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-06-04_08_50_51-16819794033715233416]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-06-04_08_50_51-16819794033715233416
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-04_08_50_51-16819794033715233416?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-04_08_50_51-16819794033715233416 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:51.273Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:51.273Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:57.825Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:58.591Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:58.639Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:58.728Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:58.768Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:58.806Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:58.842Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:58.945Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.037Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.093Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.138Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.184Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.217Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.240Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.269Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.319Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.353Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.388Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.441Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.472Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.515Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.560Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write/NativeWrite into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.617Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.651Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.696Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.729Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.821Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.869Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:50:59.911Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:51:11.184Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:51:38.821Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:52:15.850Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:52:15.891Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:59:03.436Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:59:03.502Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:59:03.543Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:59:03.583Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:59:03.624Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:59:54.033Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:59:54.076Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-04T15:59:54.132Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-04_08_50_51-16819794033715233416 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 7f4aa13589e7429e85ca322e2f5c7e94 and timestamp: 1622822403.6386173:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 95
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-04_08_50_51-16819794033715233416?project=apache-beam-testing
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1913, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2052, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 820, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2335, in expand
raise ValueError(msg)
ValueError: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 10m 42s
80 actionable tasks: 51 executed, 29 from cache
Publishing build scan...
https://gradle.com/s/lyd7pnk2he4es
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal :
beam_PerformanceTests_PubsubIOIT_Python_Streaming #376
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/376/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #375
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/375/display/redirect?page=changes>
Changes:
[randomstep] [BEAM-12185] Bump org.jenkins-ci.plugins:junit to 1.49
[vitaly.terentyev] [BEAM-8897] Fix reading Rows with Numeric fields from a JDBC data source
[kwu] [BEAM-12454] Add original filesToStage to final pipeline options
[yzhan559] [BEAM-2492] Have PipelineOptions DisplayData filter out attributes
[Robert Bradshaw] [BEAM-9547] Dataframe weighted sample.
[Robert Bradshaw] Add test for nan and 0
[danthev] Fix reporting of throttling metric.
[daniel.o.programmer] [BEAM-12380] Add KafkaIO Transforms and Kafka Taxi example.
[noreply] Update container images used by Dataflow for unreleased SDKs
[Robert Bradshaw] Rename run_test as it doens't actually test anything.
[noreply] [BEAM-9547] Add support for tz_convert, workaround DatetimeIndex hashing
[noreply] [BEAM-9547] Add implementation for first and last (#14992)
[noreply] [BEAM-12334] Re-use java 11 flag in build.gradle (#14892)
[daniel.o.programmer] [BEAM-12380] Go Kafka Fixup, documenting Taxi example, and
------------------------------------------
[...truncated 57.48 KB...]
Collecting fasteners>=0.14
Using cached fasteners-0.16.3-py2.py3-none-any.whl (28 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<2,>=1.18.0->apache-beam==2.32.0.dev0) (57.0.0)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.32.0.dev0) (20.9)
Collecting google-api-core[grpc]<2.0.0dev,>=1.29.0
Using cached google_api_core-1.30.0-py2.py3-none-any.whl (93 kB)
Collecting google-resumable-media<2.0dev,>=0.6.0
Using cached google_resumable_media-1.3.0-py2.py3-none-any.whl (75 kB)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.18.1-py3-none-any.whl (42 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-auth-httplib2
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting google-api-python-client!=1.12.0,!=2.0.2
Using cached google_api_python_client-2.9.0-py2.py3-none-any.whl (7.0 MB)
Collecting uritemplate<4dev,>=3.0.0
Using cached uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.32.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (0.13.1)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (1.10.0)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.8.0-py3-none-any.whl (48 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (2.1.1)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (3.4.1)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (160 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.0-py2.py3-none-any.whl (146 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.1.0-py2.py3-none-any.whl (68 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.32.0.dev0-py3-none-any.whl size=2495788 sha256=26945014288b28eacdcd20fce12b72300f7892f5bc70bcdcbdd7a393210ce9e3
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, chardet, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, oauthlib, more-itertools, jmespath, httplib2, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, uritemplate, requests-oauthlib, pytest, numpy, isodate, grpcio-gcp, google-crc32c, google-auth-httplib2, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, oauth2client, nose, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, google-api-python-client, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, nose-xunitmp, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-pubsub, google-cloud-profiler, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Successfully installed apache-beam-2.32.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.15.0 azure-storage-blob-12.8.1 boto3-1.17.95 botocore-1.20.95 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 crcmod-1.7 cryptography-3.4.7 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.0 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.1 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.30.0 google-api-python-client-2.9.0 google-apitools-0.5.31 google-auth-1.31.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.20.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.0 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-profiler-3.0.4 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-1.3.0 googleapis-common-protos-1.53.0 greenlet-1.1.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-2.10 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.8.0 msrest-0.6.21 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 pandas-1.2.4 parameterized-0.7.5 pbr-5.6.0 proto-plus-1.18.1 psycopg2-binary-2.9 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.11.4 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.4.2 sqlalchemy-1.4.18 tenacity-5.1.5 testcontainers-3.4.1 typing-extensions-3.7.4.3 uritemplate-3.0.1 urllib3-1.26.5 wcwidth-0.2.5 websocket-client-1.1.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.32.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210615
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210615" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0616150620.1623858629.014461/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0616150620.1623858629.014461/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0616150620.1623858629.014461/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0616150620.1623858629.014461/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0616150620.1623858629.014461/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0616150620.1623858629.014461/dataflow-****.jar in 5 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0616150620.1623858629.014461/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0616150620.1623858629.014461/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2021-06-16T15:50:36.510705Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-16_08_50_35-5856310550010815230'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0616150620'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-16T15:50:36.510705Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-06-16_08_50_35-5856310550010815230]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-06-16_08_50_35-5856310550010815230
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_08_50_35-5856310550010815230?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-16_08_50_35-5856310550010815230 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:35.332Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:35.332Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:41.542Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.282Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.326Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.390Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.428Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.500Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.526Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.576Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.633Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.662Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.696Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.735Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.766Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.815Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.850Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.882Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.937Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.968Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:42.993Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:43.014Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:43.046Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:43.084Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write/NativeWrite into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:43.119Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:43.152Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:43.175Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:43.204Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:43.265Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:43.301Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:50:43.341Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:51:14.109Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:51:32.926Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:52:06.079Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:52:06.109Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:59:45.729Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:59:45.783Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:59:45.832Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:59:45.876Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T15:59:45.914Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T16:00:27.004Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T16:00:27.052Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-16T16:00:27.088Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-16_08_50_35-5856310550010815230 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: a217d0b83af14c5eb4914c2ff495702e and timestamp: 1623859237.2432861:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 107
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_08_50_35-5856310550010815230?project=apache-beam-testing
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1913, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2052, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 820, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2335, in expand
raise ValueError(msg)
ValueError: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 11m 21s
80 actionable tasks: 51 executed, 29 from cache
Publishing build scan...
https://gradle.com/s/3ojq5bnnq5t2u
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #374
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/374/display/redirect?page=changes>
Changes:
[noreply] Match the default byref value with the default in dill.dumps()
[noreply] [BEAM-9547] Add support for `Series.cat` methods (#14934)
[noreply] [BEAM-11534] Add transform implementation (#14920)
[noreply] [BEAM-12475] Reject splits for previous bundles. (#14991)
[noreply] [BEAM-9547] argsort is order-sensitive, fixes for loc (#14908)
[noreply] [BEAM-9547] GroupBy: Add support for dtypes, ndim, filter, WontImplement
[noreply] [BEAM-9547] Add support for `Series.value_counts()`, add back the
[je.ik] [BEAM-12473] fix possible ClassCastException with UDAF
[noreply] [BEAM-9547] A few more Series operations (#14972)
------------------------------------------
[...truncated 57.53 KB...]
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<2,>=1.18.0->apache-beam==2.32.0.dev0) (57.0.0)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.18.1-py3-none-any.whl (42 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.32.0.dev0) (20.9)
Collecting google-resumable-media<2.0dev,>=0.6.0
Using cached google_resumable_media-1.3.0-py2.py3-none-any.whl (75 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.29.0
Using cached google_api_core-1.30.0-py2.py3-none-any.whl (93 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-api-python-client!=1.12.0,!=2.0.2
Using cached google_api_python_client-2.9.0-py2.py3-none-any.whl (7.0 MB)
Collecting google-auth-httplib2
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting uritemplate<4dev,>=3.0.0
Using cached uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.32.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.8.0-py3-none-any.whl (48 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (1.10.0)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (2.1.1)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (0.13.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (3.4.1)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (160 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-5.0.0-py2.py3-none-any.whl (146 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.1.0-py2.py3-none-any.whl (68 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.32.0.dev0-py3-none-any.whl size=2495008 sha256=2012b39e9f5029aa754c2a22225647f1b204bf4d76dcbdedeb1fb225987a0b9a
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, chardet, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, oauthlib, more-itertools, jmespath, httplib2, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, uritemplate, requests-oauthlib, pytest, numpy, isodate, grpcio-gcp, google-crc32c, google-auth-httplib2, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, oauth2client, nose, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, google-api-python-client, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, nose-xunitmp, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-pubsub, google-cloud-profiler, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Successfully installed apache-beam-2.32.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.15.0 azure-storage-blob-12.8.1 boto3-1.17.94 botocore-1.20.94 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 crcmod-1.7 cryptography-3.4.7 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.0 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.1 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.30.0 google-api-python-client-2.9.0 google-apitools-0.5.31 google-auth-1.31.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.20.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.0 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-profiler-3.0.4 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-1.3.0 googleapis-common-protos-1.53.0 greenlet-1.1.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-2.10 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.8.0 msrest-0.6.21 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 pandas-1.2.4 parameterized-0.7.5 pbr-5.6.0 proto-plus-1.18.1 psycopg2-binary-2.8.6 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.11.4 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.4.2 sqlalchemy-1.4.18 tenacity-5.1.5 testcontainers-3.4.1 typing-extensions-3.7.4.3 uritemplate-3.0.1 urllib3-1.26.5 wcwidth-0.2.5 websocket-client-1.1.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.32.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0615150450.1623772230.901167/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0615150450.1623772230.901167/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0615150450.1623772230.901167/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0615150450.1623772230.901167/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0615150450.1623772230.901167/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0615150450.1623772230.901167/dataflow-****.jar in 4 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0615150450.1623772230.901167/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0615150450.1623772230.901167/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2021-06-15T15:50:36.405416Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-15_08_50_35-742927289019717210'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0615150450'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-15T15:50:36.405416Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-06-15_08_50_35-742927289019717210]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-06-15_08_50_35-742927289019717210
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-15_08_50_35-742927289019717210?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-15_08_50_35-742927289019717210 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:35.449Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:35.449Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:42.205Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:42.826Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:42.854Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:42.919Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:42.957Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:42.994Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.030Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.096Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.152Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.196Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.228Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.255Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.278Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.311Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.341Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.366Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.388Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.419Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.464Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.496Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.528Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.563Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write/NativeWrite into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.607Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.654Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.684Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.717Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.772Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.804Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:50:43.829Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:51:07.436Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:51:18.483Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 2 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:51:18.516Z: JOB_MESSAGE_DETAILED: Resized **** pool to 2, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:51:28.733Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:52:02.554Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:52:02.585Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:59:47.289Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:59:47.332Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:59:47.361Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:59:47.385Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T15:59:47.419Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T16:00:44.874Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T16:00:44.914Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-15T16:00:44.940Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-06-15_08_50_35-742927289019717210 after 604 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: f1b876a46ef04807ab24aba9910f5699 and timestamp: 1623772852.3019667:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 91
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-15_08_50_35-742927289019717210?project=apache-beam-testing
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1913, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2052, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 820, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2335, in expand
raise ValueError(msg)
ValueError: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 11m 37s
80 actionable tasks: 51 executed, 29 from cache
Publishing build scan...
https://gradle.com/s/ns2wsc4tszpco
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #373
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/373/display/redirect>
Changes:
------------------------------------------
[...truncated 57.61 KB...]
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<2,>=1.18.0->apache-beam==2.32.0.dev0) (57.0.0)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting google-resumable-media<2.0dev,>=0.6.0
Using cached google_resumable_media-1.3.0-py2.py3-none-any.whl (75 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.32.0.dev0) (20.9)
Collecting google-api-core[grpc]<2.0.0dev,>=1.29.0
Using cached google_api_core-1.30.0-py2.py3-none-any.whl (93 kB)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.18.1-py3-none-any.whl (42 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-auth-httplib2
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting google-api-python-client!=1.12.0,!=2.0.2
Downloading google_api_python_client-2.9.0-py2.py3-none-any.whl (7.0 MB)
Collecting uritemplate<4dev,>=3.0.0
Using cached uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.32.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (2.1.1)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.8.0-py3-none-any.whl (48 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (1.10.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (0.13.1)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (3.4.1)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (160 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-5.0.0-py2.py3-none-any.whl (146 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.1.0-py2.py3-none-any.whl (68 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.32.0.dev0-py3-none-any.whl size=2492156 sha256=719f7f8aace38063d52c514710f434e4089fa21e44836861208d4a9545c4359e
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, chardet, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, oauthlib, more-itertools, jmespath, httplib2, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, uritemplate, requests-oauthlib, pytest, numpy, isodate, grpcio-gcp, google-crc32c, google-auth-httplib2, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, oauth2client, nose, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, google-api-python-client, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, nose-xunitmp, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-pubsub, google-cloud-profiler, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Successfully installed apache-beam-2.32.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.15.0 azure-storage-blob-12.8.1 boto3-1.17.93 botocore-1.20.93 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 crcmod-1.7 cryptography-3.4.7 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.0 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.1 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.30.0 google-api-python-client-2.9.0 google-apitools-0.5.31 google-auth-1.31.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.20.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.6.0 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-profiler-3.0.4 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-1.3.0 googleapis-common-protos-1.53.0 greenlet-1.1.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-2.10 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.8.0 msrest-0.6.21 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 pandas-1.2.4 parameterized-0.7.5 pbr-5.6.0 proto-plus-1.18.1 psycopg2-binary-2.8.6 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.11.4 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.4.2 sqlalchemy-1.4.18 tenacity-5.1.5 testcontainers-3.4.1 typing-extensions-3.7.4.3 uritemplate-3.0.1 urllib3-1.26.5 wcwidth-0.2.5 websocket-client-1.1.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.32.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0614150446.1623685834.542081/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0614150446.1623685834.542081/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0614150446.1623685834.542081/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0614150446.1623685834.542081/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0614150446.1623685834.542081/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0614150446.1623685834.542081/dataflow-****.jar in 4 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0614150446.1623685834.542081/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0614150446.1623685834.542081/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2021-06-14T15:50:40.716659Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-14_08_50_39-12243399310884444801'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0614150446'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-14T15:50:40.716659Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-06-14_08_50_39-12243399310884444801]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-06-14_08_50_39-12243399310884444801
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-14_08_50_39-12243399310884444801?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-14_08_50_39-12243399310884444801 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:39.667Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:39.667Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:44.773Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.368Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.391Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.454Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.489Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.520Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.549Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.595Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.644Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.700Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.732Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.762Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.798Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.821Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.845Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.886Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.921Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.951Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:45.976Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:46.008Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:46.041Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:46.071Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write/NativeWrite into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:46.123Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:46.163Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:46.192Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:46.212Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:46.267Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:46.302Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:50:46.341Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:51:18.536Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:51:25.115Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 1 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:51:25.151Z: JOB_MESSAGE_DETAILED: Resized **** pool to 1, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:51:35.432Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:52:11.398Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:52:11.428Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:59:51.227Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:59:51.279Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:59:51.305Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:59:51.344Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T15:59:51.369Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T16:00:42.161Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T16:00:42.239Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-14T16:00:42.274Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-14_08_50_39-12243399310884444801 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 674943a510c5407e81a515e107a6dd9f and timestamp: 1623686451.6685672:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 101
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-14_08_50_39-12243399310884444801?project=apache-beam-testing
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1913, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2052, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 820, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2335, in expand
raise ValueError(msg)
ValueError: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 11m 37s
80 actionable tasks: 51 executed, 29 from cache
Publishing build scan...
https://gradle.com/s/uibq2xiveddyw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #372
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/372/display/redirect>
Changes:
------------------------------------------
[...truncated 57.56 KB...]
Using cached fasteners-0.16.2-py2.py3-none-any.whl (37 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<2,>=1.18.0->apache-beam==2.32.0.dev0) (57.0.0)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.18.1-py3-none-any.whl (42 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.29.0
Using cached google_api_core-1.30.0-py2.py3-none-any.whl (93 kB)
Collecting google-resumable-media<2.0dev,>=0.6.0
Using cached google_resumable_media-1.3.0-py2.py3-none-any.whl (75 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.32.0.dev0) (20.9)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-api-python-client!=1.12.0,!=2.0.2
Using cached google_api_python_client-2.8.0-py2.py3-none-any.whl (7.0 MB)
Collecting google-auth-httplib2
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting uritemplate<4dev,>=3.0.0
Using cached uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.32.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (1.10.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (0.13.1)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.8.0-py3-none-any.whl (48 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (2.1.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (3.4.1)
Collecting execnet>=1.1
Downloading execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (160 kB)
Collecting docker
Using cached docker-5.0.0-py2.py3-none-any.whl (146 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.1.0-py2.py3-none-any.whl (68 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.32.0.dev0-py3-none-any.whl size=2492156 sha256=46e0bf7aaf3079368fdb48ecb2dc6fc5129eead467bbb42da047989fa58f8bb3
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, chardet, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, oauthlib, more-itertools, jmespath, httplib2, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, uritemplate, requests-oauthlib, pytest, numpy, isodate, grpcio-gcp, google-crc32c, google-auth-httplib2, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, oauth2client, nose, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, google-api-python-client, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, nose-xunitmp, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-pubsub, google-cloud-profiler, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Successfully installed apache-beam-2.32.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.15.0 azure-storage-blob-12.8.1 boto3-1.17.93 botocore-1.20.93 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 crcmod-1.7 cryptography-3.4.7 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.0 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.1 fasteners-0.16.2 freezegun-1.1.0 google-api-core-1.30.0 google-api-python-client-2.8.0 google-apitools-0.5.31 google-auth-1.31.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.20.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.6.0 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-profiler-3.0.4 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-1.3.0 googleapis-common-protos-1.53.0 greenlet-1.1.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-2.10 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.8.0 msrest-0.6.21 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 pandas-1.2.4 parameterized-0.7.5 pbr-5.6.0 proto-plus-1.18.1 psycopg2-binary-2.8.6 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.11.4 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.4.2 sqlalchemy-1.4.18 tenacity-5.1.5 testcontainers-3.4.1 typing-extensions-3.7.4.3 uritemplate-3.0.1 urllib3-1.26.5 wcwidth-0.2.5 websocket-client-1.1.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.32.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0613150452.1623599437.156420/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0613150452.1623599437.156420/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0613150452.1623599437.156420/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0613150452.1623599437.156420/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0613150452.1623599437.156420/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0613150452.1623599437.156420/dataflow-****.jar in 4 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0613150452.1623599437.156420/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0613150452.1623599437.156420/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2021-06-13T15:50:43.084881Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-13_08_50_42-15840043376754710907'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0613150452'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-13T15:50:43.084881Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-06-13_08_50_42-15840043376754710907]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-06-13_08_50_42-15840043376754710907
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-13_08_50_42-15840043376754710907?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-13_08_50_42-15840043376754710907 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:42.130Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:42.130Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:46.895Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:47.652Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:47.704Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:47.763Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:47.787Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:47.821Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:47.850Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:47.902Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:47.951Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:47.995Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.032Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.063Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.101Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.123Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.147Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.186Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.220Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.241Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.266Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.300Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.330Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.373Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write/NativeWrite into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.473Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.507Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.541Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.573Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.631Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.670Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:50:48.699Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:51:04.535Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:51:28.853Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:52:09.650Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:52:09.695Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:58:35.031Z: JOB_MESSAGE_ERROR: There were errors attempting to publish to topic projects/apache-beam-testing/topics/pubsub_io_performance_4eebca92-198b-42e4-8227-efbb404e560e. Recent publish calls completed successfully: 1946, recent publish calls completed with error: 1.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:59:51.250Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:59:51.297Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:59:51.324Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:59:51.365Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T15:59:51.397Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T16:00:43.014Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T16:00:43.095Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-13T16:00:43.134Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-13_08_50_42-15840043376754710907 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 2d4da1e71037421a9c53f33aed60ba6b and timestamp: 1623600054.0424166:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 83
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-13_08_50_42-15840043376754710907?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1913, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2052, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 820, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2335, in expand
raise ValueError(msg)
ValueError: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 11m 40s
80 actionable tasks: 51 executed, 29 from cache
Publishing build scan...
https://gradle.com/s/bt5jb3rk47syw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #371
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/371/display/redirect?page=changes>
Changes:
[heejong] bugfix in verify_release_build.sh
[heejong] fix formatting error in mass_comment.py
[heejong] use set_version.sh
[rogelio.hernandez] [BEAM-11984] Request count metrics in GcsDownloader
[rogelio.hernandez] [BEAM-11984] Request count metrics in GcsUploader
[rogelio.hernandez] [BEAM-11984] Added monitoring labels to proto file
[rogelio.hernandez] [BEAM-11984] Added projectId to metric labels
[BenWhitehead] [BEAM-8376] Google Cloud Firestore Connector - Add firestore libraries
[BenWhitehead] [BEAM-8376] Google Cloud Firestore Connector - Add common classes
[BenWhitehead] [BEAM-8376] Google Cloud Firestore Connector - Add RPC QoS Classes
[rogelio.hernandez] [BEAM-11984] Added project number to metric labels
[rogelio.hernandez] [BEAM-11984] Changed resource identifier name
[BenWhitehead] [BEAM-8376] Google Cloud Firestore Connector - Add Firestore v1 Write
[kwu] [BEAM-12419] Deadline should be reset for each gRPC call.
[noreply] [BEAM-11086] Validate and Fix Go SDK windowing. (#14966)
[noreply] Missing space in warning message.
[heejong] use --git-add
[noreply] [BEAM-12028] Add contact information to NotImplementedError (#14967)
[noreply] [BEAM-12272] Python - Backport Firestore connector's ramp-up throttling
------------------------------------------
[...truncated 57.61 KB...]
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<2,>=1.18.0->apache-beam==2.32.0.dev0) (57.0.0)
Collecting google-resumable-media<2.0dev,>=0.6.0
Using cached google_resumable_media-1.3.0-py2.py3-none-any.whl (75 kB)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.18.1-py3-none-any.whl (42 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.32.0.dev0) (20.9)
Collecting google-api-core[grpc]<2.0.0dev,>=1.29.0
Using cached google_api_core-1.30.0-py2.py3-none-any.whl (93 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-api-python-client!=1.12.0,!=2.0.2
Using cached google_api_python_client-2.8.0-py2.py3-none-any.whl (7.0 MB)
Collecting google-auth-httplib2
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting uritemplate<4dev,>=3.0.0
Using cached uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.32.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (2.1.1)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (0.13.1)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.8.0-py3-none-any.whl (48 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (1.10.0)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (3.4.1)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.8.1-py2.py3-none-any.whl (39 kB)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (160 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.0-py2.py3-none-any.whl (146 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.1.0-py2.py3-none-any.whl (68 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.32.0.dev0-py3-none-any.whl size=2492156 sha256=271cfc71e4157e2a7f85655464db72b4910550bbd0d001e37e2800d9bfbbe142
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, chardet, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, oauthlib, more-itertools, jmespath, httplib2, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, uritemplate, requests-oauthlib, pytest, numpy, isodate, grpcio-gcp, google-crc32c, google-auth-httplib2, google-api-core, docopt, botocore, apipkg, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, oauth2client, nose, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, google-api-python-client, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, nose-xunitmp, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-pubsub, google-cloud-profiler, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Successfully installed apache-beam-2.32.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.15.0 azure-storage-blob-12.8.1 boto3-1.17.93 botocore-1.20.93 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 crcmod-1.7 cryptography-3.4.7 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.0 docopt-0.6.2 execnet-1.8.1 fastavro-1.4.1 fasteners-0.16.2 freezegun-1.1.0 google-api-core-1.30.0 google-api-python-client-2.8.0 google-apitools-0.5.31 google-auth-1.31.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.20.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.6.0 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-profiler-3.0.4 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-1.3.0 googleapis-common-protos-1.53.0 greenlet-1.1.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-2.10 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.8.0 msrest-0.6.21 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 pandas-1.2.4 parameterized-0.7.5 pbr-5.6.0 proto-plus-1.18.1 psycopg2-binary-2.8.6 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.11.4 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.4.2 sqlalchemy-1.4.18 tenacity-5.1.5 testcontainers-3.4.1 typing-extensions-3.7.4.3 uritemplate-3.0.1 urllib3-1.26.5 wcwidth-0.2.5 websocket-client-1.1.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.32.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0612150449.1623513028.990154/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0612150449.1623513028.990154/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0612150449.1623513028.990154/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0612150449.1623513028.990154/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0612150449.1623513028.990154/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0612150449.1623513028.990154/dataflow-****.jar in 4 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0612150449.1623513028.990154/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0612150449.1623513028.990154/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2021-06-12T15:50:34.975299Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-12_08_50_33-2573639991091598467'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0612150449'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-12T15:50:34.975299Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-06-12_08_50_33-2573639991091598467]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-06-12_08_50_33-2573639991091598467
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-12_08_50_33-2573639991091598467?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-12_08_50_33-2573639991091598467 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:34.011Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:34.011Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:38.838Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:39.551Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:39.584Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:39.650Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:39.676Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:39.712Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:39.768Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:39.834Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:39.901Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:39.942Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:39.969Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:39.994Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.027Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.057Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.090Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.123Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.146Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.179Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.200Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.233Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.269Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.302Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write/NativeWrite into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.341Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.373Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.414Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.453Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.505Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.531Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:40.566Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:50:55.385Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:51:25.491Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:52:00.851Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:52:00.881Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:59:43.132Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:59:43.192Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:59:43.226Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:59:43.303Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T15:59:43.328Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T16:00:46.397Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T16:00:46.438Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-12T16:00:46.476Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-06-12_08_50_33-2573639991091598467 after 603 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 191f00236a1c4cf5964b4490fcf061eb and timestamp: 1623513656.0901577:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 93
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-12_08_50_33-2573639991091598467?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1913, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2052, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 820, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2335, in expand
raise ValueError(msg)
ValueError: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 11m 42s
80 actionable tasks: 51 executed, 29 from cache
Publishing build scan...
https://gradle.com/s/zay6gjsg36ta6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #370
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/370/display/redirect?page=changes>
Changes:
[noreply] Warn on BigQuery Job failures
[noreply] Fix spelling error when starting BigQuery jobs
[Robert Bradshaw] Improve basic explanation of Beam PTransforms.
[Luke Cwik] [BEAM-12459] Ensure that we use the min element timestamp for Watch
[dmytrokozhevin] Add a setting to enable dill byref pickling in Beam pickler.
[dmytrokozhevin] Add a setting to enable dill byref pickling in Beam pickler.
[noreply] [BEAM-12442] Make sure that row renames work properly in nested rows:
[noreply] Fix Doc for DatasetReference in ReadFromBigQuery (#14818)
[noreply] [BEAM-9547] Add initial implementation of sample (#14984)
[noreply] [BEAM-12476] Add specific jira for sample follow-up work (#14994)
[noreply] [BEAM-12444] Fix bug when groupby.apply is used on a series grouped by a
------------------------------------------
[...truncated 57.62 KB...]
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<2,>=1.18.0->apache-beam==2.32.0.dev0) (57.0.0)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.32.0.dev0) (20.9)
Collecting google-resumable-media<2.0dev,>=0.6.0
Using cached google_resumable_media-1.3.0-py2.py3-none-any.whl (75 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.29.0
Using cached google_api_core-1.30.0-py2.py3-none-any.whl (93 kB)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.18.1-py3-none-any.whl (42 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-api-python-client!=1.12.0,!=2.0.2
Using cached google_api_python_client-2.8.0-py2.py3-none-any.whl (7.0 MB)
Collecting google-auth-httplib2
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting uritemplate<4dev,>=3.0.0
Using cached uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.32.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (0.13.1)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.8.0-py3-none-any.whl (48 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (2.1.1)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (1.10.0)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (3.4.1)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.8.1-py2.py3-none-any.whl (39 kB)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (160 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-5.0.0-py2.py3-none-any.whl (146 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.1.0-py2.py3-none-any.whl (68 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.32.0.dev0-py3-none-any.whl size=2487550 sha256=dc2fdf192004fbec1a76fac797c60ac039e160f665865a4b37d3d027fbe0efd7
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, chardet, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, oauthlib, more-itertools, jmespath, httplib2, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, uritemplate, requests-oauthlib, pytest, numpy, isodate, grpcio-gcp, google-crc32c, google-auth-httplib2, google-api-core, docopt, botocore, apipkg, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, oauth2client, nose, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, google-api-python-client, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, nose-xunitmp, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-pubsub, google-cloud-profiler, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Successfully installed apache-beam-2.32.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.15.0 azure-storage-blob-12.8.1 boto3-1.17.92 botocore-1.20.92 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 crcmod-1.7 cryptography-3.4.7 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.0 docopt-0.6.2 execnet-1.8.1 fastavro-1.4.1 fasteners-0.16.2 freezegun-1.1.0 google-api-core-1.30.0 google-api-python-client-2.8.0 google-apitools-0.5.31 google-auth-1.31.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.20.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.6.0 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-profiler-3.0.4 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-1.3.0 googleapis-common-protos-1.53.0 greenlet-1.1.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-2.10 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.8.0 msrest-0.6.21 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 pandas-1.2.4 parameterized-0.7.5 pbr-5.6.0 proto-plus-1.18.1 psycopg2-binary-2.8.6 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.11.4 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.4.2 sqlalchemy-1.4.18 tenacity-5.1.5 testcontainers-3.4.1 typing-extensions-3.7.4.3 uritemplate-3.0.1 urllib3-1.26.5 wcwidth-0.2.5 websocket-client-1.1.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.32.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0611150450.1623426626.601645/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0611150450.1623426626.601645/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0611150450.1623426626.601645/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0611150450.1623426626.601645/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0611150450.1623426626.601645/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0611150450.1623426626.601645/dataflow-****.jar in 4 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0611150450.1623426626.601645/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0611150450.1623426626.601645/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2021-06-11T15:50:33.183957Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-11_08_50_32-10452894022949021038'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0611150450'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-11T15:50:33.183957Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-06-11_08_50_32-10452894022949021038]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-06-11_08_50_32-10452894022949021038
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-11_08_50_32-10452894022949021038?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-11_08_50_32-10452894022949021038 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:32.097Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:32.097Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:37.486Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.161Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.200Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.260Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.295Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.334Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.370Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.459Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.523Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.569Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.604Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.627Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.650Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.685Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.730Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.763Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.805Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.840Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.873Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.896Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.928Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:38.962Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write/NativeWrite into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:39.012Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:39.041Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:39.063Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:39.096Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:39.159Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:39.194Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:39.246Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:50:49.389Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:51:22.737Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:51:56.324Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:51:56.360Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:59:41.719Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:59:41.766Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:59:41.787Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:59:41.819Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T15:59:41.843Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T16:00:42.220Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T16:00:42.261Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-11T16:00:42.295Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-06-11_08_50_32-10452894022949021038 after 602 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 9536228a9f0345f98b422d7cf92d654e and timestamp: 1623427249.4984293:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 103
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-11_08_50_32-10452894022949021038?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1913, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2052, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 820, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2335, in expand
raise ValueError(msg)
ValueError: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 11m 33s
80 actionable tasks: 51 executed, 29 from cache
Publishing build scan...
https://gradle.com/s/bbagomwc67fuc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #369
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/369/display/redirect?page=changes>
Changes:
[noreply] Update tensorflow to the latest version
[noreply] Update grpcio
[Brian Hulette] Use functools.wraps in @progress_indicator
[noreply] Add license info for keras-nightly package.
[anant.damle] [BEAM-12460] Provide a simpler interface to convert Beam Row to
[Udi Meiri] [BEAM-12465] Fix nested subscripted Generics
[odidev] Add linux aarch64 wheel build support
[noreply] Merge pull request #14949: [BEAM-12356] Cache and shutdown BigQuery
[noreply] [BEAM-12379] Verify proxies in frames_test.py, and address some proxy
[noreply] Minor: Update link to wordcount pipeline, link to all examples (#14973)
[noreply] Add a blog post on how to perform release validations (#13724)
[noreply] Merge pull request #11296 from [BEAM-9640] Sketching watermark tracking
[Udi Meiri] [BEAM-12469] Fix _unified_repr to not expect __name__ to exist.
[heejong] Publish blog article for 2.30.0 release
[heejong] Publish 2.30.0 release on Beam website
[zyichi] [BEAM-12470] Increase input size of ReshuffleTest.testAssignShardFn to
[noreply] [BEAM-8137] Add Main method to ExternalWorkerService (#14942)
[pascal.gillet] [BEAM-12471] Fixes NumberFormatException
------------------------------------------
[...truncated 58.49 KB...]
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :runners:core-construction-java:classes
> Task :runners:core-construction-java:jar
> Task :runners:core-java:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :runners:core-java:classes
> Task :runners:core-java:jar
> Task :sdks:java:extensions:google-cloud-platform-core:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
> Task :sdks:java:extensions:google-cloud-platform-core:classes
> Task :sdks:java:extensions:google-cloud-platform-core:jar
> Task :sdks:java:harness:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :sdks:java:harness:classes
> Task :sdks:java:harness:jar
> Task :sdks:java:harness:shadowJar
> Task :runners:java-fn-execution:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :runners:java-fn-execution:classes
> Task :runners:java-fn-execution:jar
> Task :sdks:java:expansion-service:compileJava
Note: <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java> uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :sdks:java:expansion-service:classes
> Task :sdks:java:expansion-service:jar
> Task :sdks:java:io:kafka:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :sdks:java:io:kafka:classes
> Task :sdks:java:io:kafka:jar
> Task :sdks:java:io:google-cloud-platform:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :sdks:java:io:google-cloud-platform:classes
> Task :sdks:java:io:google-cloud-platform:jar
> Task :runners:google-cloud-dataflow-java:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :runners:google-cloud-dataflow-java:classes
> Task :runners:google-cloud-dataflow-java:jar
> Task :runners:google-cloud-dataflow-java:****:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :runners:google-cloud-dataflow-java:****:classes
> Task :runners:google-cloud-dataflow-java:****:shadowJar
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.32.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0610150633.1623340875.101927/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0610150633.1623340875.101927/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0610150633.1623340875.101927/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0610150633.1623340875.101927/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0610150633.1623340875.101927/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0610150633.1623340875.101927/dataflow-****.jar in 6 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0610150633.1623340875.101927/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0610150633.1623340875.101927/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2021-06-10T16:01:24.153328Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-10_09_01_23-8587661447571009518'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0610150633'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-10T16:01:24.153328Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-06-10_09_01_23-8587661447571009518]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-06-10_09_01_23-8587661447571009518
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-10_09_01_23-8587661447571009518?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-10_09_01_23-8587661447571009518 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:23.121Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:23.121Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:28.547Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:29.439Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:29.525Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:29.672Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:29.717Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:29.760Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:29.792Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:29.895Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:29.952Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:29.998Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.035Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.076Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.118Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.152Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.200Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.241Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.335Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.420Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.474Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.588Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.641Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.695Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write/NativeWrite into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.811Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.879Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:30.992Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:31.044Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:31.126Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:31.169Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:31.198Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:01:43.566Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:02:15.263Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:02:15.296Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:02:50.405Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:02:50.445Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:04:08.384Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:11:33.566Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:11:33.713Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:11:33.759Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:11:33.794Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-10T16:11:33.828Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-06-10_09_01_23-8587661447571009518 after 604 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 10e5f90ef7054db5b02f6cf4cfd870ac and timestamp: 1623341560.7133982:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 173
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-10_09_01_23-8587661447571009518?project=apache-beam-testing
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1913, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2052, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 820, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2335, in expand
raise ValueError(msg)
ValueError: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 23m 23s
80 actionable tasks: 67 executed, 13 from cache
Publishing build scan...
https://gradle.com/s/nkirlp6h4js4a
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #368
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/368/display/redirect?page=changes>
Changes:
[suztomo] [BEAM-11205] Added libraries-bom version to CHANGES.md
[zhoufek] [BEAM-11811] Disallow num_workers > max_num_workers for Go Dataflow
[noreply] [BEAM-12074] Add special explanation for head/tail pointing to
[noreply] [BEAM-11811] Disallow num_workers > max_num_workers for Python
[noreply] Minor: Remove pandas 0.x information in dataframe schema notes (#14965)
[noreply] [BEAM-12074] Define docstrings for all DataFrame operations (#14876)
[Kyle Weaver] [BEAM-12439] Reuse Java job servers in spark_runner.py.
------------------------------------------
[...truncated 57.30 KB...]
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<2,>=1.18.0->apache-beam==2.32.0.dev0) (57.0.0)
Collecting google-api-core[grpc]<2.0.0dev,>=1.29.0
Using cached google_api_core-1.30.0-py2.py3-none-any.whl (93 kB)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.18.1-py3-none-any.whl (42 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.32.0.dev0) (20.9)
Collecting google-resumable-media<2.0dev,>=0.6.0
Using cached google_resumable_media-1.3.0-py2.py3-none-any.whl (75 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-api-python-client!=1.12.0,!=2.0.2
Using cached google_api_python_client-2.8.0-py2.py3-none-any.whl (7.0 MB)
Collecting google-auth-httplib2
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting uritemplate<4dev,>=3.0.0
Using cached uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.32.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (0.13.1)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (1.10.0)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (2.1.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.8.0-py3-none-any.whl (48 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (3.4.1)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.8.1-py2.py3-none-any.whl (39 kB)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (160 kB)
Collecting docker
Using cached docker-5.0.0-py2.py3-none-any.whl (146 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.0.1-py2.py3-none-any.whl (68 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.32.0.dev0-py3-none-any.whl size=2479764 sha256=39a86d257fed100bca38c419944c394130ddb3a11ca0608a8178cc3940b4d250
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, chardet, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, oauthlib, more-itertools, jmespath, httplib2, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, uritemplate, requests-oauthlib, pytest, numpy, isodate, grpcio-gcp, google-crc32c, google-auth-httplib2, google-api-core, docopt, botocore, apipkg, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, oauth2client, nose, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, google-api-python-client, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, nose-xunitmp, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-pubsub, google-cloud-profiler, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Successfully installed apache-beam-2.32.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.15.0 azure-storage-blob-12.8.1 boto3-1.17.90 botocore-1.20.90 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 crcmod-1.7 cryptography-3.4.7 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.0 docopt-0.6.2 execnet-1.8.1 fastavro-1.4.1 fasteners-0.16.2 freezegun-1.1.0 google-api-core-1.30.0 google-api-python-client-2.8.0 google-apitools-0.5.31 google-auth-1.30.2 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.20.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.6.0 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-profiler-3.0.4 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-1.3.0 googleapis-common-protos-1.53.0 greenlet-1.1.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-2.10 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.8.0 msrest-0.6.21 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 pandas-1.2.4 parameterized-0.7.5 pbr-5.6.0 proto-plus-1.18.1 psycopg2-binary-2.8.6 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.11.4 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.4.2 sqlalchemy-1.4.17 tenacity-5.1.5 testcontainers-3.4.1 typing-extensions-3.7.4.3 uritemplate-3.0.1 urllib3-1.26.5 wcwidth-0.2.5 websocket-client-1.0.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.32.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0609150629.1623253840.972711/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0609150629.1623253840.972711/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0609150629.1623253840.972711/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0609150629.1623253840.972711/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0609150629.1623253840.972711/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0609150629.1623253840.972711/dataflow-****.jar in 6 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0609150629.1623253840.972711/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0609150629.1623253840.972711/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2021-06-09T15:50:49.853934Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-09_08_50_47-4015283327665340438'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0609150629'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-09T15:50:49.853934Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-06-09_08_50_47-4015283327665340438]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-06-09_08_50_47-4015283327665340438
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-09_08_50_47-4015283327665340438?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-09_08_50_47-4015283327665340438 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:47.984Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:47.984Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:53.924Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:54.717Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:54.740Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:54.793Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:54.829Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:54.865Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:54.885Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:54.949Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.016Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.054Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.084Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.115Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.149Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.192Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.225Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.258Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.294Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.327Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.348Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.382Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.416Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.440Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write/NativeWrite into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.474Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.502Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.551Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.586Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.638Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.671Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:50:55.705Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:51:32.502Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:51:41.231Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:52:20.155Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:52:20.217Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:58:58.992Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:58:59.042Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:58:59.071Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:58:59.113Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:58:59.140Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:59:57.766Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:59:57.807Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-09T15:59:57.853Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-09_08_50_47-4015283327665340438 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 38e73fa46b604868b7ddf3eaf6191346 and timestamp: 1623254406.4020138:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 80
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-09_08_50_47-4015283327665340438?project=apache-beam-testing
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1913, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2052, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 820, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2335, in expand
raise ValueError(msg)
ValueError: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 10m 50s
80 actionable tasks: 51 executed, 29 from cache
Publishing build scan...
https://gradle.com/s/f7dz7xxexfdv2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #367
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/367/display/redirect?page=changes>
Changes:
[aromanenko.dev] [WEBSITE] Add a note about how to unsubscribe from mailing lists
[noreply] [BEAM-12434] Implement side_input for num_shards in iobase (#14916)
[noreply] [BEAM-12410] SamzaPortablePipelineOptions to extend
[noreply] [BEAM-12438] Add Regression test for issue around LP coding Row coders.
[noreply] [BEAM-9547] Thorough testing for global aggregations (#14893)
[noreply] [BEAM-9547] Add support for Series.repeat (#14909)
[noreply] [BEAM-9547] Add support for `DataFrame.value_counts()` (#14922)
[Andrew Pilloud] Moving to 2.32.0-SNAPSHOT on master branch.
[Ismaël Mejía] [BEAM-12385] Handle VARCHAR and other SQL specific logical types in
[Andrew Pilloud] Add 2.32.0 section to CHANGES.md
[noreply] [BEAM-12305] Fix test_pack_combiner for Flink and Spark (#14842)
[noreply] Cython 3.0 readiness (#14848)
[noreply] [BEAM-9547] More WontImplement operations (#14910)
------------------------------------------
[...truncated 60.44 KB...]
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<2,>=1.18.0->apache-beam==2.32.0.dev0) (57.0.0)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.18.1-py3-none-any.whl (42 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.29.0
Using cached google_api_core-1.30.0-py2.py3-none-any.whl (93 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.32.0.dev0) (20.9)
Collecting google-resumable-media<2.0dev,>=0.6.0
Using cached google_resumable_media-1.3.0-py2.py3-none-any.whl (75 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-api-python-client!=1.12.0,!=2.0.2
Using cached google_api_python_client-2.8.0-py2.py3-none-any.whl (7.0 MB)
Collecting google-auth-httplib2
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting uritemplate<4dev,>=3.0.0
Using cached uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.32.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (1.10.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (0.13.1)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (2.1.1)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.8.0-py3-none-any.whl (48 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.32.0.dev0) (3.4.1)
Collecting execnet>=1.1
Using cached execnet-1.8.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (160 kB)
Collecting docker
Using cached docker-5.0.0-py2.py3-none-any.whl (146 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.0.1-py2.py3-none-any.whl (68 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.32.0.dev0-py3-none-any.whl size=2476743 sha256=a9ca05965ee0de993bb18eb9bf44df25da82584c3b56847c0d7915ae90dcb675
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, chardet, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, oauthlib, more-itertools, jmespath, httplib2, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, uritemplate, requests-oauthlib, pytest, numpy, isodate, grpcio-gcp, google-crc32c, google-auth-httplib2, google-api-core, docopt, botocore, apipkg, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, oauth2client, nose, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, google-api-python-client, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, nose-xunitmp, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-pubsub, google-cloud-profiler, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Successfully installed apache-beam-2.32.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.15.0 azure-storage-blob-12.8.1 boto3-1.17.89 botocore-1.20.89 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 crcmod-1.7 cryptography-3.4.7 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.0 docopt-0.6.2 execnet-1.8.1 fastavro-1.4.1 fasteners-0.16.2 freezegun-1.1.0 google-api-core-1.30.0 google-api-python-client-2.8.0 google-apitools-0.5.31 google-auth-1.30.2 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.20.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.6.0 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-profiler-3.0.4 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-1.3.0 googleapis-common-protos-1.53.0 greenlet-1.1.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-2.10 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.8.0 msrest-0.6.21 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 pandas-1.2.4 parameterized-0.7.5 pbr-5.6.0 proto-plus-1.18.1 psycopg2-binary-2.8.6 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.11.4 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.4.2 sqlalchemy-1.4.17 tenacity-5.1.5 testcontainers-3.4.1 typing-extensions-3.7.4.3 uritemplate-3.0.1 urllib3-1.26.5 wcwidth-0.2.5 websocket-client-1.0.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.32.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210526" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604200505.1623171838.750943/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604200505.1623171838.750943/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604200505.1623171838.750943/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604200505.1623171838.750943/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604200505.1623171838.750943/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604200505.1623171838.750943/dataflow-****.jar in 7 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604200505.1623171838.750943/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0604200505.1623171838.750943/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2021-06-08T17:04:10.636137Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-08_10_04_09-12526321693000694657'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0604200505'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-08T17:04:10.636137Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-06-08_10_04_09-12526321693000694657]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-06-08_10_04_09-12526321693000694657
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-08_10_04_09-12526321693000694657?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-08_10_04_09-12526321693000694657 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:09.547Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:09.547Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:14.420Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-2 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.049Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.071Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.146Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.174Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.215Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.243Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.300Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.356Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.389Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.429Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.459Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.494Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.521Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.547Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.574Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.617Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.645Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.670Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.707Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.744Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.776Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write/NativeWrite into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.805Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.833Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.870Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.904Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:15.966Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:16Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:16.025Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:51.971Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:04:57.030Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:05:31.907Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:05:31.933Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:13:18.492Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:13:18.530Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:13:18.562Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:13:18.597Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:13:18.632Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:14:09.860Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:14:09.915Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-06-08T17:14:09.944Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-06-08_10_04_09-12526321693000694657 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: b5077d035b75496ab3b6144bcf258632 and timestamp: 1623172461.450385:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 74
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-08_10_04_09-12526321693000694657?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1913, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 641, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 651, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2052, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 136, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 694, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 820, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2335, in expand
raise ValueError(msg)
ValueError: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 16m 53s
80 actionable tasks: 52 executed, 28 from cache
Publishing build scan...
https://gradle.com/s/xyaukqrzd2y4i
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org